Why can't I inject expressions in Compile using (only) With

Generally speaking, With can be used to inject into arbitrarily held expressions, for example:

With[{args = 2},
Hold @ Hold @ HoldComplete[args]
]
(* Out[1]= Hold[Hold[HoldComplete[2]]] *)
Attributes @ foo = HoldAll;
With[{args = {{x, _Real}}},
foo[args, x + 2]
]
(* Out[2]= foo[{{x, _Real}}, x + 2] *)


However, trying to do the same exact thing to inject inside Compile does not work:

With[{args = {{x, _Real}}},
Hold @ Compile[args, x + 2]
]
(* Hold[Compile[args, x + 2]] *)


Also, the only relevant attribute of Compile is HoldAll, which as shown in the first examples is not really a problem for this kind of injection. Does this mean that Compile follows some special evaluation rules (even though it's not listed among the magic symbols)?

Using a Trott-Strzebonski-like trick, like shown for example here, does work:

With[{args = {{x, _Real}}},
Hold @ Compile[args, x + 2] /; True
][[1]]
(* Hold[Compile[{{x, _Real}}, x + 2]] *)


An extract from Leonid's explanation in the above linked answer explains why adding the Condition works:

The basic idea is to exploit the semantics of rules with local variables shared between the body of With and the condition, but within the context of local rules. Since the condition is True, it forced the eval variable to be evaluated inside the declaration part of With (...)

This explanation does not however explain why the use of Compile in particular makes a difference, why does it prevent With from injecting into it, and why can we avoid this constraint adding a Condition?

Some more examples of functions showing or not showing this kind of behaviour:

With[{args = x}, Hold@Compile[args, 2]]
With[{args = x}, Hold@Function[args, 2]]
With[{args = x}, Hold@Module[args, 2]]
With[{args = x}, Hold@With[args, 2]]
With[{args = x}, Hold@Block[args, 2]]
(*
Out[1]= Hold[Compile[args, 2]]
Out[2]= Hold[Function[args, 2]]
Out[3]= Hold[Module[x, 2]]
Out[4]= Hold[With[x, 2]]
Out[5]= Hold[Block[x, 2]]
*)


Also, in at least some cases it matters whether there is a second argument:

With[{args = x}, Hold @ Compile[args]]
With[{args = x}, Hold @ Function[args]]
(* Out[1]= Hold[Compile[args]] *)
(* Out[2]= Hold[x &] *)

• For the last case With[{args = x}, Hold @ Function[args]], my guess would be that a function with a single arg is treated as a slot-based pure function, which is not considered a scoping construct in the context of protection of local variables, since it does not use named variables (it uses slots instead). Which is why such protection is not there in this case. – Leonid Shifrin Aug 4 '17 at 12:40

Compile is considered a scoping construct by the outer With, and its bindings are protected. This will work:

With[{args = {{x, _Real}}}, Hold[Compile @@ Hold[args, x + 2]]]


or this (if you don't want to keep Apply in code):

Unevaluated[
With[{args = {{x, _Real}}}, Hold[Compile[args, x + 2]]]
] /. Compile -> $compile /.$compile -> Compile


One relevant link is this. If you don't want to care about inner scoping constructs, use replacement rules instead of With - the injector pattern at your service:

Hold @ Compile[args, x + 2] /. Unevaluated[args ->   {{x, _Real}}]


For the Trott-Strzebonski part, it works because there, With is a secondary device serving essentially replacement rules. There, you could directly use RuleCondition instead of With. So my point is, for Trott-Strzebonski trick, the essential part is that one uses replacement rules. And rules don't care about inner scoping constructs.

It has been noticed in comments and in the edit of the original question, that not all scoping constructs are treated in the same way, and in particular for Function (with named arguments) and Compile, the entire declaration is protected, while for With and Module the protection happens on the level of individual variables (in terms of name collisions).

• Another illustration: With[{args = x}, Hold@Function[args, x + 2]] – Mr.Wizard Aug 4 '17 at 12:12
• Leonid, wouldn't With[{args = {{x, _Real}}}, Compile @@@ Hold[Hold[args, x + 2]]] be a better substitution for this? – Mr.Wizard Aug 4 '17 at 12:12
• Interesting. I have a few questions though: 1) why is Compile considered a scoping construct? And is there a list of functions that are considered scoping constructs by With? It doesn't show in Attributes or such. 2) running my second example with With instead of Compile works as expected (as in, it replaces args), doesn't this contradict your argument about inner scoping constructs? – glS Aug 4 '17 at 12:13
• @Mr.Wizard thanks, I added that and some more examples in the question – glS Aug 4 '17 at 12:22
• @glS 1) Compile binds variables (in this case, declares their types), so it has to protect those variables (they can be defined outside, and that should not affect Compile) - which is why it is considered a scoping construct. 2) I would guess that the variable - protecting mechanism treats Compile somewhat differently from other scoping constructs. For those, it looks at conflicts of individual scoped variables, while here, it protects the full declaration. Consider this, for an example or protection in action: With[{args = {{x, _Real}}}, Hold@With[{args = {1, 2, 3}}, args + 2]]. – Leonid Shifrin Aug 4 '17 at 12:28

not sure but cannot we use Evaluate in this case?

With[{args = {{x, _Real}}},
Hold@Compile[Evaluate@args, x + 2]
]
(* Hold[Compile[Evaluate[{{x, _Real}}], x + 2]] *)

• Evaluate is the wrong tool here, and it only "works" because the presence of Evaluate hides Compile from outer With (it does no longer recognize Compile as a scoping construct). But this would bite you later, for example if x has been globally defined. – Leonid Shifrin Aug 4 '17 at 11:55
• @LeonidShifrin thanks i see. But as long as no global variable is defined the approach should work. I get your point, it is better to use Unevaluated on the whole expression and make the replacements inside – Ali Hashmi Aug 4 '17 at 12:04
• Evaluate only works one level deep, and even there only for heads which are not HoldAllComplete. In this case, the expression is too deep for Evaluate to have any effect at the time when external With evaluates. But later, it will force Compile to evaluate its first argument, which is not desirable for a scoping construct. – Leonid Shifrin Aug 4 '17 at 12:06
• @LeonidShifrin to better understand your last point do you mind giving an example that i can check, especially the case when the head has an attribute of HoldAllComplete – Ali Hashmi Aug 4 '17 at 12:19
• Just try HoldComplete[Evaluate[1+1]] for a start. Also, read the chapter 7 of David Wagner's book (chapter on evaluation), he explains this very clearly. – Leonid Shifrin Aug 4 '17 at 12:29