Strange results while nesting scope constructs?

Question

I did a handful of tests with Block, With, and Module and had a couple of new surprises. First, I explicitly nested all pairs, producing the following nine tests, naively expecting 37 in each case:

{Block[{x=42}, Block[ {x=37},x]],
 Block[{x=42}, With[  {x=37},x]],
 Block[{x=42}, Module[{x=37},x]],
 With[ {x=42}, Block[ {x=37},x]],
 With[ {x=42}, With[  {x=37},x]],
 With[ {x=42}, Module[{x=37},x]],
 Module[{x=42},Block[ {x=37},x]],
 Module[{x=42},With[  {x=37},x]],
 Module[{x=42},Module[{x=37},x]]}

During evaluation of In[48]:= Block::lvset: Local variable specification {42=37} contains 42=37, which is an assignment to 42; only assignments to symbols are allowed. >>

Out[48]= {37, 37, 37, Block[{42 = 37}, 42], 37, 37, 37, 37, 37}

The surprise is that With...Block... produces a collision, but With...With... and With...Module... don't.

UNLESS, I try the test this way

With[{heads={Block,With,Module}},
  Table[heads[[i]][{x=42},heads[[j]][{x=37},x]],{i,3},{j,3}]]

During evaluation of In[49]:= Block::lvset: Local variable specification {42=37} contains 42=37, which is an assignment to 42; only assignments to symbols are allowed. >>

During evaluation of In[49]:= With::lvset: Local variable specification {42=37} contains 42=37, which is an assignment to 42; only assignments to symbols are allowed. >>

During evaluation of In[49]:= Module::lvset: Local variable specification {42=37} contains 42=37, which is an assignment to 42; only assignments to symbols are allowed. >>

Out[49]= {{37, 37, 37}, {Block[{42 = 37}, 42], With[{42 = 37}, 42],
Module[{42 = 37}, 42]}, {37, 37, 37}}

in which case ALL the With... cases produce collisions.

I thought I was starting to understand MMA's scoping constructs :(

EDIT: in case my question wasn't clear, I'd be grateful for the following explanations in more detail than the documentation offers: (1) why x collides in the explicit With...Block... case, (2) why x collides in two more cases when the nesting constructs are built using Table instead of explicitly.

it occurred to me late to include Function and Table in the mix of scoping constructs, since they also give temporary bindings to variables. But I haven't tried that yet.

Answer 1

I think, your main reported puzzle - the collision in the With - Block case can be understood by recalling that Block is a dynamic scoping construct, not lexical. The most important difference in our present context is that for dynamic scoping, its influence is felt by the code very late - only at the evaluation stage, because symbols' values are being localized rather than symbols' names. Therefore, Block presents no "danger" whatsoever, for its enclosing scoping constructs, be those lexical or dynamic.

The main essence of the "inner scoping construct wins" rule is to protect the inner scoping constructs from lexical bindings performed by outer scoping constructs, since lexical scoping constructs evaluate from outside to inside, and introduce changes (variable bindings, or even replacements for With), before the actual evaluation. For Block, while variable bindings are happening early, the code itself is not modified in any way. So, it seems rather natural to me that the designers of Mathematica chose to not enforce the "inner scoping construct wins" when Block is the inner scoping construct.

It may help to present a very crude imitation of Block here:

ClearAll[myBlock];
SetAttributes[myBlock, HoldAll];
myBlock[{Set[var_Symbol, expr_]}, code_] :=
 With[{oldvar = var},
   var = expr;
   With[{result = code},
     Clear[var];
     If[oldvar =!= var, var = oldvar];
     result
   ]
 ]

While it does not have many desirable features that real Block has, it captures the essence of what Block does for simple variable (OwnValues) localization. For example:

myBlock[{x=37},x]

(*  37  *)

myBlock[{x=37},myBlock[{x= 42},x]]

(*  42  *)

Now, we can see what is happening in With - Block case:

With[{x=42},myBlock[{x=37},x]]

(*  myBlock[{42=37},42]  *)

which is exactly what also happens to built-in Block.

To summarize: I see the reason for the "inner scoping construct wins" not being enforced for Block because this rule only makes sense for lexical scoping constructs. This is so because lexical scoping constructs physically change the code before it executes (variable renaming for Module, expression substitution for With) , and can break inner scoping constructs since those execute later. Since dynamic scoping constructs change variable values, but not code, there is no such danger for them.

As to the specific With-Block case, some contradiction is here even from the start: With signals that variables localized with it are local constants, so can not be changed in the body of With. Block, however, manifests that it localizes essentially variables, symbols that may have different values. One can also use it without a conflict in such a way:

Clear[x,y];  
With[{x = y}, Block[{x = 37}, x]]  

where effectively Block localizes the y symbol, not x, as can be easily seen using Trace. Whether or not such uses can ever be constructive I don't know.

Answer 2

Block isn't respected as a scoping construct from the outside. Variables aren't renamed when something is injected inside by other scoping constructs, or by a replacement, like they are in Module and With. This makes some sense because Block doesn't do lexical scoping anyway

Your Table example can be explained by thinking about how a scoping construct can "respect" the inner constructs.

Imagine you are With (I hope you have good imagination). You receive some expression and you have to replace all the x by 37. But you don't want to mess with certain structures. So, before doing the replacement, you scan for those structures, and avoid them... But, you haven't evaluated your argument, so you can't know if Identity[Module][...] will end up having a proper Module[...] structure or not. You simply don't recognise it as a Module, and you mess with it...

Then, you give Rojo an upvote and he goes to sleep