I define a variable as local to a module BUT then the module uses its global value! Why?

Question

I have a question about modules and local variables.

Here's my example:

h = 5;

Module[{a, h}, a[h_]= h^2; a[4]]

(*Out[2] = 25*)

I expected the module to return 16 and not 25! I believed h to be a LOCAL variable of the module!

I know that the following alternatives work and return correctly 16

Module[{a, h}, a[h_] := h^2; a[4]]
Module[{a, h}, a[h] = h^2; a[4]]
Module[{a, j}, a[j_] = j^2; a[4]]  (* with j a variable not previously defined *)
Block[{a, h}, a[h_] = h^2; a[4]]

Question: why does the module behave like that? i.e. why does it return 25 and not 16?

P.S. I am running out of variables and I am then resorting to modules to avoid errors due to variables defined more than once. I played a bit with modules to check their properties and I ran in this example.

Answer 1

What happens and why

As Daniel Lichtblau pointed out in comments, this behavior can also be viewed as a flaw in the current behavior / design / implementation of lexical scoping in Mathematica. However, it may be useful still to understand on a deeper level what happens, since it can be explained rather easily from the core rules of how lexical scoping operates in the default mode in current (and past) versions of Mathematica.

While Mr.Wizard already discussed part of what's happening in his (now deleted) answer, I think it pays off to understand this from a somewhat different angle. Set is a scoping construct, and as such, it protects its bindings from possible actions of the outer scoping constructs. The general mechanics of how this happens has been discussed in our answer, but let's focus on the specifics here.

One thing to understand is what is the action of the outer scoping construct (Module) in this case. The action is variable binding, that is, renaming of variable h to a newly produced local symbol looking like h$123, everywhere in the code, before the body of the scoping construct (Module) is allowed to execute. It is this action that Set has to guard its arguments against.

The second point here is that while the protection is needed for Set, the only function in position to enforce such protection is in fact Module, just because it is the only one being evaluated at that stage. So, Module's internal code has to detect Set and protect its arguments. This is very similar to other cases, but while there things get renamed, here, on the opposite, such renaming h -> h$123 has to be prevented.

Now, here is one way to show that this is indeed what happens: use a standard idiom to fool the inner scoping construct detection mechanism:

Module[{a, h}, Set @@ Hold[a[h_], h^2]; a[4]]

(* 16 *)

Now, Module can't detect Set inside, and therefore h is being successfully bound to the Module-generated variable, and we get what we'd naively expect.

So, to summarize: we have here the same situation as with other cases described in the linked post, but the main difference is that while there outer scoping constructs were in charge of making renamings, here it is the opposite: in the case of Set, the outer lexical scoping constructs are responsible for prevention of such renamings. In this particular case, Module is then forced to disable its own binding mechanism.

Note that SetDelayed is different, because while the Module behavior would be the same, SetDelayed makes its own bindings with it's r.h.s., which effectively localize the pattern variable correctly. Note also that this issue only holds for lexical scoping constructs, and dynamic scoping isn't affected by this.

Ways out

1. Use "StrictLexicalScoping" -> True system options setting

Thanks to Daniel's work, we have now an option available to change it. If you set

SetSystemOptions["StrictLexicalScoping" -> True];

Then the result is what it was expected to be:

Module[{a, h}, a[h_] = h^2; a[4]]

(* 16 *)

2. Use "standard" techniques to fool the inner scoping construct detection mechanism

This is as described above, but any such technique would do. See the linked post for some of the most commonly used. Admittedly, the need to use such tricks in such a case is a pain in the neck.

Summary

What happens can be viewed as a design / implementation flaw. Thanks to the work of Daniel Lichtblau, now there is an option available to switch to a mode which is free of these flaws. OTOH, the above behavior does not look like a plain bug to me, but rather a consistent behavior fully explained by the rules of how lexical scoping constructs operate in current versions of Mathematica in default mode. I have tried to explain the logic behind this behavior from the deeper principles of how scoping constructs currently behave in the default mode. It is another issue that those rules may be considered flawed, and we've seen discussions of many other instances exposing these flaws, here on SE.

Answer 2

Great question. My first reaction is that this shouldn't happen. h appears explicitly inside the Module and it should be substituted with the localized equivalent, e.g. h$123, even if it is held:

Module[{h}, HoldComplete[h^2]]

HoldComplete[h$6992^2]

However named patterns in constructs such as Set and Rule are exempt from this substitution:

Module[{h}, Hold[h_ -> h^2]]

Hold[h_ -> h^2]

Module[{h}, Hold[f[h_] = h^2]]

Hold[f[h_] = h^2]

This is related to:

• Enforcing correct variable bindings and avoiding renamings for conflicting variables in nested scoping constructs

Until now I did not realize that this affected constructs without a Hold attribute. That is I knew that Module handles Function, SetDelayed, and RuleDelayed differently:

Module[{x}, Function[x, x^2]]

Function[x, x^2]

Module[{x}, f[x_] := x^2]
?f

f[x_] := x^2

Module[{x}, x_ :> x^2]

x_ :> x^2

This actually makes sense as the x is already being localized by these constructs and additional substitution by Module would only result in e.g.:

Function[x$123, x$123^2]

However in the case of Set or Rule, which do not have hold attributes, this is problematic, as these are not localizing constructs as correctly recognized by the syntax highlighter. Daniel Lichtblau wrote plainly:

No need to dance around this: it's a flaw in the implementation of the scoping mechanism. Set SetSystemOptions["StrictLexicalScoping" -> True]; and it gives 16.

So it seems that my first reaction was correct, and this shouldn't happen but does.

---

Further thoughts and exploration

Leonid comments below that Set and Rule are still scoping constructs. I do not contest this. As noted the difference is in the evaluation behavior controlled by hold attributes.

Let us ask the question:

• How should the right-hand-side Symbols in Set and Rule be handled within Module?

The existing default behavior leaves both the LHS and RHS x unchanged in this case which results in the RHS instance evaluating to the global value of x if it exists:

Module[{x}, x_ -> x]

x_ -> x

However a slight change results in different behavior:

Module[{x, y}, x_ -> x y]

x$_ -> x$ y$748

Here the automatic renaming mechanism kicks in and changes x to x$ on both sides of the Rule.

If as in SetDelayed and RuleDelayed the RHS is not expected to immediately evaluate this difference in renaming often will not matter. However with Set and Rule where the RHS is expected to evaluate this inconsistency is troubling. One cannot generally rely on the behavior illustrated in the original question as it will break if this renaming is activated:

h = 5; i = 7;

Module[{h, i}, h_ -> h^2]

Module[{h, i}, h_ -> i*h]

h_ -> 25

h$_ -> h$ i$787

From a user's perspective the simple introduction of i in the RHS should not result in an entirely different evaluation characteristic, yet it does. (This affects Set as well.) It would be desirable to have consistency in this behavior, and it seems that is exactly what Daniel's System Option provides:

SetSystemOptions["StrictLexicalScoping" -> True];

h = 5; i = 7;

Module[{h, i}, h_ -> h^2]

Module[{h, i}, h_ -> i*h]

h$_ -> h$^2

h$_ -> h$ i$1334

Now the h is renamed to h$ on both sides of both rules.