19
$\begingroup$

I have a question about modules and local variables.

Here's my example:

h = 5;

Module[{a, h}, a[h_]= h^2; a[4]]

(*Out[2] = 25*)

I expected the module to return 16 and not 25! I believed h to be a LOCAL variable of the module!

I know that the following alternatives work and return correctly 16

Module[{a, h}, a[h_] := h^2; a[4]]
Module[{a, h}, a[h] = h^2; a[4]]
Module[{a, j}, a[j_] = j^2; a[4]]  (* with j a variable not previously defined *)
Block[{a, h}, a[h_] = h^2; a[4]]

Question: why does the module behave like that? i.e. why does it return 25 and not 16?

P.S. I am running out of variables and I am then resorting to modules to avoid errors due to variables defined more than once. I played a bit with modules to check their properties and I ran in this example.

$\endgroup$
  • $\begingroup$ I assume in your code you used a semicolon instead of a comma between h^2 and a[4], otherwise you'd get an error about giving 3 arguments to Module. I edited it to fix the problem. $\endgroup$ – Jason B. Jan 29 '15 at 10:59
  • $\begingroup$ yes, thank you!! $\endgroup$ – pan_e_gaban Jan 29 '15 at 11:07
  • $\begingroup$ @JasonB Yes there seem to be many solutions, however this exampled revealed to me that I was not understanding something properly and I wanted to know what that something was in order to be more confident in using Modules etc... Anyway for the time being I might just as well create more variable names. Actually I wanted to use modules also to have a cleaner program. $\endgroup$ – pan_e_gaban Jan 29 '15 at 11:19
  • $\begingroup$ Module is a fantastic way to have a clean program, but your problem here is using = (Set) instead of := (SetDelayed). In my opinion, there are very few instances where it's better to use Set. $\endgroup$ – Jason B. Jan 29 '15 at 12:00
  • $\begingroup$ Possible duplicate: (42351) $\endgroup$ – Mr.Wizard Jul 8 '16 at 10:05
19
$\begingroup$

What happens and why

As Daniel Lichtblau pointed out in comments, this behavior can also be viewed as a flaw in the current behavior / design / implementation of lexical scoping in Mathematica. However, it may be useful still to understand on a deeper level what happens, since it can be explained rather easily from the core rules of how lexical scoping operates in the default mode in current (and past) versions of Mathematica.

While Mr.Wizard already discussed part of what's happening in his (now deleted) answer, I think it pays off to understand this from a somewhat different angle. Set is a scoping construct, and as such, it protects its bindings from possible actions of the outer scoping constructs. The general mechanics of how this happens has been discussed in our answer, but let's focus on the specifics here.

One thing to understand is what is the action of the outer scoping construct (Module) in this case. The action is variable binding, that is, renaming of variable h to a newly produced local symbol looking like h$123, everywhere in the code, before the body of the scoping construct (Module) is allowed to execute. It is this action that Set has to guard its arguments against.

The second point here is that while the protection is needed for Set, the only function in position to enforce such protection is in fact Module, just because it is the only one being evaluated at that stage. So, Module's internal code has to detect Set and protect its arguments. This is very similar to other cases, but while there things get renamed, here, on the opposite, such renaming h -> h$123 has to be prevented.

Now, here is one way to show that this is indeed what happens: use a standard idiom to fool the inner scoping construct detection mechanism:

Module[{a, h}, Set @@ Hold[a[h_], h^2]; a[4]]

(* 16 *)

Now, Module can't detect Set inside, and therefore h is being successfully bound to the Module-generated variable, and we get what we'd naively expect.

So, to summarize: we have here the same situation as with other cases described in the linked post, but the main difference is that while there outer scoping constructs were in charge of making renamings, here it is the opposite: in the case of Set, the outer lexical scoping constructs are responsible for prevention of such renamings. In this particular case, Module is then forced to disable its own binding mechanism.

Note that SetDelayed is different, because while the Module behavior would be the same, SetDelayed makes its own bindings with it's r.h.s., which effectively localize the pattern variable correctly. Note also that this issue only holds for lexical scoping constructs, and dynamic scoping isn't affected by this.

Ways out

1. Use "StrictLexicalScoping" -> True system options setting

Thanks to Daniel's work, we have now an option available to change it. If you set

SetSystemOptions["StrictLexicalScoping" -> True];

Then the result is what it was expected to be:

Module[{a, h}, a[h_] = h^2; a[4]]

(* 16 *)

2. Use "standard" techniques to fool the inner scoping construct detection mechanism

This is as described above, but any such technique would do. See the linked post for some of the most commonly used. Admittedly, the need to use such tricks in such a case is a pain in the neck.

Summary

What happens can be viewed as a design / implementation flaw. Thanks to the work of Daniel Lichtblau, now there is an option available to switch to a mode which is free of these flaws. OTOH, the above behavior does not look like a plain bug to me, but rather a consistent behavior fully explained by the rules of how lexical scoping constructs operate in current versions of Mathematica in default mode. I have tried to explain the logic behind this behavior from the deeper principles of how scoping constructs currently behave in the default mode. It is another issue that those rules may be considered flawed, and we've seen discussions of many other instances exposing these flaws, here on SE.

$\endgroup$
  • 1
    $\begingroup$ @pan_e_gaban Was glad to help! Thanks for the accept. Just keep in mind, that generally, it is a good practice to wait with accepting an answer for a little bit, to encourage others contribute more answers. $\endgroup$ – Leonid Shifrin Jan 29 '15 at 12:40
  • 6
    $\begingroup$ No need to dance around this: it's a flaw in the implementation of the scoping mechanism. Set SetSystemOptions["StrictLexicalScoping" -> True]; and it gives 16 (but you already knew that..) $\endgroup$ – Daniel Lichtblau Jan 29 '15 at 15:30
  • 2
    $\begingroup$ Small correction: Yes, I have looked at the internal code. No, I do not understand it in all detail. If I did, I'd almost surely need to be institutionalized. $\endgroup$ – Daniel Lichtblau Jan 29 '15 at 16:35
  • 1
    $\begingroup$ @DanielLichtblau For me, the main value in such deeper digging is that occasionally, one can find constructive uses of such behavior. Because really, even though it is flawed, it can be entirely controlled by the user (who has a clue of how these things work), like those techniques to fool that mechanism in some cases. So experienced users knowing what they are doing can take advantage of that. Sealing the scoping more tightly is likely to close a number of such possibilities. Still, for the majority of users, the correct but less controllable behavior is better. $\endgroup$ – Leonid Shifrin Jan 29 '15 at 16:41
  • 1
    $\begingroup$ @DanielLichtblau As to working correctly in all cases, I am not even sure that the "transparent" model of lexical scoping where all bindings are programmatically exposed and we have transparent (say to pattern-matching) expressions, allows that even in theory. You will probably be able to construct counter-examples with leaks no matter how sophisticated the new design of lexical scoping would be, if we keep this kind of transparency. At least I wouldn't be very surprised if that were true. So the question may be more about solving it for the vast majority of actual uses, than completely. $\endgroup$ – Leonid Shifrin Jan 29 '15 at 16:47
7
$\begingroup$

Great question. My first reaction is that this shouldn't happen. h appears explicitly inside the Module and it should be substituted with the localized equivalent, e.g. h$123, even if it is held:

Module[{h}, HoldComplete[h^2]]
HoldComplete[h$6992^2]

However named patterns in constructs such as Set and Rule are exempt from this substitution:

Module[{h}, Hold[h_ -> h^2]]
Hold[h_ -> h^2]
Module[{h}, Hold[f[h_] = h^2]]
Hold[f[h_] = h^2]

This is related to:

Until now I did not realize that this affected constructs without a Hold attribute. That is I knew that Module handles Function, SetDelayed, and RuleDelayed differently:

Module[{x}, Function[x, x^2]]
Function[x, x^2]
Module[{x}, f[x_] := x^2]
?f
f[x_] := x^2
Module[{x}, x_ :> x^2]
x_ :> x^2

This actually makes sense as the x is already being localized by these constructs and additional substitution by Module would only result in e.g.:

Function[x$123, x$123^2]

However in the case of Set or Rule, which do not have hold attributes, this is problematic, as these are not localizing constructs as correctly recognized by the syntax highlighter. Daniel Lichtblau wrote plainly:

No need to dance around this: it's a flaw in the implementation of the scoping mechanism. Set SetSystemOptions["StrictLexicalScoping" -> True]; and it gives 16.

So it seems that my first reaction was correct, and this shouldn't happen but does.


Further thoughts and exploration

Leonid comments below that Set and Rule are still scoping constructs. I do not contest this. As noted the difference is in the evaluation behavior controlled by hold attributes.

Let us ask the question:

  • How should the right-hand-side Symbols in Set and Rule be handled within Module?

The existing default behavior leaves both the LHS and RHS x unchanged in this case which results in the RHS instance evaluating to the global value of x if it exists:

Module[{x}, x_ -> x]

x_ -> x

However a slight change results in different behavior:

Module[{x, y}, x_ -> x y]
x$_ -> x$ y$748

Here the automatic renaming mechanism kicks in and changes x to x$ on both sides of the Rule.

If as in SetDelayed and RuleDelayed the RHS is not expected to immediately evaluate this difference in renaming often will not matter. However with Set and Rule where the RHS is expected to evaluate this inconsistency is troubling. One cannot generally rely on the behavior illustrated in the original question as it will break if this renaming is activated:

h = 5; i = 7;

Module[{h, i}, h_ -> h^2]

Module[{h, i}, h_ -> i*h]
h_ -> 25

h$_ -> h$ i$787

From a user's perspective the simple introduction of i in the RHS should not result in an entirely different evaluation characteristic, yet it does. (This affects Set as well.) It would be desirable to have consistency in this behavior, and it seems that is exactly what Daniel's System Option provides:

SetSystemOptions["StrictLexicalScoping" -> True];

h = 5; i = 7;

Module[{h, i}, h_ -> h^2]

Module[{h, i}, h_ -> i*h]
h$_ -> h$^2

h$_ -> h$ i$1334

Now the h is renamed to h$ on both sides of both rules.

$\endgroup$
  • 2
    $\begingroup$ @Mr.Wizard I don't quite agree with your conclusions here. And here is why. The fact that Rule and Set aren't HoldAll, does not prevent them from being scoping constructs. The defining property of a scoping construct is that, apart from special evaluation stage and semantics, it also has a precluding variable binding stage. That Rule and Set are not HoldAll, influence their evaluation semantics, but not binding semantics, which is for them in fact the same, as for RuleDelayed and SetDelayed, respectively. Namely, protect their bindings from outer scoping constructs. $\endgroup$ – Leonid Shifrin Jan 29 '15 at 23:03
  • 2
    $\begingroup$ @Mr.Wizard So, during the binding stage, both l.h.s. and r.h.s. pattern variables are being protected. It is this "protection request", that forces Module to disable its binding procedure for these inner local variables. In other words, from the binding perspective, h remains local also for Rule and Set, and the fact that the global value is used for it during evaluation, is an effect of evaluation semantics, and happens during evaluation stage, which follows the binding stage. $\endgroup$ – Leonid Shifrin Jan 29 '15 at 23:06
  • 2
    $\begingroup$ @Leonid Thank you for your wisdom as always. I am/was semi-aware of that point which is why I chose the weasel-words "localizing constructs." I think somewhere in there I have a valid point and I'll try to think of how to improve this. $\endgroup$ – Mr.Wizard Jan 29 '15 at 23:09
  • 2
    $\begingroup$ @Mr.Wizard So, according to the rules of how scoping constructs work and resolve conflicts in inner constructs, the observed behavior is actually correct, when we recall that Set is 1. Scoping construct requesting to protect its variable bindings (both l.h.s. and r.h.s, this has nothing to do with its evaluation semantics), and 2. Has evaluation semantics which in fact evaluates the r.h.s during assignment. $\endgroup$ – Leonid Shifrin Jan 29 '15 at 23:09
  • 4
    $\begingroup$ @Simon It's a sticky situation now. If they change the default behavior to "StrictLexicalScoping" -> True certain legacy code will break, however I think that is the more consistent behavior as I attempt to illustrate in my latest update. A new scoping function would solve this problem at the expensive of yet another function to wrangle; as a terse code fanatic I would hate the name StrictLexicalModule but what else do you call it? Local perhaps? $\endgroup$ – Mr.Wizard Jan 29 '15 at 23:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.