9
$\begingroup$

Why does Mathematica choose the mechanism that makes the global variable a here while parsing?

In

f := Module[{a}, a;]

In

?a

Out

Global`a

I don't understand why Mathematica goes in the way that the global variable a is declared and not removed. Maybe there are its own purposes or intentions of why Mathematica is constructed in that way. Sometimes the way Mathematica works feels very confusing to me.

Thank you. :)

$\endgroup$
  • 4
    $\begingroup$ Module puts the symbol into the Global context (if it's not already there). Module replaces references to a global symbol with a temporary symbol (s$n, where n differs each time), which is how the localization is done. Unless you return the temporary, it gets thrown away on exit of module, but the Global symbol (whether created by use of Module, or one that was already in the Global context that module localized) will be there. $\endgroup$ – ciao May 28 '16 at 6:09
  • 2
    $\begingroup$ @ciao Why does it have to be that way? Why isn't it created in its s$n form right away? Why does it have to get through the process of becoming in form of global variable? $\endgroup$ – Smart Humanism May 28 '16 at 9:02
  • 5
    $\begingroup$ Possible duplicate of Local variables in Module leak into the Global context $\endgroup$ – WReach May 28 '16 at 15:38
  • $\begingroup$ Just to make it clear: I didn't find the linked discussion, of which this is possibly a dupe, when answering, and added a link to my answer later. Also, while this one is very closely related, the fundamental question asked here is different: it is why more than what to do with it, and so is my answer, which answers the why part primarily, from the language design p.o.v. So, to me this being a dupe is not a clear cut, which is why I am not putting a close vote now. $\endgroup$ – Leonid Shifrin May 28 '16 at 18:09
  • 1
    $\begingroup$ Related: "When does Mathematica create a new Symbol?" $\endgroup$ – Alexey Popkov May 29 '16 at 6:07
20
$\begingroup$

Why is this happening

The explanation was basically given by ciao in comments. You can also find a lot of information on this in this great answer of Mr.Wizard. I will perhaps try to view it from a somewhat different perspective.

To understand what happens, one should go back and consider what happens when we enter and execute some code. The steps are roughly the following

  • The string code is parsed by the Mathematica parser
    • In the FrontEnd, top-level statements in the cell are passed to the kernel as separate evaluations
    • In packages, the code is parsed line by line, and complete expressions are evaluated as soon as they are formed
  • At the time the parsing happens, all Mathematica cares about is that it is given a well-formed expression. There is no semantics attached to any expression yet, since evaluation happens only after parsing.
  • During the parsing, new symbols are created if needed. This is the central point here. The decision to create a new symbol is made by the parser, based on the set of symbols in the current context, and those available in other contexts currently on the $ContextPath. Again, no semantics of any kind is involved in this process.
  • Finally, parsed expression is allowed to evaluate, according to semantics of the symbols involved. Here, is is important to realize, that in general there is no correlation between the time code is parsed and the time it is evaluated. For example, Module in delayed definitions (e.g. functions) will only be evaluated when the function is called (which may happen at an unspecified moment in time, or not happen at all).

Since Module[{a}, a] is an expression like any other, it is parsed first, and the symbols a is created in the current working context (if it doesn't yet exist there or can not be found in other contexts currently on the $ContextPath), according to the general rule.

The fact that Module constructs symbols a$n as a part of its localization mechanism, is an evaluation - time effect. This happens much later than the construction of a during the parsing, and is a completely independent process. There is no way it can affect the parse-time artefacts like that.

Why I think this is the right behavior for Mathematica

I actually think that this is the right behavior. Why? Because in this way, the system stays completely true to its general principles, and does not create new rules and corner cases, which would make it yet harder to understand. The language is complex enough that we really don't need any additional complexity.

In fact, even though the lexical scoping as currently implemented in Mathematica has a number of issues (can be broken, is only an emulation, creates new symbols and is based on renaming rather than true localization), I am still a big fan of it. The reason is that it is very hackable, and when you realize it, you can bend it to do what you want, very easily. The whole fact that lexical scoping is done via symbol renaming, makes entire scoping completely available to the top-level user.

An example: a version of Module with an additional way to access local variables

Let me give an example, which I actually used in my code recently. We will construct a version of Module with a more fine-grained access to localized variables - it will give the user a way to access the localized variables by their string names inside Module, and also modify them:

ClearAll[varRegisteringModule];
SetAttributes[varRegisteringModule, HoldAll];
SyntaxInformation[varRegisteringModule] = {"LocalVariables" -> {"Solve", {1, 1}}};
varRegisteringModule[vars_, body_] :=
   Module[{getVars, strvars},
      SetAttributes[getVars, {HoldAll, Listable}];
      getVars[Set[v_Symbol, rhs_]] := Hold[v];
      getVars[v_Symbol] := Hold[v];
      (* String names for private variables. Must be evaluated before we enter Module *)        
      strvars = Function[v, ToString[Unevaluated[v]], {HoldAll, Listable}][vars];
      Module[vars,
        Block[{$private  = AssociationThread[strvars, getVars[vars]]},
 	    (* Evaluate inside Module, to get the vars bound to actual symbols produced by Module *)
  		      $private /: Set[$private[name_String], val_] /; KeyExistsQ[$private, name] :=
             Replace[ $private[name], Hold[var_] :> (var = val)];
          body
        ]
      ]
   ];

Here is how we can use it:

varRegisteringModule[{a, b, c},
  Print[$private["a"]];
  $private["a"] = 1;
  Print[a]
]

(*

  Hold[a$17084]

  1
*)

where you see that we can actually both access the exact name of the generated Module variable and also modify it, by using a new Module - local variable $private. Had I not put it into Block, and we could access these local variables even after Module evaluated.

Other use cases

Other class of examples comes from renaming system, which protects inner scoping constructs. More details on that can be found here and in the links therein. But the main point here is that we can always bend the system and fool that renaming mechanism, when we need to. It is a separate question how adequate is renaming scheme to implement lexical scoping, but having this sort of freedom to actually reprogram every aspect of the scoping on the top-level is very valuable and unique.

Finally, just the ability to build functions at run-time like so:

Block[{x}, Function @@ {x, f[x]}]

can be quite valuable, since it postpones function construction until run-time.

Why is that useful

Of course, you may ask when is this sort of access needed. Well, it becomes important for certain advanced applications, where you want to implement your own scoping scheme. But the main thing for me is that the system makes this sort of things possible at all, and actually quite transparent. Had Module acted as a black box, and there would be no way to make constructs like that.

The price for this freedom is, however, that things like scoping are delayed until run-time. But this means that they can in no way be coupled to events happening during the parsing of code / expressions, if the system wants to remain transparent and easy to understand. Thus, there is no way that those symbols constructed during the parsing stage could be removed by Module (and other lexical scoping constructs like With or Rule / RuleDelayed, Function, etc), because it is completely unknown in general when these constructs will actually be called, if at all, and that is completely uncorrelated with when they were parsed.

What to do about it

The best method to avoid these problems is to use packages, and place all implementations of your functions into the "`Private`" subsection of a package. In that case, those symbols will still be created, but they will be created in the "`Private`" sub-context, and won't in any way pollute your exported namespace or interact with anything important.

Actually, this is one good reason to always place your implementations in the "`Private`" sub-context, even if there are no explicit private functions / symbols you introduce yourself - because most of the lexical scoping constructs (including pattern variables in Set / SetDelayed), will produce such parasitic symbols, which you really don't want to leak into the public part of your package.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Wow, thanks so much, Leonid Shifrin! What an Awesome Explanation!! $\endgroup$ – Smart Humanism May 28 '16 at 15:16
  • $\begingroup$ @SmartHumanism Was glad to help. This is an important and rather subtle topic, and it is good to have some explanation for this on the site. Partially this has been discussed before in that question with Mr.Wizard's answer I linked to, but it wasn't really concerned with why this happens. So, I think, this discussion nicely complements that one, and together they cover this issue pretty well. Thanks for the accept, b.t.w. $\endgroup$ – Leonid Shifrin May 28 '16 at 15:28
  • 1
    $\begingroup$ Awesome reply - I should have worded my comment better - makes it sound like Module puts symbol into Global, when it's the parser... $\endgroup$ – ciao May 28 '16 at 20:06
  • $\begingroup$ @ciao Thanks. I always appreciate your opinion. In this particular case, I felt we really need more detailed discussion on this topic, and I think it is still good to have it even though the other similar discussion apparently took place before. Because this behavior, apart from being not very obvious to the uninitiated, is also a single good reason to put stuff in packages even when there are no private functions. This is important, and I haven't seen this issue discussed elsewhere in any detail. $\endgroup$ – Leonid Shifrin May 28 '16 at 20:16
  • 1
    $\begingroup$ @SmartHumanism Thanks. Your remark made my evening :) $\endgroup$ – Leonid Shifrin Jul 1 at 18:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.