Context of localised (dynamic) symbols

Question

When you localise a symbol with Module, the created unique symbol is usually in the current context. There seems to be an exception to this which has further consequences.

Let us assume we define a symbol in a package. We can mimic this by defining a symbol in some context and append this context to the $ContextPath:

AppendTo[$ContextPath, "MyContext`"]
MyContext`x = "Package Variable";

If you look at ??x you'll see that we really have only one x in MyContext`. What is surprising at first glance is the following:

Module[{x},
 Information[x]
]

While you might be tempted to say "who cares?", this has consequences! Without evaluating the following, try to guess the output:

Module[{x = "Local Variable"},
 {x, MyContext`x}
]

The output is

{"Local Variable", "Local Variable"}

There are further consequences: The standard context of DynamicModule variables is $CellContext. This means, even when you have defined some variable in Global` context, if you use the same name inside DynamicModule, it comes into the $CellContext context. With our MyContext`x this is different.

Evaluate this

Dynamic[MyContext`x]

and use this minimal dynamic

DynamicModule[{x = .5},
 {Slider[Dynamic[MyContext`x]], Dynamic[MyContext`x]}
]

Again, try to guess what you expect when you move the slider. After that, just change the local variable to

DynamicModule[{y = .5},
 {Slider[Dynamic[MyContext`x]], Dynamic[MyContext`x]}
]

The important part is that in the first place, the DynamicModuleBox defines (look at the cell expression!)

DynamicModuleBox[{MyContext`x$$ = ....

while in the second example it is $CellContext as expected.

---

Question: Can anyone explain why I should regard this as good language design or as intuitive? Wasn't it possible to solve this more carefully? Can anyone offer an explanation why I shouldn't care about this (beside the fact that it obviously works for a large code-base)?

Anyone who is further interested can read the chat log, because it was Kuba who brought the issue up.

Answer 1

I will make no attempt to defend the fact that Mathematica simulates scoping by means of variable renaming. However, the behaviour that we see is consistent with the principles under which Mathematica does operate.

Whenever Mathematica tries to interpret a symbol name, it first checks to see whether a symbol with that name already exists in a package in the context path. If it does, that symbol is used. If not, a new symbol is created. So, when presented with the expression:

Module[{x}, Information[x]]

The system will interpret it as if we wrote:

Module[{MyContext`x}, Information[MyContext`x]]

When written in full this way, it is easy to see why we get the result we see -- especially when we take into account the fact that scoping is simulated by variable renaming. But what about the alternative? What if the input was interpreted as:

Module[{Global`x}, Information[Global`x]]

This would mean that the symbol Global`x would be created. This would immediately shadow the definition from MyContext`x, rendering it inaccessible in unqualified form. It would also place in doubt the meaning of any subsequent unqualified references to x. This behaviour would be much more surprising and irritating than the present behaviour. The present behaviour is confined, at least, to some uncommon corner cases.

The consequences of the other examples are also easier to see when they are written out in full after symbol resolution (and taking the scope renaming rule into account):

Module[{MyContext`x = "Local Variable"}, {MyContext`x, MyContext`x}]

DynamicModule[{MyContext`x = .5}, {Slider[Dynamic[MyContext`x]], Dynamic[MyContext`x]}]

It is for these very reasons that the Mathematica naming recommendations are what they are. Exported names are expected to be long and descriptive to maximize the chance of global uniqueness. Global variables are expected to start with a dollar sign. It is considered bad practice to export short names that start with lower case letters (especially ones like x and y) because the chance of collision is near certainty. Short, common, names are expected to remain private to a package.

Good language design? Not so sure. But we must remember that Mathematica is focused squarely upon symbolic manipulation. It tends to avoid prejudging the role that a symbol will play in an expression: is it a variable? a function? inert data? For expediency, the designers have chosen a middle ground between complete neutrality when interpreting expressions and completely conventional scoping and symbol interpretation. That middle ground attempts to give us the power of symbolic manipulation while still supporting some of the programming paradigms we use in other languages. I think that the present behaviour under discussion, as well as the various scope leaks etc, are inevitable consequences of the seam between these two extremes.

Intuitive? I do not think that this behaviour can be considered intuitive when coming from another system. But I think that one can build a pretty good intuition over time provided one is willing to accept the heavily symbolic paradigm that Mathematica promotes. For the case in hand, this is where our intuition needs to expressly take on the concept that symbols are only created when they do not exist in a $ContextPath package. We also need to take on the fact that scoping is simulated by variable renaming. Once we accept these ideas (perhaps grudgingly), our intuition can develop.

DynamicModule

Let's now examine the DynamicModule cases. It was observed that the expressions Dynamic[{x = .5}, ...] and Dynamic[{y = .5}, ...] behaved differently.

Once we take into account the symbol resolution logic discussed above, we note that the first dynamic module is being asked to localize MyContext`x. Thus, the internal references to that symbol will be replaced at execution time by localized symbols. That is what we see: when the slider is dragged, the outer Dynamic[MyContext`x] does not change.

In contrast, the second dynamic module is localizing Global`y. This is irrelevant. What is important is that MyContext`x is not being localized. That is why the outer dynamic changes when we drag the slider.

Even though there is no apparent scope leak in this case, there remains a worry stemming from the observation that the first dynamic module used the variable placeholder name MyContext`x$$ whereas the second used a placeholder named $CellContext`y$$. Can the absence of $CellContext from the former placeholder lead to a scope leak at some point?

The answer is "no". The key word in the preceding paragraph is placeholder. The symbols used in the cell expression are not the symbols that will be used at execution time. We can observe this by evaluating the following expression:

DynamicModule[{v = 1}, { Dynamic[v], Dynamic[Unevaluated@v] }]

(* {1, FE`v$$21} *)

The cell expression of the result looks like this:

Cell[BoxData[
 DynamicModuleBox[{$CellContext`v$$ = 1}, 
  ...
    DynamicBox[ToBoxes[$CellContext`v$$, StandardForm],
  ...
    DynamicBox[ToBoxes[Unevaluated[$CellContext`v$$], StandardForm],
  ...

Notice how the return value used the fully localized symbol FE`v$$21 whereas the cell expression uses the placeholder $CellContext`v$$.

What exactly is a placeholder? It is best to think of a dynamic module as template from which user interface widgets can be generated. An important design goal of dynamic modules is that the user must be able to copy-and-paste a widget to obtain a completely detached and fully functional copy of that widget. Localization is a critically important factor that makes this possible. Placeholders are used for (at least) two purposes. First, they show where localized variables need to be inserted into the form. Second, they act as keys to hold the persistent values of the dynamic variables between front-end sessions. In these roles, the exhibited localization is adequate.

It remains a mystery to me as to why the placeholders of the form Global`x get changed into $CellContext`x. It appears to serve some arcane internal purpose -- largely unrelated to the final execution scope. [Update: @Kuba points out that there is information about $CellContext here].

The life-cycle of a dynamic module is actually a complicated business. There are multiple evaluations, performed in at least five distinct scopes. In interest of keeping this post merely unreasonably long, I shall only give a brief glimpse of the process. From a fresh session (both front-end and kernel), evaluate this expression:

AppendTo[$ContextPath, "MyContext`"];
MyContext`x = "Package Variable";

Then, as a separate action, evaluate this ugly piece of work:

DynamicModule[{MyContext`x = 0, y = 1},
  Print @ Unevaluated @ {MyContext`x, y}
; { Unevaluated @ {MyContext`x, y}
  , Dynamic @ Unevaluated @ {MyContext`x, y}
  , Button["Wormhole"
    , Print @ DynamicModule[{}
      , {Dynamic @ Unevaluated @ MyContext`x, Dynamic @ MyContext`x}
      , InheritScope -> True
      ]
    ]
  , Slider[Dynamic[MyContext`x]]
  , Dynamic[MyContext`x]
  }
]

(* {x$898, y$898}
   {Unevaluated[{x$$, y$$}], {FE`MyContext`x$$25, FE`y$$25}, <<button>>, <<slider>>}
 *)

Including our original variables, I count four versions of each of our x and y variables. If we examine the cell expression, we find the placeholders MyContext`x$$ and $CellContext`y$$. That makes five versions. Each of these versions plays a role in the various evaluation stages of the dynamic module. When the dust clears at the end, the variables are pretty damn localized. Note how the final x and y symbols were successfully localized irrespective of the presence or absence of $CellContext. There may be some bugs in this process, but they will not be from lack of trying.

And if that were not enough, press the Wormhole button that is created, to produce this output:

(* {FE`MyContext`x$$25, 0} *)

Aha! One might think that we can sneak into the dynamic module's scope like this:

Dynamic[FE`MyContext`x$$25]

If we actually try it, it does not work. Why? The answer lies in comparing the cell generated by the Wormhole button to our hand-coded Dynamic. Our hand-coded cell lacks the generated DynamicModuleBox option DynamicModuleParent->BoxObject[7345]. Without it, our evaluations do not receive notifications when the kernel variable changes. Another topic that this post will pass over.

Reproducing These Results

For those excited at the prospect of experimenting in this space, a few words of advice:

• Remember that symbols are created after an expression is read but before it is evaluated.
• Remember that active dynamics can generate symbols and perform evaluations merely by becoming visible (especially when a notebook is first opened).

Failure to take these points into careful consideration will result in symbols appearing in unexpected contexts.

How Many Scope Strategies Are There?

If you have been keeping score, there are a lot of different scoping strategies touched upon in this post, and even more in Mathematica as a whole. Here is an incomplete list to give a taste:

• module-style renaming: x$123
• function-style renaming: x$
• placeholder-style renaming: $CellContext`x$$ or MyContext`x$$
• wormhole-style renaming: FE`x$$1234 or FE`MyContext`x$$1234
• block scope: first class dynamic scoping
• compiled scope: first class traditional scoping (mostly)
• pattern scope: first class traditional scoping??

Conclusion

To conclude, I observe that most of the time we can blithely ignore these issues. This speaks to the fact that the designers seem to have done a pretty good job papering over the boundary between pure symbolic and "conventional" programming. But the boundary exists, and when we stumble over it the failure tends to be spectacular. We must defend ourselves by building our intuition about how Mathematica handles these cases. Only then do we stand a chance to foresee/avoid or diagnose/fix such problems (which unfortunately tend to arise at run-time, not design-time).

(For the record, I am still of the opinion that none of this argumentation precludes the desirability of first-class module-like scoping facilities within in the language.)