How symbol lookup actually works

Bug introduced in V6 and fixed in V11.3

The behavior indeed changed but now the documentation is clear about it.

This code is inconsistent with the description from Power Programming with Mathematica:

x = 5;
tempx = 6;
Begin["temp"]
{x, Globalx, tempx}


The result in my Mathematica session is {5, 5, 6}, but it's {6, 5, 6} in Section 8.1.1 of Power Programming with Mathematica (page 231 of the hardcopy or page 249 of the PDF).

• may be you found a bug in the book or a bug in Mathematica :) – Nasser Mar 8 '14 at 9:06
• Try this for more strange result: x = 5; Begin["temp"]; x = 6; {x, Globalx} which gives {6,6} ? – Nasser Mar 8 '14 at 9:17
• @Nasser - If you use BeginPackage to put temp  on the context path it works, but this demo also needs a line break: x = 5; BeginPackage["temp"]; (*line break*) x = 6; {x, Globalx} – Chris Degnen Mar 8 '14 at 12:21
• The very same question was asked a couple of days ago. At that time I noted that even the current v9 documentation is incorrect because it states that when looking for a symbol that was entered, Mathematica searches $ContextPath after searching $Context, even though in reality it searches $ContextPath before $Context – Szabolcs Mar 8 '14 at 13:48
• @Szabolcs Interesting thing is that the Q and each A here got so many upvotes in such a short time. My answer isn't even a full answer. Very suspicious. – István Zachar Mar 8 '14 at 15:08

This behaviour has changed since that book was published. I am writing this additional answer to make it clear how Mathematica 9 searches contexts for symbols and that even the current version 9 documentation is incorrect in describing this.

When you enter a symbol name such as x, Mathematica will check if a symbol with this name already exists. It will first search the contexts from $ContextPath for x, one by one. If it doesn't find it there, it'll search the context from $Context for it. If it still doesn't find it, then it will create a new symbol named x in $Context. Thus $ContextPath controls where to look for symbols, while $Context controls where to create new symbols. Your observations are explained by these rules, noting that Begin will change $Context only but not $ContextPath. Note that BeginPackage will change both $Context and $ContextPath. Warning: the documentation contains an error in Mathematica versions older than 11.3. The $ContextPath documentation states that

$ContextPath is a global variable that gives a list of contexts, after $Context, to search in trying to find a symbol that has been entered.

In fact $ContextPath is searched before $Context in the current version.

In old versions this was not the case, as the Wagner book describes. I don't know when the change happened.

The Contexts tutorial does correctly state the order of search in the current version:

Since $Context is searched after $ContextPath, you can think of it as having "." appended to the file search path.

• @Kuba Is everybody sure that this is really just one more documentation bug? Maybe it is not. I at least find the documented behaviour and Wagner's description more intuitive. Someone (@Daniel Lichtblau ?) from WRI should react. – Rolf Mertig Apr 8 '15 at 8:19
• @RolfMertig : Danny Lichtblau and I are looking into this. – Stefan R Apr 8 '15 at 19:37
• @Mr.Wizard It looks like the behavior was intentionally changed in V6, but the documentation was not properly updated. It has been reported internally but not fixed so far. – Stefan R Jul 14 '15 at 19:39
• @StefanR Still present in v.11. Can you prioritize fixing this documentation? It is extremely confusing, and the same bug exists in the Contexts tutorial reference.wolfram.com/language/tutorial/Contexts.html (In the first table.) It also deserves a place of prominence is "what's new in Mma 6", but I could not find it there at all. Thanks. – Alan Sep 13 '16 at 19:56
• This bug was just brought to my attention today--it is unfortunate that the original developer ignored the report. I have done the minimal fix of getting the main usage right. It looks to me like this page needs some love, so I'll add it to my queue of reference pages in need of review. And just to confirm: this was intentionally changed in V6. The wording was updated in several other places but missed here. – Itai Seggev Aug 24 '17 at 3:22

Not an answer. Just screen shot. I booted up version 2.2 to verify the book result. And it does verify:

The same result we get with version 5.2:

Something changed since the book was written (screenshot from version 9 follows):

It appears since version 3 shadowed variables are given priority, as demonstrated below. In the temp context x is taken as tempx unless Globalx exists.

Remove[tempx, Globalx]
tempx = 6;
Begin["temp"];
{x, Globalx}
End[];


{6, x}

Remove[tempx, Globalx]
tempx = 6;
x = 5;
Begin["temp"];
{x, Globalx}
End[];


{5, 5}

• Isn't what happens that contexts from $ContextPath are searched for names before $Context? (While it is still true that new symbols are created in $Context.) Then the documentation of $ContextPath is simply incorrect because it says: "$ContextPath is a global variable that gives a list of contexts, after $Context, to search in trying to find a symbol that has been entered." – Szabolcs Mar 4 '14 at 16:09
• (This is the same thing that you are saying, put in a different way.) – Szabolcs Mar 4 '14 at 16:13
• BeginPackage ... EndPackage puts temp  at the beginning of the $ContextPath but Begin ... End does not change the $ContextPath. – Chris Degnen Mar 4 '14 at 16:49

This is because of $ContextPath. If the symbol outside the Begin/End is created in a context that is listed in $ContextPath, Mathematica will recognize it as belonging to that context even when called from a different context (even if Begin/End is put in a package file and called via Get):

$ContextPath  {"DocumentationSearch", "ResourceLocator", "PacletManager", "QuantityUnits", "WebServices", "System", "Global"}  WebServicesx; Begin["temp"]; Print@Context@x; End[];   WebServices  Now if I remove "WebServices" from $ContextPath:

$ContextPath = DeleteCases[$ContextPath, "WebServices"];
WebServicesx;
Begin["temp"];
Print@Context@x;
End[];

temp


Now what has changed in related code in successive versions, I have no idea.

It seems that when the book appeared, the behaviour of how a symbol is resolved was different. We have two important things:

• the current $Context which is usually Global unless you change it with e.g. Begin as you did • the $ContextPath which is a list of contexts that are searched when you type in a symbol like x without explicit context

Now there seems to be confusion how a symbol is found. The documentation to $ContextPath reads $ContextPath is a global variable that gives a list of contexts, after $Context, to search in trying to find a symbol that has been entered. This suggests that it is first checked whether a symbol is in the current $Context and after that all entries in $ContextPath are checked. This is how it seemed to work when the book was written and it is completely understandable that you are confused why it behaves differently. In this tutorial it is written the other way around Since $Context is searched after $ContextPath, you can think of it as having "." appended to the file search path. Therefore, let me explain your code as I understand what happens. x=5 tempx=6; {x,tempx}  You create two symbols in different contexts, the first one in Global and the last one in temp. As I understand it, the x in the last line is the global x because the context Global is in $ContextPath rather than because your current $Context is Global Begin["temp"] {x,Globalx} {tempx,Globalx} End[]  With the Begin you set $Context to temp, but that doesn't matter, because when you refer to x, it is found first in the Global context which is an element of your $ContextPath. The rest should be clear, because when you specify the context of a symbol explicitly, there is no room for guessing. Finally, please note that you can refer to the symbol x in the current context (meaning in $Context) by using x and therefore

Begin["temp"]
x
End[]


will give 6.

Appendix

I haven't seen that almost the same question was already asked and has an answer which goes along what I wrote. Another important aspect was mentioned there that I haven't said: $Context controls where a new symbol is created (if there couldn't be found an existing one). Please read the following: Result about Context is inconsistent with the description of “Power Programming with Mathematica” • The documentation is not accurate about how symbol lookup works – Szabolcs Apr 8 '15 at 0:25 • I believe $Context controls where symbols are created, but it doesn't control where existing symbols are searched for. That's controlled by $ContextPath. – Szabolcs Apr 8 '15 at 0:26 • @Szabolcs If this were true, how is tempx=1 found when I call x inside Begin["temp"]? Import["http://goo.gl/NaH6rM"]["http://i.stack.imgur.com/97JkJ.png"] – halirutan Apr 8 '15 at 0:31 • @Szabolcs Nevertheless, really embarrassing that I typed the whole answer without searching whether it already existed.. – halirutan Apr 8 '15 at 0:34 • I was sloppy in my comment, see my answer for the full story. If you enter a symbol name without a context, Mma first looks for it in the contexts from $ContextPath. If it's not there, then it also looks in $Context. If it's not there, it creates it in $Context. The documentation is incorrect in that it it claims that $Context is searched before $ContextPath while it's searched only afterwards. In older versions it was the reverse (at least in v2.2 according to Nasser). – Szabolcs Apr 8 '15 at 0:34

(this is wiki answer, as just noticed it is duplicate, but will keep it here for easy reference)

The result is different from Version 2.2 and version 10.1, this is just to show the difference. Something changed between 1993 and today:

• Ahh I see the duplication, perhaps it's best to merge the answers? The complete descriptions given here are really insightful. Usually I can see a potential duplication when typing the question, but I just missed the referenced one? – tarhawk Apr 8 '15 at 11:31

I think the following also fits the present discussion.

When you run

(*cleanup*)
$Context = "Global"; Quiet@Remove["temp*", "Global*"]; (*--*) Globalx = "global"; BeginPackage["temp"]; tempx = "temp";  The last line generates the message ...::shdw: Symbol x appears in multiple contexts {temp,Global}; definitions in context temp may shadow or be shadowed by other definitions. >> (*cleanup*)$Context = "Global";
Quiet@Remove["temp*", "hemp*", "Global*"];
(*--*)
BeginPackage["temp"];
tempx = "temp";
Globalx = "global";


generates

...::shdw: Symbol x appears in multiple contexts {Global,temp}; definitions in context Global may shadow or be shadowed by other definitions. >>

It looks like even the developer of this particular message was unsure about what exactly happens here ("may shadow or be shadowed").

The following would more correct in the first case:

...::shdw: Symbol x appears in multiple contexts {temp,Global} on the context path. Definitions in the first context, temp, will shadow any other definitions. >>

The message should be the same in the second case, the order of the contexts should be exactly as they are in Append[$ContextPath,$Context].

Why is this message more correct?

A symbol x can be defined in multiple contexts that are not on the context path just fine, the following generates no error message:

Globalx = "global";
foox = "foo";
barx = "bar";


By the way: You get no error message if you later do AppendTo[$ContextPath, "foo"], even though the shadowing 'ambiguity' problem now exists. The "::shdw" message should really be saying that within the set of contexts Append[$ContextPath, $Context] ("the context path") there exist two or more symbols with SymbolName x. The original message is not clear about which definition will be given precedence, although the system knows this unambiguously: When an unqualified symbol x appears, it will be searched in Append[$ContextPath, $Context] in-order (it will be created in $Context if it cannot be found).

BeginPackage prepends the mentioned context to the context path, such that its definitions will take precedence.

• Yes, but I think the message reflects the confusion about $ContextPath and $Context and which is given priority. – masterxilo Jul 3 '16 at 2:51