Why does leaving and re-entering a context lose values defined in that context?

Question

I'm getting unexpected behavior when beginning a new context, leaving it, and then re-entering it. Specifically, when re-entering the context, symbols that had been previously defined in that context now seem to have lost their values... but I can recapture them by explicitly supplying the context. Consider this example:

Begin["sandbox`"];
a = 1;
ValueQ[a]
End[];

(* ==> True *)

ValueQ[a]

(* ==> False *)

Begin["sandbox`"];
Names[Evaluate[Context[]] <> "*"]
ValueQ[a]
ValueQ[sandbox`a]
End[];

(* ==> {"sandbox`a"} *)
(* ==> False *)
(* ==> True *)

NB: Quit the Mathematica kernel before evaluating this example, every time. Re-evaluating the whole thing subsequent times changes the results, so that all of the ValueQ tests now return True. (I also have absolutly no idea why that happens.)

My questions are: Why doesn't switching back into the "sandbox`" context give me un-prefixed access to symbols in that context, and is there any way to work around this that doesn't require explicitly providing the context? (In my real application, always specifying the context once it has been left and re-entered isn't practical.)

FYI, the reason I'm investigating this is that I'm trying to set up a kind of clean-ish "sandbox" context where some unknown code can be evaluated, and then test code I've written can inspect the definitions/values/etc. that it leaves behind. Because of the way the application is factored, its hard to avoid leaving and re-entering the sandbox context.

Answer 1

The definitions aren't being lost, they're being shadowed, as described in the tutorial on contexts. Mathematica doesn't warn you about this because it only warns when there is shadowing between contexts that are listed in the $ContextPath. Since Begin only changes $Context and not $ContextPath, you don't get a warning when the symbol that causes shadowing is created.

Here's a pared-down example. (Quit the kernel before running each code block.)

If we define a symbol in a context and then switch to the context, we get the defined value:

sandbox`a = 1;
$Context = "sandbox`";
Context[a]
(* sandbox` *)

But if a has been created in the Global` context, Global`a is used, because Mathematica looks at the contexts in $ContextPath before $Context:

a;
sandbox`a = 1;
$Context = "sandbox`";
Context[a]
(* Global` *)

No shadowing warning is issued here, because sandbox` is not in the $ContextPath.

If we add sandbox` to $ContextPath, we get the shadowing warning, and a is found in whichever context comes first in the list:

PrependTo[$ContextPath, "sandbox`"];
a;
sandbox`a = 1;
Context[a]

(* During evaluation of In[1]:= a::shdw: Symbol a appears in multiple 
   contexts {sandbox`,Global`}; definitions in context sandbox` may shadow or 
   be shadowed by other definitions. >> *)
(* ==> sandbox` *)

The standard ways of dealing with shadowing are:

• If you are only interested in sandbox`a and inadvertently create Global`a, use Remove to remove Global`a:

  Begin["sandbox`"];
  a = 1;
  End[];
  
  a;
  Remove[a]
  
  Begin["sandbox`"];
  a
  End[];
  
  (* ==> 1 *)
  

• If you want to keep both Global`a and sandbox`a but refer by default to sandbox`a, use BeginPackage instead of Begin, which adds the context to $ContextPath:

  a = 2;
  
  BeginPackage["sandbox`"];
  a = 1;
  EndPackage[];
  
  a
  Global`a
  
  (* During evaluation of In[1]:= a::shdw: Symbol a appears in multiple contexts {sandbox`,Global`}; definitions in context sandbox` may shadow or be shadowed by other definitions. >> *)
  (* ==> 1 *)
  (* ==> 2 *)
  

• If you want to keep both Global`a and sandbox`a and refer by default to Global`a, use Begin as you have been doing.

For your purposes, however, you may want to manually manipulate $ContextPath, so you can enter and leave your sandbox. You can define

EnterSandbox[] := BeginPackage["sandbox`"];
sandbox`ExitSandbox[] := (
  EndPackage[]; 
  $ContextPath = DeleteCases[$ContextPath, "sandbox`"]
)

Then you can do:

EnterSandbox[];
a = 1;
ExitSandbox[];

a = 2;

EnterSandbox[];
a
ExitSandbox[];

a 

(* 1 *)
(* 2 *)

Answer 2

Indeed confusing, but can be explained. Please read here first on how Mathematica searches contexts for symbols.

In short,

• $Context tells it where to create symbols. It's for creation, not for lookup.

• $ContextPath tells it where to look for symbols but doesn't affect symbol creation.

  If the symbol is not found in any of the $ContextPath contexts, Mathematica proceeds to create it in $Context. For this, it first needs to check if $Context already contains this name. If it does, it won't need to create it after all. Thus lookup will effectively fall back to $Context if (and only if) the symbol is not found in any $ContextPath contexts.

To add to the confusion, Begin only sets $Context, but not the $ContextPath. Compare BeginPackage which also sets the $ContextPath.

Looking at your example:

Begin["sandbox`"];
a = 1;  (* sandbox`a is created here *)
ValueQ[a] (* there's no symbol named 'a' in any of the $ContextPath contexts so Mathematica falls back to looking in $Context *)
End[];

(* ==> True *)

ValueQ[a] (* Global`a is created here *)

(* ==> False *)

Begin["sandbox`"];
Names[Evaluate[Context[]] <> "*"]
ValueQ[a] (* this refers to Global`a since that symbol already exists and Global` is in $ContextPath *)
ValueQ[sandbox`a]
End[];

(* ==> {"sandbox`a"} *)
(* ==> False *)
(* ==> True *)

I hope this clears it up.