# Memory isolation autochecker

I'm working on a project in which I'm putting everything in modules to make sure I can run multiple instances without them interfering with each other, however I'd like to have it automatically let me know if there are any variables that aren't defined as local to a module.

e.g.

DynamicModule[
{a,b,c}
a=5;
b=6;
c=7;
d=8
//Some more code here
]


What code could I put in the "Some more code here" section or after the module that would let me know that d is not defined as being local?

• You can put everything in packages having own namespaces. – yarchik Dec 8 '20 at 9:35
• Syntax colouring can be very helpful. Restart the kernel, issue all necessary function definitions and global variable assignments, then skim through the code: black is defined, blue is undefined and green/teal is local. There should be nothing that looks blue. d in your sample code would be blue. Note that the sample code is wrong: you are missing a comma and // does not start a comment. Fix these and try it. – Szabolcs Dec 8 '20 at 10:46
• Unfortunately I can't use packages as I need to deploy this out to the consumers as a single file. – Daniel Dec 9 '20 at 3:05
• Syntax coloring helps, but with the size of this file I would rather have a way to have it print out anything that is not localized so I can go easily find it and localize it rather than scrolling and hunting to check everything. – Daniel Dec 9 '20 at 3:06

## Introduction

It is an interesting problem if one tries to solve it using Mathematica's meta-programming capabilities. One idea which I have tried to implement is to process Module and DynamicModule code, lexically, to identify assignments to variables which have not been localized.

What I ended up doing was to generate extra code that would issue a message for such assignments, but otherwise keep them unchanged - this seems to be the least intrusive method. The code below is however only a proof of concept, and while I believe it covers a large portion of cases, it does not cover all possible ways to mutate a variable (in particular, it does not cover UpValues and only partially covers SubValues).

The code I ended up with is rather long, to the extent that I was hesitating to provide it all here instead of putting it elsewhere and linking, but decided to still provide it here to keep the answer self-contained.

## Illustration

In what follows, I assume that that code has been already executed first.

### Some examples to try

First, enable the "debug" regime (by default it will only track symbols from "Global" context):

setBoundVarChecks[]


Then, for example, try

Module[{a = 1, b = 2},
c = 3;
a + b + c
]

During evaluation of In[156]:= customSet::unbound: Assignment c = 3 to an unbound symbol c. The bound symbols at this level are {a$$9690,b$$9690}

(* 6 *)


whereas

Module[{c},
Module[{a = 1, b = 2},
c = 3;
a + b + c
]
]

(* 6 *)


runs without warnings.

You can try many other combinations, with both Module and DynamicModule, and different types of assignments. Some examples:

Module[{a = 1, b = 2}, c := 3; a + b + c]

Module[{a = 1, b = 2}, c++; a + b + c]

d = {}; Module[{a = 1, b = 2}, AppendTo[d, 3];a + b + c]

Module[{a = 1, b = 2}, e[1][3] = 15; a + b + c]

DynamicModule[{a, b, c}, a = 5; b = 6; c = 7; d = 8]

DynamicModule[{a, b, c},
a = 5;
b = 6;
c = 7;
Button["Press", d = 8; Print[d]]
]


where in the last example, the warning message will be printed to the Messages window when the button is pressed.

When done with this "debug" regime, call

unsetBoundVarChecks[]


### How it works

Module[{a = 1}, b = 2; a + b]


where variable b is non-local to the Module. There are two options then: it is either local to some surrounding Module, or it is global.

Assuming for the moment the first scenario, we process the above code as:

processModuleCore[Module[{a = 1}, b = 2; a + b]]

(*
Hold[Module[
{a=1},
Message[customSet::unbound,"b = 2", HoldForm[b], HoldForm[{a}]];
$dummy; b=2; a+b ]] *)  Let me explain in some detail how processModuleCore works. The following will detect inner assignments and replace them all with some custom assignment operator customSet: rc = replaceAssignments[][Hold @ Module[{a = 1}, b = 2; a + b]] (* Hold[customHold[Module @@ Hold[{a = 1}, customSet[b, 2, {a}, Set]; a + b]]] *)  (note that we wrapper the Module in Hold). The following will process the previous result, and generate the new assignment code: generateAssignmentCode @ rc (* Hold[customHold[ Module @@ Hold[ {a=1}, customHold[Message[customSet::unbound,b = 2,b,{a}];($dummy;b=2)];a+b
]
]]

*)


Some postprocessing produces the final new code for Module:

flattenCE @removeCustomHold @ generateAssignmentCode @ rc

(*
Hold[Module[
{a=1},
Message[customSet::unbound,"b = 2", HoldForm[b], HoldForm[{a}]];
$dummy; b=2; a+b ]] *)  The new lines that have been generated are a line with a warning message, and the $dummy symbol (which will only be needed to keep track that we have processed this Module).

Assuming that the variable b has been localized in some outer Module, we would do instead

flattenCE @removeCustomHold @ generateAssignmentCode @ replaceAssignments[b][
Hold @ Module[{a = 1}, b = 2; a + b]
]

(* Hold[Module[{a = 1}, $dummy; b = 2; a + b]] *)  where b has now been passed into replaceAssignments, and the end result of that has been that the message is no longer present in the generated code. Now we may want to attach a global rule which would replace all Module-s and DynamicModule- s in code by the result of processCodeCore called on them. This is admittedly a dangerous step, since the only way to do this is to add DownValues to Module and DynamicModule. So I would only suggest to do this for testing, and while I have tested on a few less trivial examples, I can't guarantee that it would always work well. The top-level functions which automate this step are setBoundVarChecks[contextsToCheck: {___String} | Automatic : Automatic] unsetBoundVarChecks[]  where the first one adds relevant definitions to Module and DynamicModule, and the second one removes them. ## Summary I have outlined one approach to the problem, which is based on generation of some additional code in the form of warning messages, at run-time during code execution. Note that it overloads Module and DynamicModule globally, and therefore is dangerous and only at best suited for debugging purposes. Note also that the code as written does not cover all ways in which a variable can be mutated, so this is rather a proof of concept, not a complete robust solution. That said, it may still be useful. ## Code $$includedContexts = {"Global"}$$excludedContexts = {"System"} ClearAll[customSet,$CustomNone]
SetAttributes[customSet, HoldAll]
customSet::unbound = "Assignment 1 to an unbound symbol 2. The bound symbols at this level are 3";

customSet[set_, lhs_, $$CustomNone] := ($$dummy;set[lhs])

customSet[set_, lhs_, rhs_] := ($dummy;set[lhs, rhs]) customSet[lhs: sym_Symbol | sym_Symbol[___] | sym_Symbol[___][___], rhs_, {___, sym_, ___}, set_] := customSet[set, lhs, rhs] customSet[ lhs: sym_Symbol | sym_Symbol[___] | sym_Symbol[___][___], rhs_, _, set_ ] /; !MemberQ[Complement[$$includedContexts,$$excludedContexts], Context[sym]] := customSet[set, lhs, rhs] customSet[lhs: sym_Symbol | sym_Symbol[___] | sym_Symbol[___][___], rhs_, vars_, set_] := With[{str = ToString @ ReplaceAll[ HoldForm[customSet[set, lhs, rhs]] /. DownValues[customSet], HoldPattern[$dummy; c_] :> c
]},
(
Message[customSet::unbound, str, HoldForm[sym], HoldForm[vars]];
customSet[set, lhs, rhs]
) /; True
]

customSet[lhs_, rhs_, _, set_] := customSet[set, lhs, rhs]

ClearAll[constructNeedsProcessingQ];
SetAttributes[constructNeedsProcessingQ, HoldAll];
constructNeedsProcessingQ[construct_] :=FreeQ[Unevaluated @ construct, $dummy] ClearAll[customHold]; SetAttributes[customHold, HoldAll] ClearAll[getHeldLocalizedVariables] SetAttributes[getHeldLocalizedVariables, HoldAll] getHeldLocalizedVariables[ expr: (head: Module | DynamicModule | With | Block)[{vars___}, __], inheritedVars___Symbol ] := Replace[ Replace[ head, { Module | DynamicModule :> Hold[inheritedVars, vars], With | Block :> Hold[vars] } ], HoldPattern[var_ = _] :> var, {1} ] ClearAll[setReplaceInModuleScope] SetAttributes[setReplaceInModuleScope, HoldAll]; setReplaceInModuleScope[ expr: (head: Module | DynamicModule | With | Block) [{vars___}, body_, rest___], topVars___ ] := Replace[ Apply[ replaceAssignments, getHeldLocalizedVariables[expr, topVars] ][Hold[body]], Hold[newbody_] :> customHold[head @@ Hold[{vars}, newbody, rest]] ] ClearAll[generateAssignmentCode] SetAttributes[generateAssignmentCode, HoldAll] generateAssignmentCode[set_customSet] := customHold[set] //. DownValues[customSet] generateAssignmentCode[expr_] := ReplaceRepeated[ expr, assignment_customSet :> RuleCondition @ generateAssignmentCode[assignment] ] ClearAll[replaceAssignments]; SetAttributes[replaceAssignments, HoldAll]; replaceAssignments[vars___] := ReplaceAll[ { HoldPattern[m:_DynamicModule|_Module|_With|_Block] :> RuleCondition @ setReplaceInModuleScope[m, vars] , HoldPattern[(set: Set | SetDelayed | AppendTo | PrependTo | AddTo | AssociateTo)[lhs_, rhs_]] :> customSet[lhs, rhs, {vars}, set] , HoldPattern[(set: Increment | PreIncrement | Decrement | PreDecrement)[val_]] :> customSet[val,$CustomNone, {vars}, set]
,
c : HoldPattern[customSet[_, _, {___, vars}, _]] :> c
,
HoldPattern[customSet[lhs_, rhs_, {prev__}, set_]] :> customSet[lhs, rhs, {prev, vars}, set]
}
]

ClearAll[removeCustomHold];
removeCustomHold = ReplaceRepeated[
{
customHold[x___] :> x
}
]

ClearAll[flattenCE]
flattenCE =
ReplaceRepeated[
HoldPattern[CompoundExpression[left___, CompoundExpression[middle___], right___]] :>
CompoundExpression[left, middle, right]
]

ClearAll[processModuleCore];
SetAttributes[processModuleCore, HoldAll]
processModuleCore[d:_Module | _DynamicModule] :=
Composition[
flattenCE,
removeCustomHold,
generateAssignmentCode
] @ Hold[Evaluate[setReplaceInModuleScope[d]]]

ClearAll[processModule];
SetAttributes[processModule, HoldFirst];
processModule[d_, context_String] := processModule[d, {context}]

processModule[d:_Module | _DynamicModule, contextsToCheck: {___String} | Automatic : Automatic] :=
Replace[
{processModuleCore[d], contextsToCheck},
{
{processed_, Automatic} :> processed,
{Hold[code_], c_} :> Hold[Block[{$includedContexts = c}, code]] } ] ClearAll[alterProtectedDownValues] alterProtectedDownValues[sym_, func_] := ( Unprotect[sym]; DownValues[sym] = func[DownValues[sym]]; Protect[sym] ) ClearAll[setModuleCheck, unsetModuleCheck, $$inConstruct]; setModuleCheck[construct_Symbol, contextsToCheck: {___String} : Automatic] := ( unsetModuleCheck[construct]; alterProtectedDownValues[ construct, Prepend[ d_construct?constructNeedsProcessingQ /; !TrueQ[$$inConstruct] :> Block[ {$inConstruct = True},
ReleaseHold @ processModule[d, contextsToCheck]
]
]
]
)

unsetModuleCheck[construct_Symbol] := alterProtectedDownValues[
construct,
DeleteCases[def_ /; !FreeQ[def, $inConstruct]] ] ClearAll[setBoundVarChecks, unsetBoundVarChecks] setBoundVarChecks[contextsToCheck: {___String} | Automatic : Automatic] := Scan[setModuleCheck, {Module, DynamicModule}] unsetBoundVarChecks[] := Scan[unsetModuleCheck, {Module, DynamicModule}]  • I'm afraid I am having an incredibly difficult time understanding this, but I greatly appreciate it. I think I may be able to make some use of this given enough time. – Daniel Dec 9 '20 at 6:33 • @Daniel Well, one can use this code also without deep understanding of the internals. If you execute is, then when you call setBoundVarChecks[], this turns on the "debug" mode where all assignments to / mutations of non-localized variables inside Module or DynamicModule will issue a message. To return back to normal code execution regime, just call unsetBoundVarChecks []. That's basically it, on the "user" level. – Leonid Shifrin Dec 9 '20 at 15:19 • Oh thanks! This is incredible, and so easy to use. It even works on my nested Dynamic Modules – Daniel Dec 9 '20 at 19:39 • @Daniel Glad you found it helpful. Thanks for the accept. Keep in mind the caveats I mentioned - there may be cases the current code will miss (probably not hard to account for them though - if you come across any, ping me here in comments and I will try to address them). – Leonid Shifrin Dec 9 '20 at 20:12 Here’s a bit of a hack, based on Module assigning unique names to localized variables that contain “$” and a number:

DynamicModule[
{a},
Echo@ StringContainsQ[SymbolName[a], "$$"]; Echo@ StringContainsQ[SymbolName[d], "$$"]
]


The above will return True for a that has been localized, but False for global variable d. The Echo is just for visualization here; you would probably use the Boolean output from StringContainsQ directly.

Of course, this relies on you avoiding $ in your global variable names (probably a good idea regardless), and on Module‘s renaming behavior not changing in the future. Hence the reason why I think this is a bit of a hack. • This isn't really what I want, this is what I'm trying to avoid. I have so many variables that I don't want to have to go through and double check every single variable name to ensure that it has been localized. I'd like to have some script run that tells me if there are any variables defined at any point that are not localized. – Daniel Dec 8 '20 at 5:01 • @Daniel that may be very hard to do. You might be interested in e.g. Names[“Global*”] but even there quite a few unexpected things will show up... perhaps more than point, why do you need to know? There might be alternative approaches... – MarcoB Dec 8 '20 at 5:30 • @Daniel, you can try to look for a list of variables that does not have a$-sign in the name: Complement[Names["Global*"], Names["Global*\$*"]] – Rom38 Dec 8 '20 at 5:58

At the start of your work, you could take a snapshot of the variables in "Global":

old = Names["Global*"];


Later, when you want to check if there are no new global variables, you take another snapshot and calculate the complement:

new = Names["Global*"];
Complement[new, old]

• This is not correct. Module[{a=0}, 1] creates the symbol a in the Global context. It does not assign any value to it, but it creates the symbol. – Szabolcs Dec 8 '20 at 10:43
• Try:f[x_]= Module[{a}, x]; Names["Global*"]; I can't see any "a" – Daniel Huber Dec 8 '20 at 11:30
• That's because of the ; after the Names call :-) You suppressed the output by accident. – Szabolcs Dec 8 '20 at 12:14
• It is so easy to fool oneself! – Daniel Huber Dec 8 '20 at 14:30
• It looks like this will work for me after some modifications, thanks! – Daniel Dec 9 '20 at 3:56