14
$\begingroup$

One of the things I noticed is that in typical medium-to-large sized packages, a huge number of symbols are created in the myPackage`Private context. You can see a list of them by running

? myPackage`Private`*

after loading a hypothetical package myPackage. Many of them are without any definitions attached to them, and are created even when they are used only as pattern names/local variable names, etc... in the source code.

So, as an experiment, I decided to take one of my large packages and Remove all symbols in the Private` context that have no definitions attached to them. I did this by adding the line

Remove@@Complement[Names["myPackage`Private`*"], {(*list of symbols to save*)}]

to the very end of the package. (It took me a couple of hours to manually save all the private symbols that do have definitions or are used as global variables). That way, I end up with a tidy list when ? myPackage`Private`* is called.

The package seems to behave normally without any problems. The amusing thing is that Definition of the private symbols that do use the removed symbols (such as pattern names etc...) are now displayed in terms of Removed[a1], Removed[a2], ...

So, my question is this: Is there any benefit to this silly exercise? Does it free up memory? And is there any risk if only those symbols that are not used globally or that do not have definitions are carefully removed?

$\endgroup$
  • 3
    $\begingroup$ Had you ever run into trouble with those extra symbols that you have now removed? If not, then don't fix it if it ain't broke... :-) $\endgroup$ – MarcoB May 1 '16 at 20:32
  • $\begingroup$ I would never do it, even with my own packages. You may break the package in very subtle ways, while the benefits are very questionable. I agree with others who suggested against it, particularly R.M. gave a great answer. I just wanted to emphasize that IMO one shouldn't do this, period. $\endgroup$ – Leonid Shifrin May 1 '16 at 21:15
11
$\begingroup$

As a more direct answer to the question of memory we can check with a test.

In[1]:= MemoryInUse[]

(* Out[1]= 34093104 *)

In[2]:= bunchOfSymbols = Table[Unique[], {26000}];

In[3]:= foo[Sequence @@ bunchOfSymbols] := bar;

In[4]:= MemoryInUse[]

(* Out[4]= 43032256 *)

In[5]:= Remove @@ bunchOfSymbols;

In[6]:= MemoryInUse[]

(* Out[6]= 41577200 *)

This indicates that (in version 10.1 under Windows x64) Removing a Symbol saves only about 56 bytes of memory which seems inconsequential by itself unless one has a truly excessive number of Symbols.

Also I noticed in this test that the Remove step caused temporary freeze in my Front End though I could not capture it with Timing, AbsoluteTiming, etc. This could make loading your package annoying for others if it is distributed. The freeze gets longer when the number of Symbols removed increases, therefore in any case where significant memory would be recovered the freeze would be problematic.

$\endgroup$
  • $\begingroup$ I confirm with my own package that Remove frees up an insignificant amount of memory. I can also reproduce the temporary freeze in the Front End when running your code from a front end notebook. But I am finding that if I run your code from within a package file (.m) then the freeze doesn't appear to happen. Would you kindly check on your system? $\endgroup$ – QuantumDot May 2 '16 at 8:32
  • $\begingroup$ @QuantumDot Hey, thanks for the Accept. Sorry I forgot to reply to your comment but this reminded me. I copied the code from this answer into a .m file, added Print to each MemoryInUse[] line, and called it with Get. My result is that there is no lag but also there is no reduction in memory consumption; actually it goes up slightly, e.g 34196768; 40986040; 41401936. However the second value is significantly less than in the interactive example above. I need to examine this further. $\endgroup$ – Mr.Wizard May 13 '16 at 9:29
  • 1
    $\begingroup$ As I understand this discussion (mathematica.stackexchange.com/questions/4921/…), Remove only clears the link between the symbol name and the data in memory. This probably explains why hardly any memory is freed.... $\endgroup$ – QuantumDot May 28 '16 at 21:04
21
$\begingroup$

I decided to take one of my large packages and Remove all symbols in the Private` context that have no definitions attached to them

...

And is there any risk if only those symbols that are not used globally or that do not have definitions are carefully removed?

In general, the complexity of the Mathematica language and the intricacies of the evaluation sequence, which are not entirely understood even by people very comfortable with Mathematica (myself included) makes it very hard to identify or predict with certainty what symbols don't have any definitions (currently) and will never have any definitions (in the future) attached to them.

As a simple example:

BeginPackage["Example`"]

ClearAll@F;
ClearAll@G;

Begin["`Private`"]

ClearAll@f;
f[x_] := (a := 1;True) /; x > 2
f[x_] := False

F[x_] := f[x^2]
G[x_] := Symbol[x] > 0    

End[]
EndPackage[]

If you load the package, Example`Private`a has no definitions associated with it and is used nowhere else "globally", so it could be removed according to your definition. Yet, if you evaluate the following:

F[5];
G["Example`Private`a"]

one would expect the result to be True, but it would simply be Example`Private`a in your case.

You shouldn't be worrying about "unnecessary" symbols in a hidden context that you are never exposed to. Truly unnecessary symbols may be bad style, but that's on the author of the package, not you. Perhaps the complexity of the package/algorithm might necessitate these extra symbols just to reduce the cognitive overhead of interpreting obscure/terse/fancy point-free style coding 2 years later when you're debugging. Sometimes it might also be a reflection of the author's own personal familiarity/comfort with the language at the time the package was written. In short, these aren't really issues that cause any discomfort to the user of the package.

Instead, what you should be focusing on are performance issues like

  • Do any of the "unnecessary" symbols in the package's private context leak memory?
  • Are any of the symbols in the private context getting exposed accidentally in other contexts?

etc. Those are issues worth investigating and fixing, as they have the potential to cause issues downstream for the users of the package.

$\endgroup$
  • 1
    $\begingroup$ Nothing to add to this, really. A great answer. +1. $\endgroup$ – Leonid Shifrin May 1 '16 at 21:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.