12
$\begingroup$

I have tried the following code in Mathematica 11.0.1.0 on my MacBook:

$HistoryLength = 0;
ClearAll["Global`*"];
ClearSystemCache[];
MemoryInUse[]

AbsoluteTiming[Do[SquaresR[2, i], {i, 1000000}];]

ClearAll["Global`*"];
ClearSystemCache[];
MemoryInUse[]

There seems to be much more memory in use at the end than at the beginning. I assume the values SquaresR[2,i] (or some related things) are saved, because when I run this a second time, it runs much faster. Is there a way to free this memory, without restarting the kernel?

$\endgroup$
  • 3
    $\begingroup$ At a glance I would consider it a bug or at least oversight that ClearSystemCache[] does not clear these values. $\endgroup$ – Mr.Wizard Jan 19 '17 at 18:45
18
$\begingroup$

Cause

Under the hood System`SquaresR is still calling functions in the context NumberTheory`.

Partial output of:

Needs["GeneralUtilities`"]

PrintDefinitions @ SquaresR
SquaresR[2, NumberTheory`SquaresRDump`n _ Integer?Positive] := 
  Block[{NumberTheory`SquaresRDump`res}, 
   NumberTheory`SquaresRDump`res = 
    NumberTheory`SquaresRDump`squaresR2[NumberTheory`SquaresRDump`n]; 
   NumberTheory`SquaresRDump`res /; IntegerQ[NumberTheory`SquaresRDump`res]];

So all these calls to SquaresR[2, i] and handed off to NumberTheory`SquaresRDump`squaresR2.

PrintDefinitions @ NumberTheory`SquaresRDump`squaresR2
(* contexts omitted *)

squaresR2[0] := 1;

squaresR2[1] := 4;

squaresR2[n_] := 
  Block[{res, fact}, 
   fact = FactorInteger[
     n]; (res = 
      If[MemberQ[fact, {p_ /; Mod[p, 4] == 3, _?OddQ}], 0, 
       4*Times @@ Cases[fact, {p_ /; Mod[p, 4] == 1, a_} :> a + 1]]; 
     squaresR2[n] = res) /; FreeQ[fact, $Failed | FactorInteger]];

squaresR2[___] := $Failed;

Note the command squaresR2[n] = res -- this is memoization (caching).

Solution

We can clear all these memoized values and reload the SquaresR package as follows:

ClearAll["NumberTheory`SquaresRDump`*"];

Get[System`Private`$SystemFileDir <> 
  System`Dump`fixfile["NumberTheory`SquaresR`"] <> "x"]

Commentary

There used to be a more clear distinction between core functions and those in standard packages. Core functions were typically faster, often being at least partially implemented in C, and better integrated with things like ClearSystemCache. Standard package functions were slower, being written in high-level Mathematica code, and one would not expect them to be entirely integrated into the system.

In recent versions however standard packages have been deprecated and replaced by functions in the System` context. At first one would think these are core functions but in many cases (e.g. SquaresR) they are not. I find this less desirable as one can no longer expect a System` function to be fast or well integrated. So in this case I would not consider the "failure" of ClearSystemCache to be a flaw, but rather I consider it a flaw in the recent namespace organization of Mathematica.

$\endgroup$
  • $\begingroup$ With the revision of the RT, I have a feeling that things will get more clear in the future (but this is just a feeling). At the same time, we all know that the language will have to, at a given moment, embrace the open trends. And having a clear division between the core and the extremely big package (the other 98%...), is, in my opinion, a fundamental step (but this is just an opinion based on a feeling...). $\endgroup$ – P. Fonseca Jan 20 '17 at 14:13
  • $\begingroup$ @P.Fonseca To what do you refer with the revision of the RT? You touch on something I have long hoped for: the separation of the core language into a stand-alone free distribution, hopefully with new refinements and improvements to performance and perhaps design. The huge function library would then become the commercial product built on top of this language. Is that what you hope for too? $\endgroup$ – Mr.Wizard Jan 20 '17 at 20:08
  • 1
    $\begingroup$ That's it. Take a look at this youtu.be/pzjhFF0wJiw. I have also heard that they are touching the pattern matching artillery. A lot of this is for certainly motivated by the new compiler. I really hope that we will have this on M12 (that should be something like 2018...). This means that everything that is core is being cleaned up, and that everything we can compile within the RT scope will most likely get close to any other "natively" achievable performance and robustness... $\endgroup$ – P. Fonseca Jan 21 '17 at 14:06
  • $\begingroup$ ...Then, we have to think on the following question: is the RT library an income competitor of Mathematica? If it isn't (and I don't think it can be), there's the unique opportunity to get to the 21st century business model, while keeping the financial independency of the 80's! $\endgroup$ – P. Fonseca Jan 21 '17 at 14:06
  • $\begingroup$ @P.Fonseca A very interesting and encouraging presentation; thank you. $\endgroup$ – Mr.Wizard Jan 26 '17 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.