5
$\begingroup$

I have a code that works over lists like

S0; (*initial list*)
S1=f[S0];
S2=g[S1];
S3=f[S2];
S4=f[S3];
.
.
.

where f and g are functions that produces larger lists that consume a large amount of RAM memory. I thought that running

S0; (*initial list*)
S1=f[S0]; Clear[S0];
S2=g[S1]; Clear[S1];
S3=f[S2]; Clear[S2];
S4=f[S3]; Clear[S3];
.
.
.

would free the memory used from the kernel, but it is not the case. The consumption of both codes are exactly the same. I stored (using Save) the result for S10, for example, and after Get it (with a clear kernel) the memory consumption is much less than running the code, even using Clear in each step as above.

Is there a stronger version of Clear which eliminates the information of a symbol from the kernel? Or does Clear do it, but the task manager keeps showing the same memory usage from before?

$\endgroup$
  • $\begingroup$ One thing to try is $HistoryLength -- change it to something much less than the default. $\endgroup$ – bill s Oct 4 '17 at 22:36
  • $\begingroup$ @bills Can I redefine $HistoryLength later? Like, if I aim to get S10 and then make other operations using it and other things, can I redefine to infinity after the computation of S10? $\endgroup$ – Filburt Oct 4 '17 at 23:13
  • $\begingroup$ @bills And it removes lines from the beginning? Is is possible to keep some definitions before start the string of calculations in the OP? $\endgroup$ – Filburt Oct 4 '17 at 23:23
6
$\begingroup$

As @bills says, you can just set $HistoryLength=0, but that's rather draconian. Instead, just make sure that Out doesn't store your results. To do this, you need to use something like:

S0; "";

instead of something like:

S0; Null;

Let's compare the two:

S0; "";
S1 = f[S0]; ""; Clear[S0];
S2 = g[S1]; ""; Clear[S1];

DownValues[Out][[-3;;]]

{HoldPattern[%56] :> "", HoldPattern[%57] :> "", HoldPattern[%58] :> ""}

versus your:

S0;
S1 = f[S0]; Clear[S0];
S2 = g[S1]; Clear[S1];

DownValues[Out][[-3;;]]

{HoldPattern[%60] :> S0, HoldPattern[%61] :> f[S0], HoldPattern[%62] :> g[f[S0]]}

Notice how the "" version doesn't store any outputs, while your version does. It is the storage of the outputs that prevents you from recovering memory.

Update

A more normal method to avoid memory bloat is to use a Module. For example:

f[x_] := x^2
g[x_] := x+1

func[s_] := Module[{res = s},
    res = f[res];
    res = g[res];
    res = f[res];
    res = f[res];
    Total[res]
]

Now, create a large variable:

MemoryInUse[]
S0 = RandomReal[1,10^8];"";
MemoryInUse[]

151355496

951357168

Note the use of ;""; above so that Out doesn't store the value of S0. Run your function:

MemoryInUse[]
func[S0]
MemoryInUse[]

951358872

4.21584*10^8

951359968

Notice that MemoryInUse has stayed basically the same.

Finally, clear S0 and recover memory:

MemoryInUse[]
Clear[S0]
MemoryInUse[]

951651720

151652648

If we hadn't used ;""; above, the memory wouldn't have gone down.

$\endgroup$
  • $\begingroup$ MemoryInUse[] is showing the same amount (about 500mb in my test) in both cases, "" or Clear. What happened? $\endgroup$ – Filburt Oct 4 '17 at 23:12
  • $\begingroup$ If those symbols are already in Out, then doing the above won't help. You could look at Clean up list of partial outputs for memory and see if the answers there help. Or you could restart your kernel. $\endgroup$ – Carl Woll Oct 4 '17 at 23:27
  • $\begingroup$ Hey man, after that edit, I think your code did the trick :D Why??? $\endgroup$ – Filburt Oct 4 '17 at 23:41
  • 1
    $\begingroup$ Because my original code prevented S1, S2, etc. from being stored in Out, but I hadn't cleared them, so their values still had to be stored in memory. $\endgroup$ – Carl Woll Oct 4 '17 at 23:53
  • 1
    $\begingroup$ You don't need to worry about Out inside a Module, so there is no need for that. $\endgroup$ – Carl Woll Oct 5 '17 at 1:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.