How to un-eat memory?

Question

Consider this code:

MemoryInUse[]
T = Table[RandomComplex[], {i, 1, 6000}, {j, 1, 6000}];
MemoryInUse[]
T += T\[ConjugateTranspose];
MemoryInUse[]
{Es, Ys} = Eigensystem[T];
MemoryInUse[]
T = Table[RandomComplex[], {i, 1, 6000}, {j, 1, 6000}];
MemoryInUse[]
T += T\[ConjugateTranspose];
MemoryInUse[]
{Es, Ys} = Eigensystem[T];
MemoryInUse[]
$HistoryLength = 0;
MemoryInUse[]
Clear[T]
MemoryInUse[]
Clear[Es, Ys]
MemoryInUse[]
ClearSystemCache[]
MemoryInUse[]

It gives me the following results:

15808208

880820520

1456822832

4919500424

5783503032

6359505096

9822181440

9822182648

9822182112

9822181384

9822162952

Clearly, the memory clears negligibly on any of ClearSystemCache, Clear and zeroing $HistoryLength. Repeating its execution leads to swapping, after start of which I hurry up to kill MathKernel before my X or WM or anything else are OOM-killed.

So what are the working ways to release the memory?

Answer 1

$HistoryLength is just a global variable. It not clear the history. You can still access the history by

ByteCount@Out[12]
ByteCount@%10

It can be cleared by

Unprotect[In, Out]
Clear[In, Out]

However, it would be better if you set $HistoryLength=0 before your resource-intensive code.

P.S. It would be great to have $HistoryMemoryLimit or something like this.

Answer 2

Here is a method I've prepared for clearing memory between function iterations. It's a working model although I haven't needed to use it yet. It fully quits memory between each iteration. Each section needs to be in a separate cell to work. The second cell runs a four-cell loop for as many iterations as specified. The functionToEvaluate[n] should be user-specified, and would be limited to not doing things that would break the loop.

Note, this code modifies a $FrontEnd option which persists across sessions, so if you experiment with it you may want to check the default value True is set when you're done. The last section does that.

(* Setup section *)
numberOfIterations = 4;
initialOptionState = Options[$FrontEnd,
"ClearEvaluationQueueOnKernelQuit"][[1, -1]];
SetOptions[$FrontEnd,"ClearEvaluationQueueOnKernelQuit" -> False]
(* initialise temporary log file *)
Put[{initialOptionState, numberOfIterations, MaxMemoryUsed[]}, 
 FileNameJoin@Append[FileNameSplit@$TemporaryDirectory, "log.m"]]
(* The log keeps a record of the initial option status, 
numberOfIterations and memory use. *)
ReadList[FileNameJoin@Append[FileNameSplit@$TemporaryDirectory, "log.m"]]

(* Evaluate only this cell to launch the iteration procedure *)
nb = EvaluationNotebook[];
SelectionMove[nb, Next, Cell];
Do[FrontEndExecute[FrontEndToken[nb, "SelectNextLine"]], {2}];
SelectionEvaluate[nb]

Quit[];

iteration = Length[log = ReadList[
     FileNameJoin@Append[FileNameSplit@$TemporaryDirectory, "log.m"]]];
numberOfIterations = log[[1, 2]];
PutAppend[{DateList[], MemoryInUse[], MaxMemoryUsed[]}, 
 FileNameJoin@Append[FileNameSplit@$TemporaryDirectory, "log.m"]];
(* Define or Get the function to be iterated *)
functionToEvaluate[n_] :=(*Print[n]*)n^2
functionToEvaluate[iteration];

If[iteration < numberOfIterations,
 nb = EvaluationNotebook[];
 SelectionMove[nb, Previous, Cell, 4];
 SelectionEvaluate[nb],
 initialOptionState = log[[1, 1]];
 SetOptions[$FrontEnd, 
  "ClearEvaluationQueueOnKernelQuit" -> initialOptionState];
 Print[iteration, " interations completed."]]

Print["This cell does not get evaluated by the loop."]

(* Viewing the log *)
Grid[ReadList[FileNameJoin@
   Append[FileNameSplit@$TemporaryDirectory, "log.m"]]]

(* Check option has been reset to former value *)
Options[$FrontEnd, "ClearEvaluationQueueOnKernelQuit"]

(* Use this to reset option, if required (for instance if loop is interrupted). *)
SetOptions[$FrontEnd, "ClearEvaluationQueueOnKernelQuit" -> True]

I have noticed the option's state sometimes get set counter to my expectations, most probably due to the quitting and resetting, so sometimes the loop only runs once. I haven't completely finished figuring that out yet, but if you have suggestions I'd be interested to hear. In practice, once it's looping it can be left to run. Code improvement suggestions welcome.

Another approach to running fresh kernels would be to use one kernel to run another repeatedly, as demonstrated here.