How to force Mathematica to clean up the cache

Question

(see the bottom of this question for the MWE)

I'm working with a set of numerical data: numerous (a few thousands) files in the style of list of vectors (quadruplets x, y, vx, vy). Files are named as v1.txt, v2.txt, ..., vk.txt, where k is a few thousands. I am using For loop in Mathematica, in each cycle several steps are performed:

1) load vi.txt
2) make list of pairs of pairs from it (in the form of {{x1, y1},{v1x, v1y}, ...})
3) construct absolute value of the vector field ({{x1, y1}, norm of {v1x, v1y}, ...})
4) Make list density plot of the absolute value list
5) Make list stream plot of the pairs of pairs
6) export both.

The problem is that after about 200 cycles, Mathematica sucks up all the physical memory available in my machine and the kernel crashes. I tried using:

1) $HistoryLength = 0
2) ClearSystemCache[] after every cycle
3) ClearAll[(plots, data)] after every cycle
4) all of the above combined

up to no avail. What am I doing wrong? What should I do to tell Mathematica to get rid of the images from the previous step, so it won't suck up all of my RAM? The code I'm using is as follows:

$HistoryLength = 0;
SetDirectory[NotebookDirectory[]];
Monitor[
 For[i = 1, i <= 5383, i++,
  v = Partition[
    Partition[
     Flatten@Import["output\\results\\v" <> ToString[i] <> ".txt", 
       "Table"], 2], 2];
  Absv = Table[{v[[i, 1, 1]], v[[i, 1, 2]], Norm[v[[i, 2]]]}, {i, 1, 
     Length@v}];
  LDP = ListDensityPlot[Absv, PlotRange -> Full, 
    ColorFunction -> ColorData[{"SolarColors", "Reverse"}], 
    MaxPlotPoints -> 100, ImageSize -> 1000];
  LSP = ListStreamPlot[v, StreamStyle -> White, StreamPoints -> Fine, 
    StreamScale -> 0.1, ImageSize -> 1000];
  Export["output\\results\\img" <> ToString[i] <> ".png", LDP];
  Export["output\\results\\img_stream" <> ToString[i] <> ".png", Show[LDP, LSP]];
  ClearSystemCache[];
  ClearAll[v,Absv,LDP,LSP];
  ], i]

Why is Mathematica completely ignoring all statements $HistoryLength, ClearSystemCache and ClearAll?

MWE

This is a minimal working example demonstrating my problem (not depending on any mysterious data that I have on my HDD):

MemoryInUse[]
Do[
  ListStreamPlot@
   Table[{{Random[], Random[]}, {Random[], Random[]}}, 1000];
  ClearSystemCache[],
  {i, 10}
  ];
MemoryInUse[]

Together with Henrik we figured out ListStreamPlot is the culprit, therefore I reported this as a bug.

Answer 1

This is also not an answer (I think you found the culprit and it needs to be solved by WRI) but a suggestion for a workaround. The idea is to run the memory leaking code in an extra kernel and restart that kernel every once in a while. Using the parallel functionality of current Mathematica releases this is pretty simple and by choosing an appropriate value for the number of parallel kernels you could even take advantage of parallelism if desired.

Here is a simple example based on your minimal working example which demonstrates the idea and shows that the master kernel will not accumulate any memory:

LaunchKernels[1];
Print[MemoryInUse[]];
calcsPerKernel = 3;
Do[
   With[{fname = "plot-" <> ToString[i] <> ".png"},
     ParallelEvaluate[
       Export[
         FileNameJoin[{$HomeDirectory, "Desktop", fname}],
         ListStreamPlot[
           Table[{{Random[], Random[]}, {Random[], Random[]}}, 1000]
         ]
       ];
       Print[MemoryInUse[]];
     ]
   ];
   If[Mod[i, calcsPerKernel] == 0,
     CloseKernels[];
     ClearSystemCache[];
     LaunchKernels[1];
   ];
   ,
  {i, 10}
];
CloseKernels[];
ClearSystemCache[];
Print[MemoryInUse[]];

The idea for a more general and flexible approach is as follows:

• use LaunchKernels to launch the desired number of parallel kernels
• use ParallelSubmit to pass the computations (e.g. per file) to the parallel kernel(s)
• use WaitNext to get the result of a computation from the other kernel(s).
• use CloseKernels[kernelid] and LaunchKernels to restart a/the parallel kernels

As it is very difficult to completely avoid memory leaks in sufficient complex Mathematica code I think this is a technique that will be helpful in many situations. We are using this approach regularly for computations that run for several days and have found that it works very reliable. Unlike some other parts of Mathematica the parallel toolkit seems to work very stable and reliable even on a larger scale.

Answer 2

Not an answer, just extended clarification

There seem to be many types of caching related leaks here.

I can get one by just doing this:

$HistoryLength = 0;
Do[Table[RandomReal[], 251], 200]
ClearSystemCache[];
MemoryInUse[]

Calling that repeatedly shows a steady increase with brief pauses.

This doesn't cause a leak:

Do[Table[RandomReal[], 249], 200]
ClearSystemCache[];
MemoryInUse[]

Which shows that something is leaking in the internal auto-compilation.

These are yet distinct from what happens if you apply ListPlot to that. This causes a leak every call:

Do[ListPlot@Table[RandomReal[], 251], 5]
ClearSystemCache[];
MemoryInUse[]

While this continues not to leak:

Do[ListPlot@Table[RandomReal[], 249], 5]
ClearSystemCache[];
MemoryInUse[]

On the other hand if we use ListStreamPlot the leaks seem to be at top-level as this:

Do[ListStreamPlot@
    Table[RandomReal[{}, {2, 2}], 2];, 2];
ClearSystemCache[];
MemoryInUse[]

Leaks every single time