15
$\begingroup$

(see the bottom of this question for the MWE)

I'm working with a set of numerical data: numerous (a few thousands) files in the style of list of vectors (quadruplets x, y, vx, vy). Files are named as v1.txt, v2.txt, ..., vk.txt, where k is a few thousands. I am using For loop in Mathematica, in each cycle several steps are performed:

1) load vi.txt
2) make list of pairs of pairs from it (in the form of {{x1, y1},{v1x, v1y}, ...})
3) construct absolute value of the vector field ({{x1, y1}, norm of {v1x, v1y}, ...})
4) Make list density plot of the absolute value list
5) Make list stream plot of the pairs of pairs
6) export both.

The problem is that after about 200 cycles, Mathematica sucks up all the physical memory available in my machine and the kernel crashes. I tried using:

1) $HistoryLength = 0
2) ClearSystemCache[] after every cycle
3) ClearAll[(plots, data)] after every cycle
4) all of the above combined

up to no avail. What am I doing wrong? What should I do to tell Mathematica to get rid of the images from the previous step, so it won't suck up all of my RAM? The code I'm using is as follows:

$HistoryLength = 0;
SetDirectory[NotebookDirectory[]];
Monitor[
 For[i = 1, i <= 5383, i++,
  v = Partition[
    Partition[
     Flatten@Import["output\\results\\v" <> ToString[i] <> ".txt", 
       "Table"], 2], 2];
  Absv = Table[{v[[i, 1, 1]], v[[i, 1, 2]], Norm[v[[i, 2]]]}, {i, 1, 
     Length@v}];
  LDP = ListDensityPlot[Absv, PlotRange -> Full, 
    ColorFunction -> ColorData[{"SolarColors", "Reverse"}], 
    MaxPlotPoints -> 100, ImageSize -> 1000];
  LSP = ListStreamPlot[v, StreamStyle -> White, StreamPoints -> Fine, 
    StreamScale -> 0.1, ImageSize -> 1000];
  Export["output\\results\\img" <> ToString[i] <> ".png", LDP];
  Export["output\\results\\img_stream" <> ToString[i] <> ".png", Show[LDP, LSP]];
  ClearSystemCache[];
  ClearAll[v,Absv,LDP,LSP];
  ], i]

Why is Mathematica completely ignoring all statements $HistoryLength, ClearSystemCache and ClearAll?

MWE

This is a minimal working example demonstrating my problem (not depending on any mysterious data that I have on my HDD):

MemoryInUse[]
Do[
  ListStreamPlot@
   Table[{{Random[], Random[]}, {Random[], Random[]}}, 1000];
  ClearSystemCache[],
  {i, 10}
  ];
MemoryInUse[]

Together with Henrik we figured out ListStreamPlot is the culprit, therefore I reported this as a bug.

$\endgroup$
  • $\begingroup$ Hard to say. The code looks okay to me. You could also try to wrap the body of the loop in a Module and scope all temporary variables within, so that they get cleared by automatic garbage collection. $\endgroup$ – Henrik Schumacher Mar 4 '18 at 0:44
  • $\begingroup$ Maybe one of the built-in functions has a memory leak. You could narrow down the problematic part of the code by commenting out various parts of the loop's body and see if the memory leak persists. Should you find one, I would advise you to contact the support. $\endgroup$ – Henrik Schumacher Mar 4 '18 at 0:53
  • $\begingroup$ @HenrikSchumacher: could you please expand on that Module idea? By looking at the documentation, I don't quite get what I should do inside the loop...Module[{v, Absv, LDP, LSP}, the body of the loop]? This does exactly the same thing as before: my memory eventually leaks. $\endgroup$ – user16320 Mar 4 '18 at 0:53
  • 2
    $\begingroup$ Aha. It's the Import then. You could play with it by using different export filters (instead of "Table") such as "Text", "Data", "CSV" what not. Maybe you find one that uses a different, hopefully leak-free backend... Thinking about it: Can you also try to strip the Monitor?These Dynamic things also feel a bit suspicious. You can use If[Mod[i,100]==0,Print[i]]; instead for monitoring. $\endgroup$ – Henrik Schumacher Mar 4 '18 at 1:16
  • 1
    $\begingroup$ @HenrikSchumacher Note that I can reproduce your MWE replacing all the ListStreamPlot stuff with just ListPlot@Table[RandomReal[], 250]; ClearSystemCache[]; MemoryInUse[] which is coming from some faulty caching of the compiled Table as ListPlot@Table[RandomReal[], 249]; ClearSystemCache[]; MemoryInUse[] doesn't do this $\endgroup$ – b3m2a1 Mar 4 '18 at 3:02
13
$\begingroup$

This is also not an answer (I think you found the culprit and it needs to be solved by WRI) but a suggestion for a workaround. The idea is to run the memory leaking code in an extra kernel and restart that kernel every once in a while. Using the parallel functionality of current Mathematica releases this is pretty simple and by choosing an appropriate value for the number of parallel kernels you could even take advantage of parallelism if desired.

Here is a simple example based on your minimal working example which demonstrates the idea and shows that the master kernel will not accumulate any memory:

LaunchKernels[1];
Print[MemoryInUse[]];
calcsPerKernel = 3;
Do[
   With[{fname = "plot-" <> ToString[i] <> ".png"},
     ParallelEvaluate[
       Export[
         FileNameJoin[{$HomeDirectory, "Desktop", fname}],
         ListStreamPlot[
           Table[{{Random[], Random[]}, {Random[], Random[]}}, 1000]
         ]
       ];
       Print[MemoryInUse[]];
     ]
   ];
   If[Mod[i, calcsPerKernel] == 0,
     CloseKernels[];
     ClearSystemCache[];
     LaunchKernels[1];
   ];
   ,
  {i, 10}
];
CloseKernels[];
ClearSystemCache[];
Print[MemoryInUse[]];

The idea for a more general and flexible approach is as follows:

  • use LaunchKernels to launch the desired number of parallel kernels
  • use ParallelSubmit to pass the computations (e.g. per file) to the parallel kernel(s)
  • use WaitNext to get the result of a computation from the other kernel(s).
  • use CloseKernels[kernelid] and LaunchKernels to restart a/the parallel kernels

As it is very difficult to completely avoid memory leaks in sufficient complex Mathematica code I think this is a technique that will be helpful in many situations. We are using this approach regularly for computations that run for several days and have found that it works very reliable. Unlike some other parts of Mathematica the parallel toolkit seems to work very stable and reliable even on a larger scale.

$\endgroup$
  • $\begingroup$ As you're the only one who have come up with something that works, I'm giving you a thumbs up and I accept this as an answer. Using your wrapper I'm exporting 5 different plots each step and I get no memory leaks (only using about 1GB of memory) - I can see the memory refresh after each step in my task manager. Thanks and keep up the good work! $\endgroup$ – user16320 Mar 5 '18 at 0:35
  • $\begingroup$ as I said, it is just a workaround but it has proven to be useful for me in many cases. Good to hear it works for you and thanks for the accept... $\endgroup$ – Albert Retey Mar 5 '18 at 7:36
  • $\begingroup$ Oh, one more thing: why neither of $HistoryLength, CleanAll, CleanSystemCache work in my example? I thought those are supposed to get rid of unneeded stuff left in memory... $\endgroup$ – user16320 Mar 5 '18 at 11:35
  • $\begingroup$ there are countless possibilities for memory leaks which are not covered by the above. In your case it looks like leaks in the autocompilation code and ListStreamPlot as the other answers demonstrates. There probably is nothing you can do about that despite not using that functionality or wait for WRI to correct it. Because that situation is not so unusual (and that is true when using other systems/libraries as well) I thought the given workaround might be of wider interest. $\endgroup$ – Albert Retey Mar 6 '18 at 8:50
  • $\begingroup$ Thank you for your answer. The workaround you provided is very useful. $\endgroup$ – user16320 Mar 6 '18 at 20:03
9
$\begingroup$

Not an answer, just extended clarification

There seem to be many types of caching related leaks here.

I can get one by just doing this:

$HistoryLength = 0;
Do[Table[RandomReal[], 251], 200]
ClearSystemCache[];
MemoryInUse[]

Calling that repeatedly shows a steady increase with brief pauses.

This doesn't cause a leak:

Do[Table[RandomReal[], 249], 200]
ClearSystemCache[];
MemoryInUse[]

Which shows that something is leaking in the internal auto-compilation.

These are yet distinct from what happens if you apply ListPlot to that. This causes a leak every call:

Do[ListPlot@Table[RandomReal[], 251], 5]
ClearSystemCache[];
MemoryInUse[]

While this continues not to leak:

Do[ListPlot@Table[RandomReal[], 249], 5]
ClearSystemCache[];
MemoryInUse[]

On the other hand if we use ListStreamPlot the leaks seem to be at top-level as this:

Do[ListStreamPlot@
    Table[RandomReal[{}, {2, 2}], 2];, 2];
ClearSystemCache[];
MemoryInUse[]

Leaks every single time

$\endgroup$
  • $\begingroup$ Hm. The first two do not leak on my machine (v. 11.0.1 for macos). But ListStreamPlot@Table[RandomReal[{}, {2, 2}], 2] leaks exactly 132752 Byte each time I execute it. $\endgroup$ – Henrik Schumacher Mar 4 '18 at 7:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.