(see the bottom of this question for the MWE)
I'm working with a set of numerical data: numerous (a few thousands) files in the style of list of vectors (quadruplets x, y, vx, vy). Files are named as v1.txt, v2.txt, ..., vk.txt, where k is a few thousands. I am using For loop in Mathematica, in each cycle several steps are performed:
1) load vi.txt
2) make list of pairs of pairs from it (in the form of {{x1, y1},{v1x, v1y}, ...})
3) construct absolute value of the vector field ({{x1, y1}, norm of {v1x, v1y}, ...})
4) Make list density plot of the absolute value list
5) Make list stream plot of the pairs of pairs
6) export both.
The problem is that after about 200 cycles, Mathematica sucks up all the physical memory available in my machine and the kernel crashes. I tried using:
1) $HistoryLength = 0
2) ClearSystemCache[] after every cycle
3) ClearAll[(plots, data)] after every cycle
4) all of the above combined
up to no avail. What am I doing wrong? What should I do to tell Mathematica to get rid of the images from the previous step, so it won't suck up all of my RAM? The code I'm using is as follows:
$HistoryLength = 0;
SetDirectory[NotebookDirectory[]];
Monitor[
For[i = 1, i <= 5383, i++,
v = Partition[
Partition[
Flatten@Import["output\\results\\v" <> ToString[i] <> ".txt",
"Table"], 2], 2];
Absv = Table[{v[[i, 1, 1]], v[[i, 1, 2]], Norm[v[[i, 2]]]}, {i, 1,
Length@v}];
LDP = ListDensityPlot[Absv, PlotRange -> Full,
ColorFunction -> ColorData[{"SolarColors", "Reverse"}],
MaxPlotPoints -> 100, ImageSize -> 1000];
LSP = ListStreamPlot[v, StreamStyle -> White, StreamPoints -> Fine,
StreamScale -> 0.1, ImageSize -> 1000];
Export["output\\results\\img" <> ToString[i] <> ".png", LDP];
Export["output\\results\\img_stream" <> ToString[i] <> ".png", Show[LDP, LSP]];
ClearSystemCache[];
ClearAll[v,Absv,LDP,LSP];
], i]
Why is Mathematica completely ignoring all statements $HistoryLength
, ClearSystemCache
and ClearAll
?
MWE
This is a minimal working example demonstrating my problem (not depending on any mysterious data that I have on my HDD):
MemoryInUse[]
Do[
ListStreamPlot@
Table[{{Random[], Random[]}, {Random[], Random[]}}, 1000];
ClearSystemCache[],
{i, 10}
];
MemoryInUse[]
Together with Henrik we figured out ListStreamPlot
is the culprit, therefore I reported this as a bug.
Module
and scope all temporary variables within, so that they get cleared by automatic garbage collection. $\endgroup$ – Henrik Schumacher Mar 4 '18 at 0:44Import
then. You could play with it by using different export filters (instead of"Table"
) such as"Text"
,"Data"
,"CSV"
what not. Maybe you find one that uses a different, hopefully leak-free backend... Thinking about it: Can you also try to strip theMonitor
?TheseDynamic
things also feel a bit suspicious. You can useIf[Mod[i,100]==0,Print[i]];
instead for monitoring. $\endgroup$ – Henrik Schumacher Mar 4 '18 at 1:16ListStreamPlot
stuff with justListPlot@Table[RandomReal[], 250]; ClearSystemCache[]; MemoryInUse[]
which is coming from some faulty caching of the compiledTable
asListPlot@Table[RandomReal[], 249]; ClearSystemCache[]; MemoryInUse[]
doesn't do this $\endgroup$ – b3m2a1 Mar 4 '18 at 3:02