# Parallel evaluation and memory use

I have never used Mathematica's parallel evaluation before and have some trouble getting it to work properly. For the record, I am running Mathematica 9 on Linux with an Intel Core i7 processor (4 physical cores).

I have a function Foo that performs randomised numerical calculations with some matrices and returns a small list. I am interested in running it multiple times and collecting the results, so if I use Table on one kernel, everything works fine and I experience no issues. I then attempt to parralelise it with something like:

LaunchKernels[2];
DistributeDefinitions[Foo];

ParallelTable[Foo[bar], {10000}];


Two kernels work fine as well, but in that case Mathematica starts to use up to 10GB of RAM. Running four kernels, however, causes a lot more problems. Mathematica quickly uses up all of my 16GB of RAM leaving the computer literally unusable, complains about running out of memory all the time and keeps shutting down kernels. Even if I get my results, I am often unable to use them because any operation on the returned list causes the shutdown of the main kernel and the loss of all data. MaxMemoryUsed[] gives 37 686 808.

I have tried alternative approaches with ParallelDo and some stranger constructs (motived by wanting to be able to stop the calculations and retain all the results obtained so far) such as

a = {};
Do[
AppendTo[a, ParallelTable[Foo[bar], {100}]],
{100}]


but these cause an even faster kernel failure due to the lack of memory.

I should also mention that after I stop using 4 kernels and go back to 1-kernel calculations, I frequently run into some random "No more memory available. Mathematica kernel has shut down. Try quitting other applications and then retry." even though Mathematica is not even using much of my system memory at this point. This does not happen prior to attempting 4-kernel evaluation in the same session.

Am I approaching this problem incorrectly? Resources on the internet about Mathematica's parallel evaluation are sparse, so I was unable to find much information. One thread here on StackExchange mentioned that calling ClearSystemCache[] in the function can free up some memory used by Mathematica in parallel computations, but it seems to have no effect at all.

I would appreciate any information and help. Thank you.

• Do you have \$HistoryLength set to 0? (On the remote and master kernel). – Ymareth Jan 31 '14 at 9:46
• Which process is taking up memory? The main kernel or the subkernels? It is expected that if you use four subkernels, they'll use four times the memory. But if you see extreme memory usage in the main kernel only, and only when parallelizing (ParallelTable) but not otherwise (plain Table), then it's likely you're running into this issue/bug. If you can confirm that that is indeed the issue, consider complaining to support @ wolfram, maybe they'll fix it finally ... – Szabolcs Jan 31 '14 at 18:28