Parallel evaluation and memory use

Question

I have never used Mathematica's parallel evaluation before and have some trouble getting it to work properly. For the record, I am running Mathematica 9 on Linux with an Intel Core i7 processor (4 physical cores).

I have a function Foo that performs randomised numerical calculations with some matrices and returns a small list. I am interested in running it multiple times and collecting the results, so if I use Table on one kernel, everything works fine and I experience no issues. I then attempt to parralelise it with something like:

LaunchKernels[2];
DistributeDefinitions[Foo];

ParallelTable[Foo[bar], {10000}];

Two kernels work fine as well, but in that case Mathematica starts to use up to 10GB of RAM. Running four kernels, however, causes a lot more problems. Mathematica quickly uses up all of my 16GB of RAM leaving the computer literally unusable, complains about running out of memory all the time and keeps shutting down kernels. Even if I get my results, I am often unable to use them because any operation on the returned list causes the shutdown of the main kernel and the loss of all data. MaxMemoryUsed[] gives 37 686 808.

I have tried alternative approaches with ParallelDo and some stranger constructs (motived by wanting to be able to stop the calculations and retain all the results obtained so far) such as

a = {};
Do[
 AppendTo[a, ParallelTable[Foo[bar], {100}]],
{100}]

but these cause an even faster kernel failure due to the lack of memory.

I should also mention that after I stop using 4 kernels and go back to 1-kernel calculations, I frequently run into some random "No more memory available. Mathematica kernel has shut down. Try quitting other applications and then retry." even though Mathematica is not even using much of my system memory at this point. This does not happen prior to attempting 4-kernel evaluation in the same session.

Am I approaching this problem incorrectly? Resources on the internet about Mathematica's parallel evaluation are sparse, so I was unable to find much information. One thread here on StackExchange mentioned that calling ClearSystemCache[] in the function can free up some memory used by Mathematica in parallel computations, but it seems to have no effect at all.

I would appreciate any information and help. Thank you.

Answer 1

On a regular basis i have subkernels eating up my Memory. Still have not a great solution however if finished anyway (Or between two ParallelTables) I handle this problem via: CloseKernels[];LaunchKernels[];

This frees the memory of the subkernels.