Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I have never used Mathematica's parallel evaluation before and have some trouble getting it to work properly. For the record, I am running Mathematica 9 on Linux with an Intel Core i7 processor (4 physical cores).

I have a function Foo that performs randomised numerical calculations with some matrices and returns a small list. I am interested in running it multiple times and collecting the results, so if I use Table on one kernel, everything works fine and I experience no issues. I then attempt to parralelise it with something like:

LaunchKernels[2];
DistributeDefinitions[Foo];

ParallelTable[Foo[bar], {10000}];

Two kernels work fine as well, but in that case Mathematica starts to use up to 10GB of RAM. Running four kernels, however, causes a lot more problems. Mathematica quickly uses up all of my 16GB of RAM leaving the computer literally unusable, complains about running out of memory all the time and keeps shutting down kernels. Even if I get my results, I am often unable to use them because any operation on the returned list causes the shutdown of the main kernel and the loss of all data. MaxMemoryUsed[] gives 37 686 808.

I have tried alternative approaches with ParallelDo and some stranger constructs (motived by wanting to be able to stop the calculations and retain all the results obtained so far) such as

a = {};
Do[
 AppendTo[a, ParallelTable[Foo[bar], {100}]],
{100}]

but these cause an even faster kernel failure due to the lack of memory.

I should also mention that after I stop using 4 kernels and go back to 1-kernel calculations, I frequently run into some random "No more memory available. Mathematica kernel has shut down. Try quitting other applications and then retry." even though Mathematica is not even using much of my system memory at this point. This does not happen prior to attempting 4-kernel evaluation in the same session.

Am I approaching this problem incorrectly? Resources on the internet about Mathematica's parallel evaluation are sparse, so I was unable to find much information. One thread here on StackExchange mentioned that calling ClearSystemCache[] in the function can free up some memory used by Mathematica in parallel computations, but it seems to have no effect at all.

I would appreciate any information and help. Thank you.

share|improve this question
    
Do you have $HistoryLength set to 0? (On the remote and master kernel). –  Ymareth Jan 31 at 9:46
1  
Which process is taking up memory? The main kernel or the subkernels? It is expected that if you use four subkernels, they'll use four times the memory. But if you see extreme memory usage in the main kernel only, and only when parallelizing (ParallelTable) but not otherwise (plain Table), then it's likely you're running into this issue/bug. If you can confirm that that is indeed the issue, consider complaining to support @ wolfram, maybe they'll fix it finally ... –  Szabolcs Jan 31 at 18:28

1 Answer 1

On a regular basis i have subkernels eating up my Memory. Still have not a great solution however if finished anyway (Or between two ParallelTables) I handle this problem via: CloseKernels[];LaunchKernels[];

This frees the memory of the subkernels.

share|improve this answer
    
Is there a question here? –  rasher Apr 15 at 4:57

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.