2
$\begingroup$

I am having some problems with parallel computations. I'm afraid it may be a "known issue" but I haven't found any solution.

In general, my script looks like this ("prepare" and "process" are my own functions, of course):

LaunchKernels[];
Needs["MyTools`"]; (* contains my "process" function *)
ParallelNeeds["MyTools`"]; (* all parallel kernels get it, too *)
output = {};
Do[input = prepare[i]; (* "input" is just a list of some matrices *)
   AppendTo[output, ParallelSubmit[{i, input}, {i, process[input]}]],
   {i, 1, 1000000}]; (* several hundred thousands to a million *)
WaitAll[output];

The problem is that the Do loop takes a very long time (10 to 20 hours or so) during which no single available "parallel kernel" is used (i.e. the ParallelSubmit calls do not "spawn" any calculations and first the WaitAll call triggers "parallel computations").

So, I thought that I could use:

LaunchKernels[];
Needs["MyTools`"]; (* contains my "process" function *)
ParallelNeeds["MyTools`"]; (* all parallel kernels get it, too *)
output = {};
WaitAll[
  Do[input = prepare[i]; (* "input" is just a list of some matrices *)
     AppendTo[output, ParallelSubmit[{i, input}, {i, process[input]}]],
     {i, 1, 1000000}]]; (* several hundred thousands to a million *)

Unfortunately, ParallelSubmit still does not trigger an immediate start of "parallel computations".

So, I am looking for a way to start "parallel computations" from inside of the Do loop and then wait for the completion of all "spawned" cases after the loop finishes.

Could you, please, advice me how to improve it. Thanks in advance.

Update (2019.08.14): No matter what I tried, I had not been able to convince ParallelSubmit to immediately start ("submitted") calculations. Then, Mathematica ran amok when I tried (it increasingly started to acquire tens of GB RAM for each running kernel and, at any moment, only two kernels were really using CPU):

output = {}; SetSharedVariable[output];
ParallelDo[input = prepare[i]; AppendTo[output, {i, process[input]}],
  {i, 1, 1000000}];

However, I was able to solve my problem by using ParallelTable (all available "parallel kernels" are used immediately). I realize that this is not a "general purpose" solution but, it may help someone so, I show it here:

output = ParallelTable[input = prepare[i]; {i, process[input]},
  {i, 1, 1000000}, Method -> "FinestGrained"];
$\endgroup$
  • 2
    $\begingroup$ One should distribute the definitions of all the related functions to all the subkernals by using DistributeDefinitions, and use ParallelSubmit[{x1___},expr] to substitute the current values of the x1.. into expr before submitting it for evaluation. $\endgroup$ – Wen Chern Aug 9 at 5:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.