6
$\begingroup$

I would like to use Mathematica to speed-up a parallel calculation.

I envisage each sub-process producing a list of 4 million machine reals. My problem is that sometimes the sub-process (based around NDSolve) fails and the list is shorter than expected.

When this happens I would like to reject the list and re-try the calculation using slightly different initial conditions. What I need is an arrangement where I can submit the individual calculation to sub-process and have Mathematica wait for any sub-process to complete and then test if it really reached the end of the calculation.

Is there a general "submit & wait" parallel programming mechanism in Mathematica?

$\endgroup$
6
$\begingroup$

This is my way, based on ParallelSubmit and WaitNext.

Here is a task that may take long time to finish and fails 50% of the time. We want to return not only the result, but also metadata that would allow us call the task again, analyze performance and have a decent log of what's happening.

task[s_] := Block[
  {
   p = RandomInteger[9],
   success
  },
  Pause[p]; (* adds some dramatism *)
  success = RandomChoice[ {True, False}];
  <|
   "parameter" -> s,
   "success" -> success,
   "kernel" -> $KernelID,
   "wait" -> p,
   "result" -> If[success, 42, $Failed]
   (* Answer to the Ultimate Question of... *)
   |>
  ]

The following function checks a list of EvaluationObject and resubmit a task if it isn't a success. If the queue is not empty, it retrieves the next task using WaitNext and logs the result. Then, if the return is considered a success, the value is saved into resultList . Otherwise, the task is submitted again using ParallelSubmit. The queue of tasks is updated accordingly.

check[] := Block[
  {return, evalobj, rest},
  If[
   Length[eids] > 0,
   {return, evalobj, rest} = WaitNext[eids];
   AppendTo[taskLog, return];
   If[return["success"],
    AppendTo[resultList, return["result"]], (* Success *)
    AppendTo[rest,                          (* Failure *)   
      With[
       {par = return["parameter"]},
       ParallelSubmit[task[par]]            (* Submit again *)
       ]];
    ];
   eids = rest;                             (* Update queue *)
   ]
  ]

Define empty lists for results and logs, launch kernels if necessary and distribute the definition of task.

eids = {};
resultList = {};
taskLog = {};
LaunchKernels[];
DistributeDefinitions[task]

Then we submit all the tasks using Table and ParallelSubmit. Notice we need to use With so the parameter value is given correctly to the task function.

eids = Table[
   With[
    {par = k},
    ParallelSubmit[ task[par] ]
    ]
   , {k, 20}
];

And keep checking for results until the list of evaluations tasks empties.

While[Length[eids] > 0, check[]]

See how the queue empties, 4 kernels working at the time, failed tasks are resubmitted and added at the end of the queue.

enter image description here

The evaluation log shows all attempts, successes and failures

enter image description here

$\endgroup$
  • 1
    $\begingroup$ Well I've modified rhermans' code to use my routines and I'm delighted to report it works flawlessly. Many thanks to you rhermans. $\endgroup$ – Donald Burns Aug 16 at 9:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.