# Parallel Computation with the possibility of calculation failure

I would like to use Mathematica to speed-up a parallel calculation.

I envisage each sub-process producing a list of 4 million machine reals. My problem is that sometimes the sub-process (based around NDSolve) fails and the list is shorter than expected.

When this happens I would like to reject the list and re-try the calculation using slightly different initial conditions. What I need is an arrangement where I can submit the individual calculation to sub-process and have Mathematica wait for any sub-process to complete and then test if it really reached the end of the calculation.

Is there a general "submit & wait" parallel programming mechanism in Mathematica?

• reference.wolfram.com/language/guide/Concurrency.html – Henrik Schumacher Jul 18 '19 at 13:43
• A combination of ParallelSubmit and WaitNext. – rhermans Jul 18 '19 at 13:45
• Many thanks rhermans for a detailed and complete answer. I've not done parallel programming before and so it may take a little time to get to grips with your code. – SkyCat Jul 24 '19 at 9:37

This is my way, based on ParallelSubmit and WaitNext.

Here is a task that may take long time to finish and fails 50% of the time. We want to return not only the result, but also metadata that would allow us call the task again, analyze performance and have a decent log of what's happening.

task[s_] := Block[
{
p = RandomInteger[9],
success
},
Pause[p]; (* adds some dramatism *)
success = RandomChoice[ {True, False}];
<|
"parameter" -> s,
"success" -> success,
"kernel" -> $$KernelID, "wait" -> p, "result" -> If[success, 42,$$Failed]
(* Answer to the Ultimate Question of... *)
|>
]


The following function checks a list of EvaluationObject and resubmit a task if it isn't a success. If the queue is not empty, it retrieves the next task using WaitNext and logs the result. Then, if the return is considered a success, the value is saved into resultList . Otherwise, the task is submitted again using ParallelSubmit. The queue of tasks is updated accordingly.

check[] := Block[
{return, evalobj, rest},
If[
Length[eids] > 0,
{return, evalobj, rest} = WaitNext[eids];
If[return["success"],
AppendTo[resultList, return["result"]], (* Success *)
AppendTo[rest,                          (* Failure *)
With[
{par = return["parameter"]},
ParallelSubmit[task[par]]            (* Submit again *)
]];
];
eids = rest;                             (* Update queue *)
]
]


Define empty lists for results and logs, launch kernels if necessary and distribute the definition of task.

eids = {};
resultList = {};
LaunchKernels[];


Then we submit all the tasks using Table and ParallelSubmit. Notice we need to use With so the parameter value is given correctly to the task function.

eids = Table[
With[
{par = k},
]
, {k, 20}
];


And keep checking for results until the list of evaluations tasks empties.

While[Length[eids] > 0, check[]]


See how the queue empties, 4 kernels working at the time, failed tasks are resubmitted and added at the end of the queue.

The evaluation log shows all attempts, successes and failures

• Well I've modified rhermans' code to use my routines and I'm delighted to report it works flawlessly. Many thanks to you rhermans. – SkyCat Aug 16 '19 at 9:51