It is important to understand that in Mathematica, semi-automatic parallelization, such as the one provided by ParallelTable
or ParallelMap
works only if the chunks of code executed in parallel have no side effects.
What does it really mean to say that f
has no side effect? Another way to say it is that the output of f
depends only on it's input and nothing else. In other words, f
has no internal state that influences its result.
Here's a function that does have side effects:
c=0;
f[x_] := x + (c++)
Evaluating f
twice with the same input gives different results:
f[2]
(* 2 *)
f[2]
(* 3 *)
The output does not only depend on the input, but also on some (possibly private-to-f) state that is preserved between subsequent calls to f
, in this case c
.
Functions like f
above are simply not suitable for parallelization in Mathematica. Yes, this is a limitation in some sense, but it's also a great advantage in that side-effect-less functions are really easy to parallelize.
A comment on the order of evaluations:
As your colleague notes, the order of evaluations may affect the results for functions with side effects:
c = 0;
KeySort@AssociationMap[f, {1, 2, 3}]
(* <|1 -> 1, 2 -> 3, 3 -> 5|> *)
c = 0;
KeySort@AssociationMap[f, {3, 2, 1}]
(* <|1 -> 3, 2 -> 3, 3 -> 3|> *)
It also goes the other way: if the order of evaluations does affect the result, that tells me that his function must have had side effects.
He believed that parallelization fails because the order of evaluations is unpredictable. This is not correct. Parallelization fails much worse than that because f
's internal state (the variable c
) exists as a separate and independent copy for each parallel thread. The different threads are not al modifying a global c
, they're instead modifying their own private local c
. To put it plainly: the results will be rubbish.
The golden rule to keep in mind: code evaluated in parallel must never have any side effects, i.e. it must not maintain an internal state between subsequent evaluations.
ParallelMap
andParallelTable
are supposed to do some post-processing to hide this from the user, but otherwise order is never guaranteed across multiple processes. $\endgroup$