There are multiple issues here. Let us take them one by one.
Why do you get 0
as the output?
There is a quirk in how DistributeDefinition
works. To me, this looks like a bug ... Let me show it through a smaller example that isolates the problem.
Let's define a function in a fresh kernel using Set
instead of SetDelayed
, then distribute its definitions, and finally check what definitions are present on the subkernels.
Quit[]
foo[a_] = a;
DistributeDefinitions[foo];
ParallelEvaluate[Print@Definition["foo"]]
foo[a_]=a
foo[a_]=a
foo[a_]=a
foo[a_]=a
Everything is as one would expect it. But now let's tweak the code a bit ...
Quit[]
foo[a_] = a;
a = 0; (* set a value for 'a' before distributing foo's definition *)
DistributeDefinitions[foo];
What's the definition on the main kernel?
Definition[foo]
foo[a_] = a
What's the definition on the subkernels?
foo[a_]=0
foo[a_]=0
foo[a_]=0
foo[a_]=0
Oops! That's now what I expected, and it's not what it should be: it's not the same as on the main kernel. a
got evaluated during the distribution process.
I see the same behaviour in all versions between 9.0–11.0, and I find it very disturbing. There are legitimate reasons for using Set
instead of SetDelayed
, and there are legitimate reasons for using the resulting function in parallelized code. Therefore I would consider this a bug.
What should you do then?
If you don't need to, don't use =
. Use :=
instead, which prevents the RHS from evaluating during distribution.
If you need to use =
, don't define a global a
.
If you both need to use =
and define a global a
, then manually evaluate DistributeDefinitions[foo]
and LaunchKernels[]
before you set the global a
; or evaluate Block[{a}, DistributeDefinitions[foo]; LaunchKernels[]]
to temporarily unset a
during distribution. The point is not to allow ParallelDo
or other parallel functions do the distribution automatically—preemptively distribute them in advance.
Be sure to launch the kernels right after distributing the definitions. DistributeDefinitions
only registers a symbol for distribution. Actual distribution happens only upon kernel launch. Unfortunately, kernel launch is hard to control and may be triggered by mayny things. For example, kernels that die for some reason get automatically re-launched mid-computation. For this reason, this third workaround isn't fully robust. Better stick to the first two whenever you can.
Obviously, these are only workarounds.
Does Module
work with parallel functions?
This works:
Module[{a = 1},
ParallelDo[Print[a], {4}]
]
Each kernel on my machine prints 1
. But please do not do this because it effectively creates a memory leak. Module
"localizes" symbols through renaming.
Each evaluation of the Module
creates a new symbol with a name like a$123
. These symbols are removed immediately when a Module
exits. But if we use them in a parallel function, they get automatically copied (distributed) to all subkernels, and remain there. Therefore this code effectively has a memory leak. After three runs of the code, I see this on the subkernels:
ParallelEvaluate[Information["Global`a$*"]]
a$2695 a$4410 a$4421
a$2695 a$4410 a$4421
a$2695 a$4410 a$4421
a$2695 a$4410 a$4421
For similar reasons, also do not use Block
this way. It will mess up the bookkeeping information about distributed symbols.
If Block
and Module
variables shouldn't be used in parallel functions, what can I use instead?
The usual solution would be to inline the value of the symbol. This can be done using With
.
With[{a=1},
ParallelDo[foo[a], ...]
]
It replaces a
with its value in ParallelDo
, so ParallelDo
never even sees the symbol a
(only its value). This approach does not have a performance hit because the value needs to be copied to each subkernel either way. Subkernels are separate processes, and do not share any memory. Things like SetSharedVariable
simulate shared memory by always evaluating and setting that variable on the main kernel. This has the drawback that all those callbacks can reduce performance significantly. It is only worth using this if you know that these callbacks still take less time than the evaluation of foo
.
a
in the code, does usingWith
instead ofModule
work? $\endgroup$tmpMat
is not shared between parallel kernels, so you can't modify it. You can make it shared (SetSharedVariable
), but that tends to destroy any speed benefits brought by parallelization. Effective parallelization in Mathematica requires that the parallel threads not try to access the same data at the same time. Why don't you just useParallelTable
, or better:ParallelMap
, to compute the elements first, then assign them to a matrix only after the computation is done. $\endgroup$foo
takes extra long to evaluate". AboutWith
vsModule
: you can always place a smallerWith
inside of aModule
. I can't at this moment see any reason that would truly prevent you from usingWith
. Please explain, preferably through an example, if you disagree. $\endgroup$Module
this way is a very very bad idea. Why? Because every run of your program will distribute (DistributeDefinitions
) a different localized version ofa
(i.e.a$1
,a$2
, ...) to the parallel kernels. These never get cleared. You effectively created a memory leak. Always keep in mind that parallel threads are actually completely separate processes, and any data sharing must be explicit at some level. Never use aModule
variable inside of aParallel*
function this way. $\endgroup$