Parallelization inside a module: Proper use of local variables

Question

I have the following Module which includes a ParallelDo call. Unfortunately, ParallelDo is not using the local variable:

Clear[a];
foo[a_] = a;
nmax = 3;
tmpMat = IdentityMatrix[nmax];
SetSharedVariable[tmpMat]
a = 0;
foo2[a0_] := Module[{a = a0},
  ParallelDo[tmpMat[[n1, n2]] = foo[a], {n1, 1, nmax}, {n2, 1, nmax}
   ];
  tmpMat
  ]

Now foo2[1] returns

{{0, 0, 0}, {0, 0, 0}, {0, 0, 0}}

instead of

{{1, 1, 1}, {1, 1, 1}, {1, 1, 1}}

when Do[] is used. How to change the property of a to make the parallelized code use the local variable a and not the global one which is set to zero?

Answer 1

There are multiple issues here. Let us take them one by one.

Why do you get 0 as the output?

There is a quirk in how DistributeDefinition works. To me, this looks like a bug ... Let me show it through a smaller example that isolates the problem.

Let's define a function in a fresh kernel using Set instead of SetDelayed, then distribute its definitions, and finally check what definitions are present on the subkernels.

Quit[]
foo[a_] = a;
DistributeDefinitions[foo];
ParallelEvaluate[Print@Definition["foo"]]

foo[a_]=a

foo[a_]=a

foo[a_]=a

foo[a_]=a

Everything is as one would expect it. But now let's tweak the code a bit ...

Quit[]
foo[a_] = a;    
a = 0; (* set a value for 'a' before distributing foo's definition *)    
DistributeDefinitions[foo];

What's the definition on the main kernel?

Definition[foo]

foo[a_] = a

What's the definition on the subkernels?

foo[a_]=0

foo[a_]=0

foo[a_]=0

foo[a_]=0

Oops! That's now what I expected, and it's not what it should be: it's not the same as on the main kernel. a got evaluated during the distribution process.

I see the same behaviour in all versions between 9.0–11.0, and I find it very disturbing. There are legitimate reasons for using Set instead of SetDelayed, and there are legitimate reasons for using the resulting function in parallelized code. Therefore I would consider this a bug.

What should you do then?

• If you don't need to, don't use =. Use := instead, which prevents the RHS from evaluating during distribution.

• If you need to use =, don't define a global a.

• If you both need to use = and define a global a, then manually evaluate DistributeDefinitions[foo] and LaunchKernels[] before you set the global a; or evaluate Block[{a}, DistributeDefinitions[foo]; LaunchKernels[]] to temporarily unset a during distribution. The point is not to allow ParallelDo or other parallel functions do the distribution automatically—preemptively distribute them in advance.

  Be sure to launch the kernels right after distributing the definitions. DistributeDefinitions only registers a symbol for distribution. Actual distribution happens only upon kernel launch. Unfortunately, kernel launch is hard to control and may be triggered by mayny things. For example, kernels that die for some reason get automatically re-launched mid-computation. For this reason, this third workaround isn't fully robust. Better stick to the first two whenever you can.

Obviously, these are only workarounds.

Does Module work with parallel functions?

This works:

Module[{a = 1},
 ParallelDo[Print[a], {4}]
]

Each kernel on my machine prints 1. But please do not do this because it effectively creates a memory leak. Module "localizes" symbols through renaming. Each evaluation of the Module creates a new symbol with a name like a$123. These symbols are removed immediately when a Module exits. But if we use them in a parallel function, they get automatically copied (distributed) to all subkernels, and remain there. Therefore this code effectively has a memory leak. After three runs of the code, I see this on the subkernels:

ParallelEvaluate[Information["Global`a$*"]]

a$2695 a$4410 a$4421

a$2695 a$4410 a$4421

a$2695 a$4410 a$4421

a$2695 a$4410 a$4421

For similar reasons, also do not use Block this way. It will mess up the bookkeeping information about distributed symbols.

If Block and Module variables shouldn't be used in parallel functions, what can I use instead?

The usual solution would be to inline the value of the symbol. This can be done using With.

With[{a=1},
  ParallelDo[foo[a], ...]
]

It replaces a with its value in ParallelDo, so ParallelDo never even sees the symbol a (only its value). This approach does not have a performance hit because the value needs to be copied to each subkernel either way. Subkernels are separate processes, and do not share any memory. Things like SetSharedVariable simulate shared memory by always evaluating and setting that variable on the main kernel. This has the drawback that all those callbacks can reduce performance significantly. It is only worth using this if you know that these callbacks still take less time than the evaluation of foo.