# Parallelization inside a module: Proper use of local variables

I have the following Module which includes a ParallelDo call. Unfortunately, ParallelDo is not using the local variable:

Clear[a];
foo[a_] = a;
nmax = 3;
tmpMat = IdentityMatrix[nmax];
SetSharedVariable[tmpMat]
a = 0;
foo2[a0_] := Module[{a = a0},
ParallelDo[tmpMat[[n1, n2]] = foo[a], {n1, 1, nmax}, {n2, 1, nmax}
];
tmpMat
]


Now foo2[1] returns

{{0, 0, 0}, {0, 0, 0}, {0, 0, 0}}


{{1, 1, 1}, {1, 1, 1}, {1, 1, 1}}


when Do[] is used. How to change the property of a to make the parallelized code use the local variable a and not the global one which is set to zero?

• If you don't need to update a in the code, does using With instead of Module work? – march Feb 8 '17 at 16:46
• There's a bigger problem: tmpMat is not shared between parallel kernels, so you can't modify it. You can make it shared (SetSharedVariable), but that tends to destroy any speed benefits brought by parallelization. Effective parallelization in Mathematica requires that the parallel threads not try to access the same data at the same time. Why don't you just use ParallelTable, or better: ParallelMap, to compute the elements first, then assign them to a matrix only after the computation is done. – Szabolcs Feb 8 '17 at 17:28
• That should be "it will destroy benefits brought by parallelization, unless foo takes extra long to evaluate". About With vs Module: you can always place a smaller With inside of a Module. I can't at this moment see any reason that would truly prevent you from using With. Please explain, preferably through an example, if you disagree. – Szabolcs Feb 8 '17 at 17:31
• Mathematica and C are very different. If you try to program in Mathematica the way you would in C, brace yourself for a lot of frustration. OpenMP uses a shared memory model; Mathematica doesn't. Each model comes with its advantages and disadvantages—for example you cannot use OpenMP across multiple computers. Consider reading up on this before you call it bad design. – Szabolcs Feb 8 '17 at 18:09
• However, using Module this way is a very very bad idea. Why? Because every run of your program will distribute (DistributeDefinitions) a different localized version of a (i.e. a$1, a$2, ...) to the parallel kernels. These never get cleared. You effectively created a memory leak. Always keep in mind that parallel threads are actually completely separate processes, and any data sharing must be explicit at some level. Never use a Module variable inside of a Parallel* function this way. – Szabolcs Feb 8 '17 at 18:15

There are multiple issues here. Let us take them one by one.

### Why do you get 0 as the output?

There is a quirk in how DistributeDefinition works. To me, this looks like a bug ... Let me show it through a smaller example that isolates the problem.

Let's define a function in a fresh kernel using Set instead of SetDelayed, then distribute its definitions, and finally check what definitions are present on the subkernels.

Quit[]
foo[a_] = a;
DistributeDefinitions[foo];
ParallelEvaluate[Print@Definition["foo"]]


foo[a_]=a

foo[a_]=a

foo[a_]=a

foo[a_]=a

Everything is as one would expect it. But now let's tweak the code a bit ...

Quit[]
foo[a_] = a;
a = 0; (* set a value for 'a' before distributing foo's definition *)
DistributeDefinitions[foo];


What's the definition on the main kernel?

Definition[foo]


foo[a_] = a

What's the definition on the subkernels?

foo[a_]=0

foo[a_]=0

foo[a_]=0

foo[a_]=0

Oops! That's now what I expected, and it's not what it should be: it's not the same as on the main kernel. a got evaluated during the distribution process.

I see the same behaviour in all versions between 9.0–11.0, and I find it very disturbing. There are legitimate reasons for using Set instead of SetDelayed, and there are legitimate reasons for using the resulting function in parallelized code. Therefore I would consider this a bug.

What should you do then?

• If you don't need to, don't use =. Use := instead, which prevents the RHS from evaluating during distribution.

• If you need to use =, don't define a global a.

• If you both need to use = and define a global a, then manually evaluate DistributeDefinitions[foo] and LaunchKernels[] before you set the global a; or evaluate Block[{a}, DistributeDefinitions[foo]; LaunchKernels[]] to temporarily unset a during distribution. The point is not to allow ParallelDo or other parallel functions do the distribution automatically—preemptively distribute them in advance.

Be sure to launch the kernels right after distributing the definitions. DistributeDefinitions only registers a symbol for distribution. Actual distribution happens only upon kernel launch. Unfortunately, kernel launch is hard to control and may be triggered by mayny things. For example, kernels that die for some reason get automatically re-launched mid-computation. For this reason, this third workaround isn't fully robust. Better stick to the first two whenever you can.

Obviously, these are only workarounds.

### Does Module work with parallel functions?

This works:

Module[{a = 1},
ParallelDo[Print[a], {4}]
]


Each kernel on my machine prints 1. But please do not do this because it effectively creates a memory leak. Module "localizes" symbols through renaming. Each evaluation of the Module creates a new symbol with a name like a$123. These symbols are removed immediately when a Module exits. But if we use them in a parallel function, they get automatically copied (distributed) to all subkernels, and remain there. Therefore this code effectively has a memory leak. After three runs of the code, I see this on the subkernels: ParallelEvaluate[Information["Globala$*"]]


a$2695 a$4410 a$4421 a$2695 a$4410 a$4421

a$2695 a$4410 a$4421 a$2695 a$4410 a$4421

For similar reasons, also do not use Block this way. It will mess up the bookkeeping information about distributed symbols.

### If Block and Module variables shouldn't be used in parallel functions, what can I use instead?

The usual solution would be to inline the value of the symbol. This can be done using With.

With[{a=1},
ParallelDo[foo[a], ...]
]


It replaces a with its value in ParallelDo, so ParallelDo never even sees the symbol a (only its value). This approach does not have a performance hit because the value needs to be copied to each subkernel either way. Subkernels are separate processes, and do not share any memory. Things like SetSharedVariable simulate shared memory by always evaluating and setting that variable on the main kernel. This has the drawback that all those callbacks can reduce performance significantly. It is only worth using this if you know that these callbacks still take less time than the evaluation of foo`.