4
$\begingroup$

I'm new to Mathematica's parallel evaluation and can't really figure out how to properly use packages on parallel kernels: I get ::shdw messages complaining about shadowing of symbols in Global` and package contexts.

I have a test package in FileNameJoin[{$UserBaseDirectory, "Applications", "Test.m"}] that looks like this:

BeginPackage["Test`"];

f::usage = "Test function.";

Begin["`Private`"];

f[x_] := x*2;

End[];
EndPackage[];

Then I start a local kernel and load the package there:

k = First@LaunchKernels[1];
ParallelEvaluate[Get["Test`"], k]

In the kernel, the package is now on the context path:

ParallelEvaluate[$ContextPath, k]

(* {"Test`", "CloudPublishMenu`", "SystemTools`", "ExternalEvaluateLoader`", "PacletManager`", "System`", "Global`"} *)

When I call the function f like so, it works fine:

ParallelEvaluate[Symbol["f"][3], k]

(* 6 *)

But when I call the function like this it doesn't evaluate and I get an error message:

ParallelEvaluate[f[3], k, DistributedContexts -> None]

Global`f::shdw :  Symbol f appears in multiple contexts {Global`,Test`}; definitions in context Global` may shadow or be shadowed by other definitions.
(* f[3] *)

Having the DistributedContexts option there doesn't make a difference (I've set it to None to prevent potential shadowing by Global` definitions of f in the master kernel).

Am I doing anything fundamentally wrong? Not entirely sure about how symbol lookup works.

$\endgroup$
7
$\begingroup$

How to fix this

In short, either load the package on the master kernel as well, or use

ParallelEvaluate[Test`f[3], k]

Why does this happen

Let's look at what's going on:

k = First@LaunchKernels[1];
ParallelEvaluate[Get["Test`"], k]

This launched parallel kernels and loads the Test package on the first parallel kernel. At this point, $ContextPath contains "Test`" on that kernel, but not on the master kernel.

ParallelEvaluate[f[3], k]

At this point it's important to realize that when Mathematica encounters a symbol that's not in any of the contexts on $ContextPath, a new symbol with that name is created in $Context (unless an explicit context is specified). So let's take a look at the example at hand:

  • ParallelEvaluate: Exists in System`
  • k: Exists in Global`
  • f: Does not exist (remember, on the master kernel, Test` is not on $ContextPath). Therefore, it is created in $Context, i.e. Global`f

This means that the expression really is

System`ParallelEvaluate[Global`f[3], Global`k]

Now, the expression is sent to the parallel kernel k, which then complains that Global`f is shadowing Test`f.

On the other hand, with

ParallelEvaluate[Symbol["f"][3], k]

the symbol is only created once on the parallel kernel. And there f already exists, because Test` is in $ContextPath, so everything works as expected.

$\endgroup$
  • $\begingroup$ Thanks, that explains everything! Too bad, I wanted to use this for evaluating code on a remote kernel with packages that aren't necessarily installed on my machine. Using fully qualified symbols looks tedious. Will try to keep packages in sync between local and remote. $\endgroup$ – wigg0t Nov 14 '18 at 19:35
  • $\begingroup$ Just out of curiosity: If I evaluate ParallelEvaluate[Unevaluated@ToString[f, InputForm], k] on a fresh parallel kernel, I get "f". I would have expected "Global`f". Is the Global` context stripped by ToString? $\endgroup$ – wigg0t Nov 20 '18 at 8:43
  • $\begingroup$ @wigg0t ToString only prepends the context to a symbol's name if that context is not in $ContextPath or $Context (because as long as it is, f will refer to the correct symbol). You can see this by comparing ToString@f with Block[{$ContextPath = {}, $Context = "Test`"}, ToString@f] (the second example temporarily clears both $Context and $ContextPath, so ToString has to prepend Global` in order for the result to refer to the correct symbol) $\endgroup$ – Lukas Lang Nov 21 '18 at 11:30
2
$\begingroup$

As explained by Lukas, it seems one has to either use fully qualified names or load the package on both master and parallel kernels. I find the first option really inconvenient, and in my use case the parallel kernel is running on a remote machine and I don't want to sync the remote packages with my local computer.

What I have tried now (and it seems to work so far) is converting expressions to strings, sending them over to the parallel kernel and then evaluating them with ToExpression.

ClearAll[packExpression];
SetAttributes[packExpression, HoldFirst]
packExpression[expr_] := 
    With[
        {str = ToString[Unevaluated@expr, InputForm]},
        Hold[ToExpression@str]
    ];

ClearAll[evaluate];
SetAttributes[evaluate, HoldFirst];
evaluate[expr_, k_] := 
    With[
        {packed = packExpression@expr}, 
        ParallelEvaluate[Unevaluated@ReleaseHold@packed, k]
    ];

k = First@LaunchKernels[1];
evaluate[Get["Test`"], k];
evaluate[f[3], k]

(* 6 *)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.