Surprising interaction of DistributeDefinition and ParallelNeeds in 11.3

Question

Bug introduced in 11.3.0 and persisting through 12.1.1

(CASE:4039452)

---

I believe I am seeing a new sort of behavior around some parallel functionality in Mathematica 11.3, because code that used to function properly in earlier versions no longer works. There's a bit of setup required, which is probably more convoluted than it strictly needs to be since it requires building some packages.

packageA =
  ToString[
   Unevaluated[
    BeginPackage["A`"];
    A`A = 17;
    EndPackage[];
   ],
   InputForm];
dir = CreateDirectory[];
WriteString[FileNameJoin@{dir, "A.wl"}, packageA];

Let's launch subkernels and make sure each one has the right $Path:

LaunchKernels[];
PrependTo[$Path, dir];
With[{path = $Path},
  ParallelEvaluate[$Path = path]];

Now, let's set a value for A`A on our local kernel:

A`A = 18;
(* 18 *)

And let's load the A` in each subkernel:

ParallelNeeds["A`"];
ParallelEvaluate[A`A]
(* {17, 17, 17, 17} *)

OK, they would never have a reason to see the new value of A`A. Let's give them one:

DistributeDefinitions["A`"]
(* {"A`"} *)

But nothing seems to have happened!

ParallelEvaluate[A`A]
(* {17, 17, 17, 17} *)

I'd expect the output to be {18, 18, 18, 18}. What am I misunderstanding? And am I mistaken that the output would have been {18, 18, 18, 18} in prior versions?

Answer 1

I think this is likely to be a bug.

Historically, DistributeDefinitions has always ignored all symbols in System` and various other system package contexts.

The handling of the excluded contexts has been reorganized in 11.3, in particular there now is

?Parallel`Protected`$ExcludedContexts

$ExcludedContexts is the list of contexts to always exclude from DistributeDefinitions.

which is generated on the fly and unfortunately it looks like our package context ends up on the blacklist for some reason:

MemberQ[Parallel`Protected`$ExcludedContexts, "A`"]

(* True *)

The following works as expected

Block[{Parallel`Protected`$ExcludedContexts = 
    DeleteCases[Parallel`Protected`$ExcludedContexts, "A`"]}, 
      DistributeDefinitions["A`"]
];
ParallelEvaluate[A`A]

(* {18, 18, 18, 18} *)

Answer 2

Another straightforward workaround is to just use ParallelEvaluate @* Needs in the place of ParallelNeeds, like so:

ParallelNeeds["A`"];
ParallelEvaluate[A`A]

(* {17, 17, 17, 17} *)

A`A = 18
(* 18 *)

DistributeDefinitions["A`"];
ParallelEvaluate[A`A]
(* {18, 18, 18, 18} *)