15
$\begingroup$

I am using Mathematica 11.3 and this seems to me to be a bug. I would like, if possible, some idea on a workaround.

Here is an example of a trivial code that works as expected:

nI = 10;
(NM[#] :=  Print[#] ) & /@ Range[1, nI];
LaunchKernels[];
DistributeDefinitions[NM];

That is, the code above generates no output, as expected.

Now, if the first line is changed to

nI = 20;

The same code leads to 40 lines being printed! From 1 to 20 two times.

For some reason, the DistributeDefinitions is forcing the definition of NM to be executed, and I do not want that to happen before I use ParallelSubmit and WaitAll. I tried this in two computers with Mathematica 11.3, any ideas on what is happening?

$\endgroup$
  • 2
    $\begingroup$ On my machine the behaviour changes from nI=17 to 18. If I look at Definitions[NM], up to 17 it prints as :=-definitions, but above that it prints as =-definitions. Thus the core issue may be not with the parallel tools, but with how definitions are stored. $\endgroup$ – Szabolcs Dec 28 '18 at 20:58
  • 3
    $\begingroup$ The reason appears to be that Language`ExtendedFullDefinition switches from using RuleDelayed to Rule after 18 down-values. This causes DistributeDefinitions to subsequently leak the evaluation. $\endgroup$ – Lukas Lang Dec 28 '18 at 20:58
  • $\begingroup$ Can you please report this to Wolfram support? $\endgroup$ – Szabolcs Dec 31 '18 at 15:55
  • 1
    $\begingroup$ Thanks for the suggestion. I already sent an email to wolfram support, I am wating for their reply. $\endgroup$ – Davi Rodrigues Jan 1 at 16:08
  • 1
    $\begingroup$ I confirm this is still a bug in M12 $\endgroup$ – Andreas Lauschke May 15 at 14:31
14
$\begingroup$

TL;DR;

Execute the following code to fix the problem:

DistributeDefinitions;
DownValues[Parallel`Protected`DistDefs] =
  DownValues[Parallel`Protected`DistDefs] /.
   HoldPattern[
     u : Parallel`Parallel`Private`updates =
      rhs_Language`ExtendedFullDefinition
     ] :>
    (
     u = Replace[
       rhs,
       defs : {(_HoldPattern -> _) ..} :> With[
         {res = RuleDelayed @@@ Unevaluated@defs},
         res /; True
         ],
       {4}
       ]
     );

What does this do?

As mentioned in the comments, the issue is that Language`ExtendedFullDefinition seems to change the return format at 18 down-values. This causes the subsequent manipulations of DistributeDefinitions on the returned Language`DefinitionList[…] expression to leak the evaluation of the definitions (as their r.h.s. are no longer protected by the HoldRest attribute of RuleDelayed).

The code above fixes this by wrapping the call to Language`ExtendedFullDefinition (which happens in Parallel`Protected`DistDefs) with a piece of code that replaces definitions of the form HoldPattern[…]->… with HoldPattern[…]:>…, which prevents the evaluation leak.

$\endgroup$
  • $\begingroup$ It worked, thanks! I think this is a terrible Mathematica bug, since no "hack" should be necessary to demand the same behaviour from nI=17 to nI=18. Although the code above works, I am worried if it will not have side effects. That is, can i simply insert it in my init.m file and always use it? $\endgroup$ – Davi Rodrigues Dec 29 '18 at 15:54
  • 1
    $\begingroup$ Yes, stuff like this is really annoying... Regarding potential side effects: As far as I can tell, the fix should be completely safe: The only thing it does is to replace the definitions using Rule with ones using RuleDelayed, which is definitely supported (since it is what happens for less than 18 definitions). It is also a very localized modification, so it should only affect DistributeDefinitions, which is anyway half broken for more than 17 definitions. But in case you do encounter any issues, please let me know $\endgroup$ – Lukas Lang Dec 29 '18 at 16:00
2
$\begingroup$

When dealing with a larger number of definitions my recommendation is to disable the automatic distribution of definitions with

$DistributedContexts = None

and simply redo the definitions on all parallel kernels:

nI = 20;
(NM[#] := Print[#]) & /@ Range[1, nI];

With[{nI = nI}, ParallelEvaluate[(NM[#] := Print[#])& /@ Range[1, nI];]]

It's faster, too.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.