7
$\begingroup$

I have the following code, which fails to distribute the DownValues for the symbol f. Could anybody please explain what is wrong.

f[1] = Range[10] + 1;
f[2] = Range[10] + 2;
f[3] = Range[10] + 3;

DownValues[f]

(* {HoldPattern[f[1]] :> {2, 3, 4, 5, 6, 7, 8, 9, 10, 11}, 
    HoldPattern[f[2]] :> {3, 4, 5, 6, 7, 8, 9, 10, 11, 12}, 
    HoldPattern[f[3]] :> {4, 5, 6, 7, 8, 9, 10, 11, 12, 13}, 
    HoldPattern[f[4]] :> {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}} *)

ParallelEvaluate[Clear[f]]
DistributeDefinitions[f]

(* {Null, Null, Null, Null, Null, Null, Null, Null} *)
(* {} *)

ParallelEvaluate[DownValues[f]]

(* {{}, {}, {}, {}, {}, {}, {}, {}} *)

Any suggestions, pointers would be greatly appreciated

Improve this question
$\endgroup$
15
  • 1
    $\begingroup$ @MarcoB I think there is something fishy. When I quit all kernels and start a fresh Matematica session I get the parallel downvalues correct. The problem is erratic, sometimes I get empty braces and sometimes the right definition. $\endgroup$
    – Iconoclast
    Commented Jun 29, 2020 at 19:55
  • $\begingroup$ @MarcoB: I would request you to kindly run this short piece of code if you have time, please. I am really bogged down because of this as I want to do a huge parallel computation for my work. $\endgroup$
    – Iconoclast
    Commented Jun 29, 2020 at 20:03
  • 1
    $\begingroup$ @MarcoB Definition[f] is tricky. It does not evaluate, it just formats in a certain way. Try ParallelEvaluate[Echo@Definition[f]] to see that f has no definitions on the subkernels. $\endgroup$
    – Szabolcs
    Commented Jun 29, 2020 at 20:21
  • 2
    $\begingroup$ However, by this point, the parallel tools framework has made a note for itself (on the main kernel) that "f is already distributed, no need to touch it until it changes on the main kernel again". You are not supposed to mess with it on the subkernel, as distribution is one way only (main kernel -> subkernel) and the system assumes that it is fully managing the definitions of f on the subkernel. It assumes that you are not interfering. $\endgroup$
    – Szabolcs
    Commented Jun 29, 2020 at 21:00
  • 2
    $\begingroup$ Your DistributeDefinitions[f] does nothing as the system believes that distribution has already happened, and there's no need to do it again. Notice that it returns {}. That means that nothing was distributed. $\endgroup$
    – Szabolcs
    Commented Jun 29, 2020 at 21:00

1 Answer 1

Reset to default
4
$\begingroup$

I believe the hypothesis of Szabolcs made in the comments is correct, and Mathematica only distributes definitions it deems to be "new".

Digging a little into the definition of DistributeDefinitions, it calls Parallel`Protected`DistDefs, which then calls Parallel`Parallel`Private`updatedDefs. When the definitions of f have not yet been distributed, updatedDefs returns them untouched and creates the downvalue Parallel`Parallel`Private`$distributedDefs[HoldForm[f]], which in this case takes the form

Parallel`Parallel`Private`$distributedDefs[HoldForm[f]] =
{OwnValues -> {}, SubValues -> {}, UpValues -> {},  
 DownValues -> {HoldPattern[f[1]] -> {2, 3, 4, 5, 6, 7, 8, 9, 10, 11},
    HoldPattern[f[2]] -> {3, 4, 5, 6, 7, 8, 9, 10, 11, 12}, 
   HoldPattern[f[3]] -> {4, 5, 6, 7, 8, 9, 10, 11, 12, 13}}, 
 NValues -> {}, FormatValues -> {}, DefaultValues -> {}, 
 Messages -> {}, Attributes -> {}}

When DistributeDefinitions is called again without having modified f, updateDefs deletes the definitions of f from the the list of definitions to be distributed because

Parallel`Parallel`Private`updatedDefs[s : "HoldForm"[_Symbol] -> vals_List, True] /; 
  Parallel`Parallel`Private`$distributedDefs[s] === vals :=
  Sequence @@ {}

The solution to force a redistribution of definitions is to unset the appropriate $distributedDefs beforehand:

f[1] = Range[10] + 1;
f[2] = Range[10] + 2;
f[3] = Range[10] + 3;

ParallelEvaluate[Clear[f]];

Parallel`Parallel`Private`$distributedDefs[HoldForm[f]] =.

DistributeDefinitions[f]

(* {f} *)

ParallelEvaluate[DownValues[f]]

(* {{HoldPattern[f[1]]:>{2,3,4,5,6,7,8,9,10,11}, 
    HoldPattern[f[2]]:>{3,4,5,6,7,8,9,10,11,12}, 
    HoldPattern[f[3]]:>{4,5,6,7,8,9,10,11,12,13}}
   ,
   {HoldPattern[f[1]]:>{2,3,4,5,6,7,8,9,10,11}, 
    HoldPattern[f[2]]:>{3,4,5,6,7,8,9,10,11,12}, 
    HoldPattern[f[3]]:>{4,5,6,7,8,9,10,11,12,13}}} *)

This sets the definitions in the subkernels correctly even when evaluated multiple times.

The simpler option would of course be not to clear f in the subkernels by hand, as DistributeDefinitions does that in any case according to the documentation. But I guess this behavior is generally good to know about, if the subkernels run code that actively modifies the local definitions.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.