2
$\begingroup$

I couldn't find a better title, so I'll jump straight to the example!

There is this long list of slow jobs to be executed in a parallel way, at the press of a button:

DynamicModule[{dispatch, list, doStuff},
 list = Range[8];
 dispatch[] := ParallelMap[(Pause[0.5]; Print@{$KernelID, #}) &, list];
 Button["Go", dispatch[] , Method -> "Queued"]
 ]

This works as expected, with every kernel printing it's job and ID. As soon as I start modularizing code, things break:

DynamicModule[{dispatch, list, doStuff},
 list = Range[8];
 doStuff[a_] := (Pause[0.5]; Print@{$KernelID, a});
 dispatch[] := ParallelMap[doStuff, list];
 Button["Go", dispatch[] , Method -> "Queued"]
 ]

This executes sequentially using only the main kernel (ID 0).

I've tried distributing definitions, contexts and even sharing variables, but the closest I've gotten to parallelization was the following:

DynamicModule[{dispatch, list, doStuff, doSlowStuff},
 list = Range[8];
 doSlowStuff[] := Pause[0.5];
 doStuff[a_] := (doSlowStuff[]; Print@{$KernelID, a});
 dispatch[] := ParallelMap[doStuff, list];
 dispatch[];  (*This call is correctly executed in parallel*)
 Button["Go", dispatch[] , Method -> "Queued"]
 ]

The "inline" dispatch[] call is correctly executed (using all kernels) even with the added nested definition of doSlowStuff, but the button's dispatch[] still executes sequentially on kernel 0.

What am I missing?

$\endgroup$
  • $\begingroup$ This question This explains the definitions' "unneeded" brackets! The sad part is that I've only found out about it after spending 1h+ brute-forcing tentative solutions, only to arrive at the same answer. $\endgroup$ – Aisamu Nov 9 '14 at 13:05
2
$\begingroup$

It might be that this is due to dispatch, list and doStuff being completely owned by the frontend, since you wrapped them in a DynamicModule. Frontend variables cannot be shared between kernels. When you localize them in a Module, and therefore keep them as kernel variables, it works:

Module[{dispatch, list, doStuff}, list = Range[8];
 doStuff[a_] := (Pause[0.5]; Print@{$KernelID, a});
 dispatch[] := ParallelMap[doStuff, list];
 Button["Go", dispatch[], Method -> "Queued"]]

(* {4,1} {3,3} {2,5} {1,7} {4,2} {3,4} {2,6} {1,8} *)

It is possible to accomplish this task in a DynamcModule:

DynamicModule[{dispatch, doSlowStuff, list = Range[6]},
 doSlowStuff[x_] := (Pause[0.1]; x^2);
 dispatch[] := ParallelMap[Print[{$KernelID, doSlowStuff[#]}] &, list];
 Button["Go", SetSharedFunction[doSlowStuff, dispatch]; dispatch[]]
]

(* {4,1} {3,9} {2,25} {1,36} {4,4} {3,16} *)

The local frontend variables dispatch and doSlowStuff are linked to kernel variables in the context FE`. It seems that these variables are not automatically distributed over the sub kernels, so we have to use SetSharedFunctions. When this is done outside the button, the kernel variables and not the frontend variables are distributed. Since the kernel variables are removed when the evaluation of the DynamicModule is completed, that will not work. So SetSharedFunctions must be used inside the button.

$\endgroup$
  • $\begingroup$ Oh... but why does the last, "inline", dispatch[] works? I'm afraid I might need the DynamicModule, as it is part of a GUI. $\endgroup$ – Aisamu Nov 9 '14 at 12:56
  • 1
    $\begingroup$ @Aisamu When you define DynamicModule[{f}, f[] := something;...], my feeling is that the frontend variable f is automatically linked to a kernel variabel FE`f$$nn and that to the latter in the kernel the function definition is assigned. I have done some experiments that seem to confirm this. Therefore your inline dispatch works. I still have to study your not working button, maybe I come back to that later. $\endgroup$ – Fred Simons Nov 9 '14 at 19:53
  • $\begingroup$ Let's hope you find something! I'm still struggling with it... $\endgroup$ – Aisamu Nov 9 '14 at 19:59
  • $\begingroup$ @Aisamu. I just edited my answer with respect to the button inside the DynamicModule. Hope this helps. $\endgroup$ – Fred Simons Nov 10 '14 at 10:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.