1
$\begingroup$

I am scheduling myFun[x] to evaluate every couple of seconds in a parallel kernel with ParallelSubmit and Parallel`Developer`QueueRun[], where it sometimes triggers (multiple) additional follow-up functions, which need to be submitted for evaluation on the next available parallel kernels, i.e.

CreateScheduledTask[Parallel`Developer`QueueRun[],.1]
CreateScheduledTask[ParallelSubmit[(myFun[x];If[triggerQ,ParallelSubmit[myFollowUpFun[x]]);]],1]

But nesting ParallelSubmit in this way doesn't seem to work.

Is there a way to submit an expression for evaluation on the next available parallel kernel from within a parallel kernel?


EDIT Per Szabolcs comment I tried the following code which seems to behave as intended.

<< Parallel`Developer`
ClearKernels[]; CloseKernels[]; LaunchKernels[4];
SetSharedVariable[x, y, z]

qRunTask = CreateScheduledTask[QueueRun[], .1];
StartScheduledTask[qRunTask];

f[u_] := ParallelSubmit[ReleaseHold[u]];
SetSharedFunction[f]

Dynamic[Refresh[{x, y, z}]]

x = y = z = 0;

ParallelSubmit[x = $KernelID];
ParallelSubmit[f[Hold[Pause[5]; z = $KernelID]]; y = $KernelID;]
$\endgroup$
  • 2
    $\begingroup$ I believe this can only be done on the main kernel. Thus you need to call back to the main kernel. One way to do this is to use SetSharedFunction, which effectively marks a function as evaluatable on the main kernel only. I have not tested this though. $\endgroup$ – Szabolcs May 18 '18 at 11:10
  • $\begingroup$ @Szabolcs thank you for the tip, I did not know that SetSharedFunction restricted functions to the main kernel only. It seems to do the trick. $\endgroup$ – user19218 May 18 '18 at 12:31
  • 3
    $\begingroup$ When a subkernel encounters a shared function, it sends it back to the main kernel for evaluation, then retrieves the result. The same with shared variables: their value is always read/set on the main kernel. This provides an additional data exchange method between subkernels (whose memory is otherwise entirely isolated). The drawback is that all these callbacks to the main kernel can have a severe performance impact. Cross-kernel communication is generally slow, so typically we want to reduce it as much as possible. $\endgroup$ – Szabolcs May 18 '18 at 12:35
3
$\begingroup$

Just to preserve Szabolcs' answer in comments:

I believe this can only be done on the main kernel. Thus you need to call back to the main kernel. One way to do this is to use SetSharedFunction, which effectively marks a function as evaluatable on the main kernel only.

When a subkernel encounters a shared function, it sends it back to the main kernel for evaluation, then retrieves the result. The same with shared variables: their value is always read/set on the main kernel. This provides an additional data exchange method between subkernels (whose memory is otherwise entirely isolated).

The drawback is that all these callbacks to the main kernel can have a severe performance impact. Cross-kernel communication is generally slow, so typically we want to reduce it as much as possible.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.