Selecting the parallel kernels where tasks can be executed

When I use ParallelSubmit, is it possible to select on which kernel a task can possibly be executed?

For example if have 3 parallel kernels, can I only use 2 of them so that one is remaining in case I need an emergency task to run on it? But my question is not just for 3 kernels.

This question is equivalent to "Is it possible to associate different kernels with different queues?".

• Maybe worth looking at ParallelEvaluate? – Chen Stats Yu Aug 2 '15 at 22:19
• ParallelEvaluate[expr,kernel]  evaluates expr on the parallel kernel specified. – Chen Stats Yu Aug 2 '15 at 22:20
• Yes but the main kernel waits for the result, thus the main kernel cannot be used at the same time, like when using ParallelSubmit. – faysou Aug 3 '15 at 7:41

1 Answer

First answer

It doesn't seem possible but ParallelSubmit has a Scheduling option which allows to set a priority to a task. The lower it is the earlier the task will be executed.

For example

LaunchKernels[1]
ParallelSubmit[2+2, Scheduling -> 5]


Maybe in case of needing to execute an emergency task on a parallel kernel, a solution would be to submit a task with a high priority (i.e. lower Scheduling than other tasks), launch a new kernel, do QueueRun so that the task gets executed, and then close the new kernel once the task is done, in order not to have a growing amount of kernels.

Second answer

A better possibility is to create a link without LaunchKernels and send to it expressions to evaluate. I've followed the idea of Szabolcs in this answer.

The only disadvantage I see, is that shared functions and variables won't probably be shared with the new kernel (maybe it's possible to get this done, I don't know yet).

kernel=LinkLaunch@"mathkernel -mathlink";
LinkRead[kernel]; (*Return the first input line*)

readFromLink[link_]:=
If[LinkReadyQ@link,
LinkRead[link]//First
,
Missing["LinkNotReady"]
];

LinkWrite[kernel,Unevaluated[Pause@5; 2+2]];


For this example you can evaluate several times the expression below, or just wait a little bit. Note that the main kernel is free.

So you can evaluate to a specific kernel like with ParallelEvaluate, and the main kernel is free like with ParllelSubmit.

readFromLink@kernel


Then you can close the kernel.

LinkClose@kernel


A reference on these functions is here.

Third answer

Here's a third way that I ended up using. I looked at how the Parallel package is implemented and found a hack in order to remove some kernels from the list of kernels where queued evaluations can run. Thus I can still have a queue and kernels that are dedicated to some specific tasks and can be accessed immediately.

A bonus of this method is that reserved kernels can still be debugged in Wolfram workbench and have accesse to shared variables.

The ability to send an instruction to a specific kernel without blocking the main kernel also allows to build one's own queues.

LaunchKernels[3];
reservedKernel = First@ParallelProtected$sortedkernels; ParallelProtected$sortedkernels=Rest@ParallelProtected\$sortedkernels
ParallelDeveloperSend[reservedKernel,2+2]
ParallelDeveloperReceiveIfReady[reservedKernel]