Selecting the parallel kernels where tasks can be executed

Question

When I use ParallelSubmit, is it possible to select on which kernel a task can possibly be executed?

For example if have 3 parallel kernels, can I only use 2 of them so that one is remaining in case I need an emergency task to run on it? But my question is not just for 3 kernels.

This question is equivalent to "Is it possible to associate different kernels with different queues?".

Answer 1

First answer

It doesn't seem possible but ParallelSubmit has a Scheduling option which allows to set a priority to a task. The lower it is the earlier the task will be executed.

For example

LaunchKernels[1]
ParallelSubmit[2+2, Scheduling -> 5]

Maybe in case of needing to execute an emergency task on a parallel kernel, a solution would be to submit a task with a high priority (i.e. lower Scheduling than other tasks), launch a new kernel, do QueueRun so that the task gets executed, and then close the new kernel once the task is done, in order not to have a growing amount of kernels.

---

Second answer

A better possibility is to create a link without LaunchKernels and send to it expressions to evaluate. I've followed the idea of Szabolcs in this answer.

The only disadvantage I see, is that shared functions and variables won't probably be shared with the new kernel (maybe it's possible to get this done, I don't know yet).

kernel=LinkLaunch@"mathkernel -mathlink";
LinkRead[kernel]; (*Return the first input line*)

readFromLink[link_]:=
    If[LinkReadyQ@link,
        LinkRead[link]//First
        ,
        Missing["LinkNotReady"]
    ];

LinkWrite[kernel,Unevaluated[Pause@5; 2+2]];

For this example you can evaluate several times the expression below, or just wait a little bit. Note that the main kernel is free.

So you can evaluate to a specific kernel like with ParallelEvaluate, and the main kernel is free like with ParllelSubmit.

readFromLink@kernel

Then you can close the kernel.

LinkClose@kernel

A reference on these functions is here.

---

Third answer

Here's a third way that I ended up using. I looked at how the Parallel package is implemented and found a hack in order to remove some kernels from the list of kernels where queued evaluations can run. Thus I can still have a queue and kernels that are dedicated to some specific tasks and can be accessed immediately.

A bonus of this method is that reserved kernels can still be debugged in Wolfram workbench and have accesse to shared variables.

The ability to send an instruction to a specific kernel without blocking the main kernel also allows to build one's own queues.

LaunchKernels[3];
reservedKernel = First@Parallel`Protected`$sortedkernels;
Parallel`Protected`$sortedkernels=Rest@Parallel`Protected`$sortedkernels
Parallel`Developer`Send[reservedKernel,2+2]
Parallel`Developer`ReceiveIfReady[reservedKernel]