Communication between local kernels

Question

this is my first post but you have already helped me so many times that I have to start by saying thanks a lot!

I have a really long calculation to do, and I have used the Evaluation->Kernel configuration options to create 2 new local kernels and divided the task in three groups of little ones. The problem is that I don't know how to share the information between different notebooks running on different local kernels. Can you help me?

Thanks again anyway.

Answer 1

You can use Mathematica's parallel tools, which already implement inter-kernel communication. What I show below is not the typical way to use parallelization. I try to follow your requirements more closely. I do strongly recommend you read through the parallelization docs and learn the basics before you use this though.

When using the parallelization framework, we have one main kernel and several subkernels. Parallelization works by dividing the task into parts automatically (as I understand you want to do this manually) and evaluating the parts in parallel on the suberkernels. E.g. ParalellTable will do this mostly automatically for a Table-like computation. Now we're going to do something similar semi-manually:

First, launch 3 kernels:

LaunchKernels[3]

Then determine which symbols need to be sent to the subkernels, and pass them to DistributeDefinitions:

a = 1
f[x_] := a x^2
b = 2

DistributeDefinitions[f,b]

Note that a will also be distributed because f depends on it.

Kernels[] will give you the handles to all subkernels.

ParallelEvaluate[f[b], Kernels[][[1]]]

will evaluate f[b] on the first kernel. You could also just refer to this kernel by its $KernelID, which will be 1 for the first startup up kernel: ParallelEvaluate[f[b], 1].

The result will be automatically returned to the main kernel.

ParallelEvaluate[f[b]] will evaluate f[b] on all three kernels simultaneously.

The problem with ParallelEvaluate is that it blocks: it won't return until the suberkenl evaluation has finished. So if you need to do different evaluations on the three kernels, then ParallelEvaluation is not the most convenient solutions.

If you need to do asynchronous evaluations, e.g. you need to use the main kernel while the subkernels are working, then use the lower level constructs described in this tutorial.

A small example:

task1 = ParallelSubmit[1 + 1]

will submit a task to be evaluated.

task2 = ParallelSubmit[5!]

will submit another task

WaitAll[{task1, task2}]

will start both, wait wait for them to finish and will return the results in a list as {2, 120}.

---

The parallel tools use MathLink (called WSTP since version 10) to communicate between the main kernel and subkernels. Theoretically you could implement a communication framework yourself using MathLink, but this is likely going to take more effort than what's reasonable to spend on this problem ...

Here's a short example to give a taste of how this would work:

LinkLaunch will take care of launching a kernel, creating a MathLink connection, and connecting to it:

In[1]:= kernel = LinkLaunch["/Applications/Mathematica\\ 10.app/Contents/MacOS/MathKernel -mathlink"]    
Out[1]= LinkObject["/Applications/Mathematica\\ 10.app/Contents/MacOS/MathKernel -mathlink", 73, 4]

These steps could also be broken down, and you could use LinkCreate and LinkConnect.

In[2]:= LinkRead[kernel]    
Out[2]= InputNamePacket["In[1]:= "]

Let's send something to that kernel for evaluation:

In[3]:= LinkWrite[kernel, Unevaluated@EvaluatePacket[1 + 1]]

And read back the result:

In[4]:= LinkRead[kernel]    
Out[4]= ReturnPacket[2]

If the evaluation has not finished yet, LinkRead will block until there is something to read from the link. To avoid this, you can check whether the link is ready before reading using LinkReadyQ. Be careful not to try to read from a link that won't be able to send anything, or the kernel will lock up.

Unfortunately the workings of MathLink are not quite as well documented as the rest of Mathematica, but you'll find several example here and here.

Answer 2

The standard way is to use in the main notebook the DistributeDefinitions[{list of yoyr variables}]command.

It will automatically share all mentioned mathematica names and it's content to all kernels. This in turn will make it available in all notebooks related to those kernels. But why do you use some notebooks for parallelization? Any tasks could be run in parallel from the one notebook - just use parallel routines instead of usual.