Parallel evaluation hangs before doing anything

Question

[I use Mathematica 9.0.1.0 on OS X Yosemite 10.10.2.]

I am experiencing difficulty with parallel evaluation that I am finding extremely perplexing to troubleshoot.

I define the following function in my notebook:

evolve[k_] := (
    Print[$KernelID];

    systemX = Flatten[NDSolve[{
        scEqn[t, k] == 0,
        s[t1] == xIni[t1, k],
        s'[t1] == xdIni[t1, k]
    }, s, {t, t1, t2}]];    

    systemY = Flatten[NDSolve[{
        scEqn[t, k] == 0,
        s[t2] == yIni[t2, k],
        s'[t2] == ydIni[t2, k]
    }, s, {t, t2, t3}]];

    aa = I a[t2]^3 ((s'[t2] /. systemX)Conjugate[s[t2] /. systemY] - (s[t2] /. systemX)Conjugate[s'[t2] /. systemY]);
    bb = -I a[t2]^3 ((s'[t2] /. systemX)(s[t2] /. systemY) - (s[t2] /. systemX)(s'[t2] /. systemY));
    cc = s[t3] /. systemY;
    result = {k, aa, bb, cc}
)

Definitions for scEqn, xIni, xdIni, yIni, ydIni, a, t1, t2 and t3 all appear previously in the code.

If (CASE I) I evaluate only this function definition (and not any of the preceding definitions in the notebook), and then evaluate

pt = ParallelTable[evolve[k], {k, 1, 4}];

I get the output you'd expect, namely

(kernel 3)   3
(kernel 4)   4
(kernel 2)   2
(kernel 1)   1

followed by a whole bunch of errors because of all the undefined functions and variables.

But if (CASE II) I have evaluated the whole notebook up to that point, so that all the relevant stuff is defined, and run the same ParallelTable command, nothing happens. Specifically:

• The cell bracket highlights to show the cell is evaluating.
• Nothing whatsoever is printed, not even the $KernelIDs.
• This continues indefinitely (I've tried waiting upwards of 30 minutes, just to see if anything would happen).
• The evaluation can be successfully aborted during the first ~10 seconds. After that, attempting to abort has no effect and the only way to recover is to quit the kernel entirely.

Note that the same code runs successfully if I use plain old Table in place of ParallelTable.

I tried running the same code through the command line on a remote Linux server, and I see identical behavior, so it's apparently not just something with my computer.

I am super-confused about what could be going wrong that would cause the Print[$KernelID] statement not to fire. Any difference that I could imagine between CASE I and CASE II all would manifest after that line had already been evaluated.

I realize the code I've written here is not a self-contained minimal example, but if anyone could give any suggestions for related questions or for where in particular I might focus my own troubleshooting efforts, that would be much appreciated.

Answer 1

Did you try

DistributeDefinitions[evolve]

before running ParallelTable ?

Answer 2

In retrospect, all the clues were there…

In order to run evolve on the parallel kernels Mathematica must first distribute the necessary definitions to those kernels, and recursively distribute the definitions involved in those definitions, all the way back.

Which in my case turned out to involve way too much data (mostly a bunch of InterpolatingFunctions) to be trying to copy to a bunch of parallel kernels without hanging.

I refactored so less data needed to be copied to the parallel kernels. This likely means I'm taking a small hit in efficiency, since now there are a couple of things the individual kernels all compute identically, but it's a small price in the scheme of things.

Thanks to @penguin77 for what proved to be the decisive hint for me to figure this out.