Running bits of code in a separate kernel

Question

I am using Mathematica to generate 3D graphics frame for a video. The process goes as follows:

• Loop over a counter i
  • Use Import to load in data from data.$i.hdf5 into data
  • Apply some expression to data, then use Interpolation on it
  • Make a ParametricPlot3D of the surface with PlotPoints -> 50 and store it in outgfx
  • Export result to outgfx.png with ImageSize -> {2048, 2048}

The problem is that MathKernel keeps crashing every 3-5 frames with no reason given. I tried putting the entire body of the loop in a Block and store all variables locally, hoping that the garbage collector would be better behaved, but this did not help. My current solution is to manually babysit Mathematica and manually change the loop limits and re-run the code every time I hear a beep.

I figured that way to sidestep this issue is to launch a new kernel for each iteration, do the processing, then shut down the kernel before looping. If the new kernel crashes, since the loop itself runs in a different kernel, it can keep going.

Is there a way to do this in the frontend?

Answer 1

After a bit of consideration I recommend that you use ParallelSubmit and related functionality, e.g. WaitAll. Please consider this example:

job = Array[ParallelSubmit[Pause[1]; If[Random[] > 0.7, Quit[]]; Print[#]] &, 10]

This creates a series EvaluationObject tasks to perform. Within each is a 30% chance of failure simulated by If[Random[] > 0.7, Quit[]]. To start the job use:

WaitAll[job]

You can watch the progress of the tasks in real time by looking at the displayed output of job:

You will note that when a kernel "crashes" a number of messages are printed (these could be suppressed with Quiet) and the task is tried again on a different kernel. Ultimately every Print task is performed once:

(kernel 4) 1

(kernel 3) 2

(kernel 4) 5

(kernel 6) 3

(kernel 6) 10

(kernel 8) 4

(kernel 9) 6

(kernel 10) 7

(kernel 8) 9

(kernel 11) 8

Answer 2

After multiple failures to achieve truly unattended batch plotting setup solely in Mathematica, I have finally decided to go with a hybrid approach instead. I have exported the plotting commands into a Mathematica script, then use a separate Python script to call WolframScript.

After a couple of plots, the MathKernel will still crash as usual. This will pass control back to the Python script, which can check the output directory to determine how far the Kernel has gotten to. It would then restart WolframScript again with a new starting number as appropriate.

My Python script looks like this:

#!/usr/bin/python

import re
import os
import subprocess

r = re.compile(r"frame([0-9]{3}).png")

firstframe = 0
lastframe  = 500

def maxframe():
    out = -1
    for file in os.listdir('/path/to/input/files/'):
        m = r.match(file)
        if m:
            frame = int(m.group(1))
            if out < frame:
                out = frame
    return out

while maxframe() < lastframe:
    subprocess.call(
        './make_frame.m %u %u' % (maxframe() + 1, lastframe),
        shell = True,
        preexec_fn = os.setpgrp)

and the Mathematica script make_frame.m:

#!/Applications/Mathematica.app/Contents/MacOS/WolframScript -script

firstframe = ToExpression[$ScriptCommandLine[[2]]];
lastframe  = ToExpression[$ScriptCommandLine[[3]]];

coords = Flatten[Outer[List,
    Table[i Pi/254, {i, 0, 254}], Table[2 i Pi/128, {i, 0, 128}]], 1];

x[r_, th_, ph_] := r Sin[th] Cos[ph];
y[r_, th_, ph_] := r Sin[th] Sin[ph];
z[r_, th_, ph_] := r Cos[th];

For[n = firstframe, n <= lastframe, n++,
    data = Import[
        "/path/to/input/files/data" <> IntegerString[n, 10, 3] <> ".hdf5",
        {"Datasets", "/r"}
    ];

    rfun = Interpolation[
        Transpose[{coords, data}],
        PeriodicInterpolation -> {False, True}
    ];

    gfx = ParametricPlot3D[
        {
            x[th, ph, rfun[th, ph]],
            y[th, ph, rfun[th, ph]],
            z[th, ph, rfun[th, ph]]
        },
        {th, 0, Pi}, {ph, 0, 2 Pi},
        PlotRange -> {{-1, 1}, {-1, 1}, {-1, 1}},
        PlotPoints -> 50,
        Boxed -> False,
        Axes -> None
    ];

    Export[
        "/path/to/output/files/frame" <> IntegerString[n, 10, 3] <> ".png",
        gfx,
        ImageSize -> {1024, 1024}
    ];

    Print[n];
];