5
$\begingroup$

I am using Mathematica to generate 3D graphics frame for a video. The process goes as follows:

  • Loop over a counter i
    • Use Import to load in data from data.$i.hdf5 into data
    • Apply some expression to data, then use Interpolation on it
    • Make a ParametricPlot3D of the surface with PlotPoints -> 50 and store it in outgfx
    • Export result to outgfx.png with ImageSize -> {2048, 2048}

The problem is that MathKernel keeps crashing every 3-5 frames with no reason given. I tried putting the entire body of the loop in a Block and store all variables locally, hoping that the garbage collector would be better behaved, but this did not help. My current solution is to manually babysit Mathematica and manually change the loop limits and re-run the code every time I hear a beep.

I figured that way to sidestep this issue is to launch a new kernel for each iteration, do the processing, then shut down the kernel before looping. If the new kernel crashes, since the loop itself runs in a different kernel, it can keep going.

Is there a way to do this in the frontend?

$\endgroup$
  • 1
    $\begingroup$ a quick fix which will not solve your problem but probably makes it less problematic could be to set $HistoryLength=0 which will probably help in decreasing the memory consumption. For running in extra kernels you could have a look at the parallel functionality. If you use that with only one parallel kernel and do CloseKernels once in a while that probably is the most convenient interface to do what you want. For your application using more than one kernel in parallel would probably also make sense. It would certainly also be possible to start a kernel with lower level functionality... $\endgroup$ – Albert Retey May 19 '15 at 9:20
5
$\begingroup$

After a bit of consideration I recommend that you use ParallelSubmit and related functionality, e.g. WaitAll. Please consider this example:

job = Array[ParallelSubmit[Pause[1]; If[Random[] > 0.7, Quit[]]; Print[#]] &, 10]

enter image description here

This creates a series EvaluationObject tasks to perform. Within each is a 30% chance of failure simulated by If[Random[] > 0.7, Quit[]]. To start the job use:

WaitAll[job]

You can watch the progress of the tasks in real time by looking at the displayed output of job:

enter image description here

You will note that when a kernel "crashes" a number of messages are printed (these could be suppressed with Quiet) and the task is tried again on a different kernel. Ultimately every Print task is performed once:

(kernel 4) 1

(kernel 3) 2

(kernel 4) 5

(kernel 6) 3

(kernel 6) 10

(kernel 8) 4

(kernel 9) 6

(kernel 10) 7

(kernel 8) 9

(kernel 11) 8

$\endgroup$
  • $\begingroup$ Thanks for this! Unfortunately running 3D routines in parallel seems to cause a kernel panic on OSX... I guess I could limit the number of kernels. $\endgroup$ – Saran May 19 '15 at 10:18
  • $\begingroup$ @Saran Please try evaluating CloseKernels[]; LaunchKernels[1] before running this code. That should limit it to the use of one kernel at a time. $\endgroup$ – Mr.Wizard May 19 '15 at 10:22
  • 1
    $\begingroup$ I haven't had success with this approach, yet, as mma keeps failing on me with irreproducible symptoms (mangled graphics, black graphics, OSX kernel panic, making my entire system unresponsive). However, I will accept this answer as I think it's the closest thing there is to a solution to this problem! $\endgroup$ – Saran May 20 '15 at 15:31
  • $\begingroup$ @Saran I am sorry to hear of your ongoing problems. As a Windows user I am not sure how many of them I could help with but I wish you luck. If you post questions with as much clarity as possible and sufficient detail people here are usually willing (and able) to help. Thanks for the Accept anyway. $\endgroup$ – Mr.Wizard May 20 '15 at 16:23
2
$\begingroup$

After multiple failures to achieve truly unattended batch plotting setup solely in Mathematica, I have finally decided to go with a hybrid approach instead. I have exported the plotting commands into a Mathematica script, then use a separate Python script to call WolframScript.

After a couple of plots, the MathKernel will still crash as usual. This will pass control back to the Python script, which can check the output directory to determine how far the Kernel has gotten to. It would then restart WolframScript again with a new starting number as appropriate.

My Python script looks like this:

#!/usr/bin/python

import re
import os
import subprocess

r = re.compile(r"frame([0-9]{3}).png")

firstframe = 0
lastframe  = 500

def maxframe():
    out = -1
    for file in os.listdir('/path/to/input/files/'):
        m = r.match(file)
        if m:
            frame = int(m.group(1))
            if out < frame:
                out = frame
    return out

while maxframe() < lastframe:
    subprocess.call(
        './make_frame.m %u %u' % (maxframe() + 1, lastframe),
        shell = True,
        preexec_fn = os.setpgrp)

and the Mathematica script make_frame.m:

#!/Applications/Mathematica.app/Contents/MacOS/WolframScript -script

firstframe = ToExpression[$ScriptCommandLine[[2]]];
lastframe  = ToExpression[$ScriptCommandLine[[3]]];

coords = Flatten[Outer[List,
    Table[i Pi/254, {i, 0, 254}], Table[2 i Pi/128, {i, 0, 128}]], 1];

x[r_, th_, ph_] := r Sin[th] Cos[ph];
y[r_, th_, ph_] := r Sin[th] Sin[ph];
z[r_, th_, ph_] := r Cos[th];

For[n = firstframe, n <= lastframe, n++,
    data = Import[
        "/path/to/input/files/data" <> IntegerString[n, 10, 3] <> ".hdf5",
        {"Datasets", "/r"}
    ];

    rfun = Interpolation[
        Transpose[{coords, data}],
        PeriodicInterpolation -> {False, True}
    ];

    gfx = ParametricPlot3D[
        {
            x[th, ph, rfun[th, ph]],
            y[th, ph, rfun[th, ph]],
            z[th, ph, rfun[th, ph]]
        },
        {th, 0, Pi}, {ph, 0, 2 Pi},
        PlotRange -> {{-1, 1}, {-1, 1}, {-1, 1}},
        PlotPoints -> 50,
        Boxed -> False,
        Axes -> None
    ];

    Export[
        "/path/to/output/files/frame" <> IntegerString[n, 10, 3] <> ".png",
        gfx,
        ImageSize -> {1024, 1024}
    ];

    Print[n];
];
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.