7
$\begingroup$

I run a heavy simulation, using NDSolve and the method of lines. This means that the result of NDSolve is a list of, say, 400-1000 interpolation functions with many interpolation points. The whole thing can weigh up to 4GB.

As it is, this poses no problem. The challenge is when I try to make a movie out the simulation. I do the following: I have a function 

snapshot[sol_,t_]:=GraphicsGrid[{Plot[...],Plot[..],...}]

which take the result of NDSolve (what I call sol) and a specific time t, and makes some plots of the simulation at that time. Then, I export these graphics with something like

Table[
  Export["~/tmp/" <> IntegerString[k, 10, 4] <> ".png",snapshot[sol, tList[[k]] ],
{k, 1, n}]

where tList is a list of times. Then I can make a movie out of the images with an external program (ffmpeg).

The problem is that exporting the graphics like this takes a lot of time and I want to run it in parallel. Using ParallelTable is very bad, because it creates a local copy of the very heavy variable sol in all subkernels, and this immediately freezes my computer. I had no luck with ParallelSubmit or the like. Any ideas?

EDIT

In response to all the commenters:

  1. The plot is indeed local in the time variable.
  2. I tried converting the InterpolatingFunctions to an numerical array. It saves some memory (a factor of ~50) but that's not enough - Still if you duplicate the data 1000 times (for 1000 frames) you'll be in bad shape.

Basically, what I want is to send requests to multiple processors, but that will all run on the same kernel. This way you don't have to duplicate I thought it was possible because in MATLAB I do that naturally with the parfor command (or, at least, I think that's what I'm doing).

Improve this question
$\endgroup$
16
  • $\begingroup$ If you sequentially generate your graphics (which will not be that memory-heavy) e.g. with Table and parallelize the Export of those afterwards, you should be fine. But this will only save time spent on Export, obviously. $\endgroup$
    – Yves Klett
    Commented Dec 31, 2012 at 9:45
  • 3
    $\begingroup$ @yohbs: I think you should be somewhat more specific about what you plot. Do you need all of the 400-1000 interpolating functions for one snapshot call? If you only need a few of them you could try to only distribute what you need. Another thing I have done in similar situations is to deconstruct the interpolating functions and store the result in a large (packed) numeric array. Working with that is much more efficient (both concerning memory and runtimes) and usually the information you loose isn't relevant, especially for plotting purposes... $\endgroup$
    – Albert Retey
    Commented Dec 31, 2012 at 11:24
  • 1
    $\begingroup$ Someone may correct me, but my experience with Exporting graphics tells me that all the rendering is done by the Front End, and the parallelization commands can only send things to different kernels. The Front End (through v8, and maybe v9 is different) does not parallelize any computations. I would love to know I'm wrong, because I've wished to do similar things. $\endgroup$
    – Michael E2
    Commented Jan 1, 2013 at 1:44
  • 1
    $\begingroup$ @MichaelE2 see my answer - this confirms multiple frontend instances for rendering/export, at least for PNGs. $\endgroup$
    – Yves Klett
    Commented Jan 2, 2013 at 9:56
  • 1
    $\begingroup$ @yohbs: one more thing: you will only need as many copies of the data as you have parallel Kernels. So if you managed to reduce the size by a factor ~50 you should be able to distribute that to about 50 parallel Kernels, which most probably is much more than your hardware will run in parallel. I would expect that should solve your problem, can you explain why you think that it doesn't? $\endgroup$
    – Albert Retey
    Commented Jan 2, 2013 at 14:41

1 Answer 1

Reset to default
2
$\begingroup$

Only a partial answer (excluding memory issues), but... Exporting in parallel seems to work (here with version 9 in Win 7 64bit):

gfx = Table[PolarPlot[Sin[3 t*i], {t, 0, Pi}], {i, 1, 10}];

ParallelTable[Print[$KernelID]; 
 Export["test" <> ToString[i] <> ".png", gfx[[i]]], {i, 1, 10}]

See the $KernelID - seems to work in parallel. Also, a number of "Wolfram Mathematica 9.0 Server" windows show up and ("Mathematica.exe" processes in the task manager), plus related kernels.

Mathematica graphics

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for looking into it. I modified it to make it take long enough I could monitor the processes and processors: ParallelTable[Print[$KernelID]; Export["test" <> ToString[i] <> ".png", Show[Table[Plot3D[k + Sin[3 t*i + \[Pi] i Cos[s]], {t, 0, Pi}, {s, 0, Pi}], {k, 10}]]], {i, 1, 10}]. Each kernel computes a Graphics3D and sends it to a front-end, each process waits on the other. You can see the switching in the (Mac) Activity Monitor. I believe this is new. When I tested it in v8, each kernel waited on the same, one and only, front-end process. $\endgroup$
    – Michael E2
    Commented Jan 2, 2013 at 16:06
  • $\begingroup$ @MichaelE2 Did you use "Evaluation"->"Parallel kernel status" for that as well? I did not bother, but is comes in useful at times. $\endgroup$
    – Yves Klett
    Commented Jan 2, 2013 at 16:10
  • $\begingroup$ No, I wasn't aware of that option -- I mean, menu command. Thanks. $\endgroup$
    – Michael E2
    Commented Jan 2, 2013 at 16:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.