# Launching external program in background process with ParallelSubmit

I work on the implementation of an image processing pipeline in Mathematica in which I also would like to use external programs to do certain processing steps. As an example I have macros for the program FIJI that are written, parameterized and then executed with the Run command in Mathematica. Some of these processing steps just need to be executed and I do not necessarily need the results in the pipeline later on. For those computations I thought it would be nice to let the computation be run in a background process and continue with other steps while the background process is still running.

I've read about pushing computations to background kernels (Computing Many Slow I/O Operations, Concurrency: Managing Parallel Processes) and it works fine. What I've tried is the following: I launch a bunch of kernels, generate some processes and queue those processes using ParallelSubmit. I then use QueueRun to start the processes in the background and collect the results with WaitAll after a while.

CloseKernels[];
Needs["ParallelDeveloper"];
ResetQueues[];
LaunchKernels[5];

ids = Function[i,
ParallelSubmit[(Pause[i^2]; {i^2, DateString[]}),
Scheduling -> i]] /@ {1, 2, 3, 4, 5}


{QueueRun[], $QueueLength}  {True, 0} WaitAll[ids]  {{1, "Mon 10 Mar 2014 09:37:53"}, {4, "Mon 10 Mar 2014 09:37:56"}, {9, "Mon 10 Mar 2014 09:38:01"}, {16, "Mon 10 Mar 2014 09:38:08"}, {25, "Mon 10 Mar 2014 09:38:17"}} DateString[]  Mon 10 Mar 2014 09:41:23 I included DateString to see, when the tasks have been processed in the background kernels. As can be seen from the timings it worked as expected and I also was able to do computations in the front end while the tasks were running. Now I'd like to do the same thing with calling external programs like FIJI but for some reasons the tasks are processed not before WaitAll is evaluated. The kernels receive the jobs but they do not start until I evaluate WaitAll. CloseKernels[]; Needs["ParallelDeveloper"]; ResetQueues[]; LaunchKernels[3]; ids = { ParallelSubmit[runFijiMacro["command line code"]], ParallelSubmit[runFijiMacro["command line code"]], ParallelSubmit[runFijiMacro["command line code"]] }  {QueueRun[],$QueueLength}


{True, 0}

(The returned EvaluationObjects seem to be running but in fact they don't)

WaitAll[ids]


(now FIJI is started and the tasks are processed. But, also the main kernel is blocked.)

Questions

1. Why does this construct not work with Run?
2. Is there a possibility to force the evaluation on the background kernels?
3. Is there a workaround to do this?

Thanks for any help or comments!

-
I think you just need to DistributeDefinitions for runFijiMacro (and potentially other self-defined functions you are using within it). My guess of what happens is that the function is not known in the parallel kernels and they return the unevaluated expression. Only when these are returned to the main kernel with the WaitAll they are evaluated -- sequentially on the main kernel. Search the documentation for ParallelSubmit for DistributeDefinitions to find an example usage and some more details... –  Albert Retey Mar 10 at 13:09
@AlbertRetey: Thanks very much for your fast comment (which is in fact the answer). You guess was totally right. Would you be so kind and post your comment as an answer? –  g3kk0 Mar 10 at 14:01

ParallelSubmit does not automatically distribute the definition from the main kernel to the parallel kernel. So what's happening is that the parallel kernels don't know any definitions for runFijiMacro and return these unevaluated. Only when the main kernel calls WaitAll it will get the yet unevaluated expressions runFijiMacro["command line code"] and evaluate them serially itself. To really evaluate on the parallel kernels you need to distribute the definitions for runFijiMacro and those of other self-defined functions you are using within it with DistributeDefinitions.

More details and an example usage can be found in the documentation for ParallelSubmit when searching for DistributeDefinitions.

-