8
$\begingroup$

Possible Duplicate:
Computing Many Slow I/O Operations

I'm launching an external command with the Mathematica front end that takes two arguments:

Run["program", "arg1", "\"S:\\filelocation.data\""]

This operation takes ~10 minutes to complete. While it's running, all other calculations will not evaluate until the external program completes.

In theory, I would want the front end to still be able to evaluate and even to run the kill command for the program

Run["program", "kill"]

I have an 8-core machine, so a parallel solution would work.

UPDATE:

So, as answered below, ParallelSubmit works well if submitted together:

enter image description here

But still doesn't free up the notebook to evaluate other cells:

enter image description here

Improve this question
$\endgroup$
6
  • $\begingroup$ I can't play with this right now but I think you will want to look at ParallelSubmit. $\endgroup$
    – Mr.Wizard
    Oct 23, 2012 at 13:34
  • 1
    $\begingroup$ Isn't this a duplicate of mathematica.stackexchange.com/q/2874/5 ? $\endgroup$
    – rm -rf ♦
    Oct 23, 2012 at 19:23
  • $\begingroup$ @rm-rf It seems so. However, Mr.Wizards answer contains another (if not programatic) approach I didn't know about. $\endgroup$
    – sebhofer
    Oct 24, 2012 at 8:21
  • $\begingroup$ @rm-rf do you think this is one that is a legit Merge? The answers here appear to be neutral. $\endgroup$
    – Mr.Wizard
    Oct 25, 2012 at 0:27
  • $\begingroup$ @Mr.Wizard I don't see a reason to merge. Simply closing should be sufficient. Merging two questions each with accepted answers will result in one answer losing the accept (in this case, sebhofer's). $\endgroup$
    – rm -rf ♦
    Oct 25, 2012 at 1:28

1 Answer 1

Reset to default
7
$\begingroup$

As Mr.Wizard pointed out, you can do this with ParallelSubmit. But you need a little more, and that's the (not so) tricky part as it is not very well documented. I think something like the following should work for you:

Needs["Parallel`Developer`"]
f[x_] := (Pause[x]; x)
LaunchKernels[1]
DistributeDefinitions[f]
eid = ParallelSubmit[f@5]
QueueRun[]

Now you can go do some other work. When you think the external program has finished you do

WaitNext@{eid}

to collect the result.

Let me briefly explain what happens in the above code: First, I launch as many slave kernels as external functions I want to run in parallel (in this case only 1). Then the definition of f is distributed to all slave kernels. The code will also work without this step, you will not gain any speedup, however. (If the definition of f is not distributed, the slave kernels just return f@5 to the master, who will evaluate it and therefore pause for 5 seconds.) ParallelSubmit submits the computation to the queue. At this point the slave kernels are still idling. The crucial step is to run QueueRun, which is loaded from the Developer context. It starts the parallel queue but returns immediately, without waiting for results from the slave kernels. Results are collected by WaitNext (or WaitAll if you want), but as the name says, this does not return until it gets the next result from the slaves, so you can't run other computations at the same time.

To address your problem a bit more precisely, the following worked for me (on Linux that is):

Needs["Parallel`Developer`"]
LaunchKernels[2]
cmd := "echo " <> ToString[$KernelID] <> " >> foo.txt"
DistributeDefinitions[cmd]
eid = {ParallelSubmit@Run[cmd], ParallelSubmit@Run[cmd]}
QueueRun[]
WaitAll[eid]

As I understand it, the functionality in the Parallel`Developer context evolved from the "Parallel Computing Toolkit" by Roman Maeder, which was at some point integrated into main MMA. Unfortunately the documentation largely got lost in the process (which is really a shame!), but it is still available at Wolfram. The example section is definitely worth a read! Note that some commands where renamed though.

Improve this answer
$\endgroup$
6
  • $\begingroup$ So this works lovely if I ParallelSubmit everything, but if I launch this process and then go down the notebook and try to evaluate 2+2 the evaluation won't return until the external command stops. $\endgroup$
    – kale
    Oct 23, 2012 at 18:18
  • $\begingroup$ Did you actually read the answer above the last example?? (I admit I should have put more thought into the last example though... editing) $\endgroup$
    – sebhofer
    Oct 23, 2012 at 18:37
  • $\begingroup$ This looks interesting but how do you know when the background task has completed? $\endgroup$
    – Mr.Wizard
    Oct 23, 2012 at 18:47
  • $\begingroup$ @Mr.Wizard Good question, I can't remember :) I'll have to think about it. Also, there may be a hint in the examples (see edit above). $\endgroup$
    – sebhofer
    Oct 23, 2012 at 18:51
  • $\begingroup$ @Mr.Wizard I think there are several ways to check when the calculations are finished. One of them is to check $QueueLength. Also there is DoneQ, but I couldn't figure out how that works right now. $\endgroup$
    – sebhofer
    Oct 23, 2012 at 19:09

Not the answer you're looking for? Browse other questions tagged or ask your own question.