Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Currently we have a limited number of ways to perform asynchronous evaluation.

The most common is through Dynamic and Manipulate.

Using Dynamic, we can have part of a cell update independent of whatever we're working on:

Dynamic["x = " <> ToString@x]
For[i = 0, i < 100, i++, Pause[0.05]; x = i^2];

Similarly, we can use Manipulate to continuously evaluate an expression and output the result:

Manipulate[
 {r, v} = {r + dt v, v - dt r};
 Show[Graphics@Point[r], PlotRange -> {{-2, +2}, {-2, +2}}],
 {{dt, 0.01}, 0.01, 1},

 Initialization -> {
   r = {0, 1};
   v = {1, 0};
   }]

We also have a way to submit jobs and have them work independently of each, however it does block any further input:

jobs = Table[
   ParallelSubmit[
    SingularValueList[RandomReal[1, {1000, 1000}]]],
   {10}];

WaitAll[jobs]

The third option gets the closest to being an asynchronous evaluation, however it fails in that it requires any further input to be blocked. What I would love to see is some function, tentatively named AsynchronousEvaluate, which does exactly what it name says:

AsynchronousEvaluate[Pause[10];]
Print["Hello!"]; (* Printed immediately *)

Is there any way we can get close to achieving this?

My dream is to be able to queue up a job for processing on a parallel kernel with certain constraints (Like MemoryConstrained) and be able to just "set and forget." When processing is finished, the result will be returned to me. But in the mean time I can still be productive by doing something else.

Asynchronous evaluation is the only piece that's missing from this, and I'd like to do it if it's possible.

share|improve this question
    
What should be done with the result from AsynchronousEvaluate? –  Brett Champion Jan 19 '12 at 4:17
    
I would think it would be output below the cell where it started evaluating. ParallelSubmit has this behavior when you call WaitNext or WaitAll. –  Mike Bantegui Jan 19 '12 at 4:39
    
When you asked this on StackOverflow a month ago, I mentioned Parallel`Developer`QueueRun[] in my answer. Can you please explain why that solution doesn't work for your problem? It seems to me it'd do exactly what you're asking for. –  Szabolcs Jan 19 '12 at 7:19
    
@Szabolcs: I thought it did what I needed, but after I really looked into it I realized it still had the blocking behavior that WaitNext and WaitAll have. –  Mike Bantegui Jan 19 '12 at 7:21
1  
@Szabolcs: I can't tell whether this is relevant to OP's issue, but it appears that interprocess communication (SetSharedVariable, SetSharedFunction) is handled through QueueRun. "Asynchronous" processes which evaluate shared expressions will hang until QueueRun is next executed. –  Tobias Hagge Feb 6 '13 at 3:19
show 1 more comment

3 Answers

First regarding Dynamic and related functions:

If I understand right, the Mathematica kernels has two evaluation "channels", one used by main evaluations submitted by SHIFT-ENTER, and one for Dynamic things (called preemptive evaluations). Preemptive evaluations can interrupt main evaluations, but we still have only two "channels", and an already running preemptive evaluation can't be interrupted again. So this is not suitable for asynchronous evaluation.


When you asked a very similar question on StackOverflow, I pointed you to Parallel`Developer`QueueRun[]. You have not explained in this new question whether that works for the problem, but it seems to me that it answers the question "How to run a job in the background without blocking further input?" perfectly.


The original answer:

I have next to zero experience with parallel computation in Mathematica, so this might not be the best way, but this is what I managed to dig up from the docs:

Launch a kernel:

In[1]:= LaunchKernels[1]

Out[1]= KernelObject[1, "local"]

Submit some long to finish job:

In[2]:= job = 
 ParallelSubmit[First@SingularValueList[RandomReal[1, {2000, 2000}]]]

Mathematica graphics

Start job:

In[3]:= Parallel`Developer`QueueRun[]

Out[3]= True

Now the job is running in parallel in the background ...

Mathematica graphics

... and we're free to do whatever we want in the main kernel. If I understand your question, this is what you needed. We can run Parallel`Developer`QueueRun[] again to check which parallel evaluations have finished (the display of the evaluation object will dynamically update).

In[4]:= 1 + 1

Out[4]= 2

Wait until the evaluation finishes (if it hasn't yet) and collect the result:

In[5]:= WaitAll[job]

Out[5]= 1000.23

Mathematica graphics

share|improve this answer
    
hm, this doesn't seem to work on my machine. let me try to see what is going on –  acl Jan 19 '12 at 11:10
    
@acl What happens on your machine? It works here in 8.0.4 in a fresh kernel, but once it happened that the EvaluationObjects didn't render right. They did work but they didn't render as shown here. I still don't know why, today they work fine. –  Szabolcs Jan 20 '12 at 23:37
    
hm, good question. it works fine now. who knows what happened yesterday. very useful, +1 –  acl Jan 20 '12 at 23:48
add comment

By using the ideas of @Szabolcs, I've managed to write a convenient function which almost does what you want (edit: there is a better version of this function in the third edit, that does exactly what you want):

ClearAll[AsynchronousEvaluate];
SetAttributes[AsynchronousEvaluate, HoldAll];
AsynchronousEvaluate[exp_] := DynamicModule[{eval, display},
  display = EventHandler[
    eval= ParallelSubmit[exp],
    {"MouseClicked" :> (display = WaitAll[eval])}];
  Parallel`Developer`QueueRun[];
  Dynamic[display]]

What this function does is put the expression you want to evaluate in the queue, and makes it run in the background. In the mean while it displays the EvaluationObject that the function ParallelSubmit outputs, and you are free to perform other operations in the same Mathematica notebook.
Once the evaluation of the expression is complete, all you have to do is click the EvaluationObject, and it will be replaced by the result.

Of course it would be better if the EvaluationObject would be automatically replaced by the result when the evaluation finishes, but I don't know how to do that. If anyone thinks of way - feel free to edit my answer and add it (edit: I already found a way, see below).

Edit: By using $Pre = AsynchronousEvaluate; you can get Mathematica to automatically apply the function AsynchronousEvaluate to every expression you evaluate, saving you some time and making the notebook look neater.

Edit #2: Be careful when assigning a function to $Pre though, the only way to change it back is to restart Mathematica, since trying to evaluate $Pre=. will result in sending the command to one of the parallel kernels :)

Edit #3: OK here it is, the same function, only now it automatically replaces the EvaluationObject by the result when the evaluation finishes. In order for it to work you need to have a scheduled task running in the background:

qRunTask = CreateScheduledTask[Parallel`Developer`QueueRun[]];
StartScheduledTask[qRunTask];

Keeping the default interval of one second for the scheduled task seems reasonable. Now the AsynchronousEvaluate function:

ClearAll[AsynchronousEvaluate];
SetAttributes[AsynchronousEvaluate, HoldAll];
AsynchronousEvaluate[exp_] := DynamicModule[{eval,display},
display = eval = ParallelSubmit[exp];
Dynamic[
  If[MatchQ[eval[[4]],
    Parallel`Developer`finished[_]], display = eval[[4]][[1]]]; display]]

That's it!
If you're not using this function for a while, you can stop the scheduled task (though it doesn't cause any slow down in my experience):

StopScheduledTask[qRunTask];

Edit #4: I improved the function to fix a problem where the results did not persist after restarting Mathematica. Now they do.

share|improve this answer
add comment

If it is not important that you share variables during the calculation, maybe one solution would be to start a separate kernel to calculate (using LinkLaunch) and have the asynchronous calculation run there.

One disadvantage of this method is of course that it uses up another kernel license.

share|improve this answer
    
Interesting. I'll have to look into that. I don't mind using up a second kernel license. –  Mike Bantegui Jan 19 '12 at 6:59
    
@Mike If you want to reimplement this yourself, independent of the existing Parallel* functionality, a good starting point is this doc page. It shows you how to launch another kernel, send an expression to it to evaluate and retrieve the result. To make this practically useful (for general use), you'd need to implement something like DistributeDefinitions as well. When using links, take care to check them with LinkReadyQ before reading from them to avoid freezing Mathematica up. –  Szabolcs Jan 19 '12 at 14:01
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.