Recovering intermediate results from a Table after an interruption

I am interested in knowing if it is possible to recover intermediate results from a Table after an interruption.

Let us say I want to produce a really useful table

tt = Table[Pause[0.1*i]; i, {i, 50}]


and since Mathematica takes too long to evaluate it, I get impatient.

Is it possible to interrupt the calculation and recover what it has already calculated? I am guessing the answer is no, but would like to be proven wrong.

• Exact duplicate of this. See also this Jun 28, 2012 at 12:24
• Indeed; sorry about that. The short answer is "it can't be done if it has not been planed" then? Jun 28, 2012 at 12:26
• @LeonidShifrin Not an exact duplicate of the first question you mention. That deals with parallel calcluations in which side effects don't work. However the poster answers this question in the statement of his. Jun 28, 2012 at 12:40
• @chris Looks like that. Jun 28, 2012 at 12:43
• Retagged, please feel free to suggest better tags. Jul 9, 2012 at 14:48

Since I was corrected and this is indeed not an exact duplicate of the parallel computations question, I will reproduce here my implementation of abortable table, to have it here on SE:

ClearAll[abortableTable];
SetAttributes[abortableTable, HoldAll];
abortableTable[expr_, iter : {_Symbol, __} ..] :=
Module[{indices, indexedRes, sowTag, depth =  Length[Hold[iter]] - 1},
Hold[iter] /. {sym_Symbol, __} :> sym /. Hold[syms__] :> (indices := {syms});
indexedRes =  Replace[#, {x_} :> x] &@ Last@Reap[
CheckAbort[Do[Sow[{expr, indices}, sowTag], iter], Null],sowTag];
AbortProtect[
SplitBy[indexedRes, Array[Function[x, #[[2, x]] &], {depth}]][[##,1]] & @@
Table[All, {depth + 1}]
]];


the usage is

tt=abortableTable[Pause[0.1*i];i,{i,50}]

(*  {1,2,3,4,5,6,7,8} *)


(I aborted the computation after some time). Generally, abortableTable accepts the same iterator syntax as Table, and can work with multiple dimensions. The details on how this works can be found in the linked discussion.

EDIT

Per request, a simple abortable Map (which only maps on the first level):

abortableMap[f_, expr_] :=
Module[{sowTag},
Head[expr] @@ If[# === {}, #, First@#] &[
Last[Reap[
CheckAbort[Do[Sow[f[part], sowTag], {part, List @@ expr}],Null]]]]]


The usage is, for example:

abortableMap[(Pause[0.1*#];f[#])&,Range[10]]

(*  {f[1],f[2],f[3]}  *)


(again, I aborted manually soon after it started to compute).

• Just a naive question: why is this not the default then? Jun 28, 2012 at 15:01
• @Chris I think it should not be, because this abortable table has more complex behavior than Table, and this complexity probably can not be justified in most cases, from the langauge design viewpoint. For one thing, it does not produce deterministic results, because the results depend on when you abort the computation. There are other reasons as well. Jun 28, 2012 at 15:22
• An interesting bit of code, and somewhat different than the original you referenced. It's like an onion, the more you peal back, the more there is, +1. Jun 28, 2012 at 15:25
• @rcollyer Thanks :) It is at the bottom of that post, I added it a little later and it probably went largely unnoticed. Jun 28, 2012 at 15:28
• You're right it did. I didn't bother reading past the examples. :) Jun 28, 2012 at 15:32

You could use side effects instead, thus writing

tt = ConstantArray[{}, 50];
Table[Pause[0.1*i]; tt[[i]] = i, {i, 50}]


If you interrupt you will find tt partially filled, and you can then finish the rest of an aborted calculation, by letting i take on the values corresponding to the unfilled places:

Table[Pause[0.1*i]; tt[[i]] = i, {i, Flatten[Position[tt, {}]]}]


Note however that since it relies on side effects, it will for example not work if you Parallelize[] it.

• This is clever! +1 Jun 28, 2012 at 18:10

I'm sure there must be more elegant solutions than this, but if each step takes significant time, then saving the intermediate results to disk ( or better still a RAM disk or solid state disk ) shouldn't be too much of an overhead.

The benefit here is this is partially resistant to some forms of system lockup or crash.

I've used Nest but no reason this couldn't be done with Table as well.

NestWhileList[
With[{res = #[[2]][Last@#]}, #[[2]] >>
"int_res_" <> ToString[Last@#]; {Append[First@#, res], #[[2]],
Last@# - 1}] &, {{}, Sin, 10}, Positive@Last@# & ]


In this example Sin represents a placeholder which can be exchanged for your function of choice.

Reflection suggests that extending this to allow restarting might be a wothwhile exercise.

Update

Kernel Crash Resilient Function Application

Here is a naive, but hopefully useful, stateful solution to applying a time expensive function to a set of parameters or integers that is robust to interrupts and kernel crashes and is similar in spirit to Map or Table.

The approach is to maintain an externally stored state vector of the progression through the computation. The state vector collects the list of results and holds the unevaluated parameters and applied function.

A history option can be used to store the state for each step, allowing computation to be restarted from anywhere in the sequence.

Support is provided for named execution threads.

Basic usage

AirBag[Block[{}, Pause[#]; Print@#; #] &, {1, 3, 5, 7, 5, 3, 2}]


1

3

Out[335]=\$Aborted

AirBag[]


5

7

5

3

2

Out[336]={1, 3, 5, 7, 5, 3, 2}

AirBag[Composition[#^2 &, Plus], {{2, 3}, {3, 3}, {1, 4}, {5, 7}, {6, 7}},
History -> True, FileSpec -> "./air_bag_composite."]


{25, 36, 25, 144, 169}

Restart the named execution thread from an arbitrary step:

AirBag["./air_bag_composite.3"]


{25, 36, 25, 144, 169}

Clear@AirBag;
Options@AirBag = {FileSpec -> "./air_bag_int_res.", History -> False};

(* Process a state vector -> {results list, function, parameter list, \
iteration counter }   *)

AirBag[state_List, OptionsPattern[]] :=
NestWhile[
With[{res = #[[2]][Sequence @@ (First@#[[3]])]}, {Append[First@#,
res], #[[2]], Rest@#[[3]], Last@#} >>
OptionValue@FileSpec <>
If[OptionValue@History, ToString[Last@#], "1"]; {Append[First@#,
res], #[[2]], Rest@#[[3]], Last@# + 1}] &, state,
Length@#[[3]] > 0 &]//First

(* Apply function to the integers 1 to iterations *)

AirBag[function_, iterations_Integer, opts : OptionsPattern[]] :=
AirBag[function, Range@iterations, opts]

(* Apply function to a list of parameter values *)

AirBag[function_, parameters_List, opts : OptionsPattern[]] :=
AirBag[{{}, function, parameters, 0}, opts]

(* Restart from a saved AirBag state *)

AirBag[fileName_String, opts : OptionsPattern[]] :=
If[FileExistsQ@fileName,
With[{state = Get@fileName}, AirBag[state, opts]]]

AirBag[OptionsPattern[]] := AirBag[OptionValue@FileSpec <> "1"]


c = Dynamic@a
a = {};
tt = Table[AppendTo[a, {i, Pause[0.1*i]}]; i, {i, 8}]


Here, most code has to do with "automatic" abortion

kill = 0;