Recovering intermediate results from a Table after an interruption

Question

I am interested in knowing if it is possible to recover intermediate results from a Table after an interruption.

Let us say I want to produce a really useful table

tt = Table[Pause[0.1*i]; i, {i, 50}]

and since Mathematica takes too long to evaluate it, I get impatient.

Is it possible to interrupt the calculation and recover what it has already calculated? I am guessing the answer is no, but would like to be proven wrong.

Answer 1

Since I was corrected and this is indeed not an exact duplicate of the parallel computations question, I will reproduce here my implementation of abortable table, to have it here on SE:

ClearAll[abortableTable];
SetAttributes[abortableTable, HoldAll];
abortableTable[expr_, iter : {_Symbol, __} ..] :=
  Module[{indices, indexedRes, sowTag, depth =  Length[Hold[iter]] - 1},
   Hold[iter] /. {sym_Symbol, __} :> sym /. Hold[syms__] :> (indices := {syms});
   indexedRes =  Replace[#, {x_} :> x] &@ Last@Reap[
      CheckAbort[Do[Sow[{expr, indices}, sowTag], iter], Null],sowTag];
   AbortProtect[
      SplitBy[indexedRes, Array[Function[x, #[[2, x]] &], {depth}]][[##,1]] & @@ 
      Table[All, {depth + 1}]
   ]];

the usage is

tt=abortableTable[Pause[0.1*i];i,{i,50}]

(*  {1,2,3,4,5,6,7,8} *)

(I aborted the computation after some time). Generally, abortableTable accepts the same iterator syntax as Table, and can work with multiple dimensions. The details on how this works can be found in the linked discussion.

EDIT

Per request, a simple abortable Map (which only maps on the first level):

abortableMap[f_, expr_] :=
 Module[{sowTag},
   Head[expr] @@ If[# === {}, #, First@#] &[
    Last[Reap[
       CheckAbort[Do[Sow[f[part], sowTag], {part, List @@ expr}],Null]]]]]

The usage is, for example:

abortableMap[(Pause[0.1*#];f[#])&,Range[10]]

(*  {f[1],f[2],f[3]}  *)

(again, I aborted manually soon after it started to compute).

Answer 2

You could use side effects instead, thus writing

tt = ConstantArray[{}, 50];
Table[Pause[0.1*i]; tt[[i]] = i, {i, 50}]

If you interrupt you will find tt partially filled, and you can then finish the rest of an aborted calculation, by letting i take on the values corresponding to the unfilled places:

Table[Pause[0.1*i]; tt[[i]] = i, {i, Flatten[Position[tt, {}]]}]

Note however that since it relies on side effects, it will for example not work if you Parallelize[] it.

Answer 3

I'm sure there must be more elegant solutions than this, but if each step takes significant time, then saving the intermediate results to disk ( or better still a RAM disk or solid state disk ) shouldn't be too much of an overhead.

The benefit here is this is partially resistant to some forms of system lockup or crash.

I've used Nest but no reason this couldn't be done with Table as well.

NestWhileList[
 With[{res = #[[2]][Last@#]}, #[[2]] >> 
    "int_res_" <> ToString[Last@#]; {Append[First@#, res], #[[2]], 
    Last@# - 1}] &, {{}, Sin, 10}, Positive@Last@# & ]

In this example Sin represents a placeholder which can be exchanged for your function of choice.

Reflection suggests that extending this to allow restarting might be a wothwhile exercise.

Update

Kernel Crash Resilient Function Application

Here is a naive, but hopefully useful, stateful solution to applying a time expensive function to a set of parameters or integers that is robust to interrupts and kernel crashes and is similar in spirit to Map or Table.

The approach is to maintain an externally stored state vector of the progression through the computation. The state vector collects the list of results and holds the unevaluated parameters and applied function.

A history option can be used to store the state for each step, allowing computation to be restarted from anywhere in the sequence.

Support is provided for named execution threads.

Basic usage

AirBag[Block[{}, Pause[#]; Print@#; #] &, {1, 3, 5, 7, 5, 3, 2}]

1

3

Out[335]=$Aborted

AirBag[]

5

7

5

3

2

Out[336]={1, 3, 5, 7, 5, 3, 2}

Named execution threads

AirBag[Composition[#^2 &, Plus], {{2, 3}, {3, 3}, {1, 4}, {5, 7}, {6, 7}}, 
       History -> True, FileSpec -> "./air_bag_composite."]

{25, 36, 25, 144, 169}

Restart the named execution thread from an arbitrary step:

AirBag["./air_bag_composite.3"]

{25, 36, 25, 144, 169}

Clear@AirBag;
Options@AirBag = {FileSpec -> "./air_bag_int_res.", History -> False};

(* Process a state vector -> {results list, function, parameter list, \
iteration counter }   *)

AirBag[state_List, OptionsPattern[]] := 
 NestWhile[
  With[{res = #[[2]][Sequence @@ (First@#[[3]])]}, {Append[First@#, 
       res], #[[2]], Rest@#[[3]], Last@#} >> 
     OptionValue@FileSpec <> 
      If[OptionValue@History, ToString[Last@#], "1"]; {Append[First@#,
       res], #[[2]], Rest@#[[3]], Last@# + 1}] &, state, 
  Length@#[[3]] > 0 &]//First

(* Apply function to the integers 1 to iterations *)

AirBag[function_, iterations_Integer, opts : OptionsPattern[]] := 
 AirBag[function, Range@iterations, opts]

(* Apply function to a list of parameter values *)

AirBag[function_, parameters_List, opts : OptionsPattern[]] := 
 AirBag[{{}, function, parameters, 0}, opts]

(* Restart from a saved AirBag state *)

AirBag[fileName_String, opts : OptionsPattern[]] := 
 If[FileExistsQ@fileName, 
  With[{state = Get@fileName}, AirBag[state, opts]]]

AirBag[OptionsPattern[]] := AirBag[OptionValue@FileSpec <> "1"]

Answer 4

Perhaps something like this may help you.

c = Dynamic@a
a = {};
tt = Table[AppendTo[a, {i, Pause[0.1*i]}]; i, {i, 8}]

Here, most code has to do with "automatic" abortion

kill = 0;
t = CreateScheduledTask[kill = 1];
StartScheduledTask[t];
c = Dynamic@a
a = {};
tt = Table[If[kill == 1, Abort[], AppendTo[a, {i, Pause[0.1*i]}]];  i, {i, 8}]
RemoveScheduledTask[t];