# How to determine how much of a table is generated?

If I'm generating a table from a slow function, like this:

foo = Table[SuperSlowExpression, {1000}]


is there a way to determine after submission (so it is too late to instrument the call) to find out how many values already have been generated (so that when it takes longer than expected I can find out whether to wait a bit longer or to abort the calculation)?

So is there still a way to get at the information? Note that there is no iteration variable in the call which I could read.

Optimally, I'd also want to get the already calculated values, of course.

Edit:

Since this seems to have been consistently overlooked:

This is about a calculation which is already running. It is no longer possible to make changes to the call of Table or to the expression inside.

-
I don't know about after, but if you already know that you have SuperSlowExpression to begin with, it seems worthwhile to set up a cache variable beforehand, and then modify SuperSlowExpression to store to the cache (with e.g. Append[]/AppendTo[]) with each iteration within Table[]. –  Ｊ. Ｍ. Feb 8 '12 at 10:41

Given your current situation, there is another option that might help if you have set "Enable notebook history tracking" in Preferences > Advanced:

Then, you go to Cell > Notebook history and navigate to your currently evaluating cell and look at the time stamp when you last edited it. Chances are that you edited it just prior to evaluating (note that if you open an old notebook and execute the cell right away, this might not work, because the last edit timestamp will not be what you want it to be).

Now find the current time from your system clock. Assuming you know how long it takes for one evaluation of SuperSlowExpression, you can now simply do:

$$\mathrm{approx\ progress=\frac{current\ time - last\ edited\ time}{time\ for\ one\ evaluation}}$$

This will give you a rough estimate of the progress — enough to make a decision whether to wait a bit longer or to give up.

Remember, this should be a last ditch option! There are many ifs and buts here, but seeing as all the other options require an iterator and you don't have one (and your Table is running), it's worth trying...

-
Thank you. This seems indeed to be the best option available. And the history tracking was up to now unknown to me (it turned out that it already was enabled, though). While your advice came too late for me (I stopped the calculation before I saw it; however a later, monitored run showed me that it was indeed the correct thing to do), it's really what I should have done (with estimating the time of one step by simply entering a subsession and timing a single evaluation of the expression). –  celtschk Feb 9 '12 at 14:10

Yes, there is. You can use Monitor

Monitor[
Table[Pause[1]; i, {i, 10}],
i
]


Some explanations

Table uses dynamic scoping to localize its variable (i in this case), just like Block. This means that while Table is evaluating, i has a "global" value that one can inspect even from outside the Table if we interrupt the evaluation.

On way of interrupting the evaluation is to request i's value on the pre-emptive link (i.e. using Dynamic expressions) We could just put Dynamic[i] somewhere and it would count up just like Manipulate while the Table is evaluating.

Another way of interrupting the evaluation is through entering a Dialog: You can use the Evaluation -> Interrupt evaluation... menu item, choose Enter subsession. You will get a prompt where you can inspect the kernel state (including the value of i, Stack[], etc.). Evaluate Return[] to resume he evaluation.

I have been using both of these techniques extensively to monitor very lengthy evaluations. The second one even seems to work for parallel evaluations, and least in those cases where I tried it.

Iteration variable and already calculated values

If there is no iteration variable in the Table, just insert one, and use Table[..., {i, 1000}]. Then we can use Monitor or Dynamic[i].

It is not possible to get the already calculated values. If you want to do this (it would often be desirable to be able to!), you need to implement your own version of Table. I asked about how to implement an interruptible parallel Table or Map in this question. The solution I got does not allow inspecting the so far calculated results, but it lets me interrupt the calculation, and continue from where I stopped later.

A table function that allows monitoring partial results

As an example, I provide a custom table function that allows monitoring the partial results.

Try evaluating

Dynamic[tableResults[]]
table[Pause[1]; i^2, {i, 10}]


tableResults[]could be used inMonitor too.

I am using a linked list to collect the results for better performance than what AppendTo would give. I cannot use Sow/Reap here because they don't allow inspecting the partial results (at least I am not aware of this being possible).

Note: If you use nested tables, only the inner one can be monitored, but all of them will work correctly.

The code:

ClearAll[table, tableresult, tableResults]
SetAttributes[table, HoldAll]

tableresult = {};

table[expr_, iterator__] :=
Block[{tableresult, tableelem},
tableresult = {};
Do[
tableresult = {tableresult, tableelem[expr]},
iterator
];
tableResults[]
]

tableResults[] := Block[{tableres, tableelem},
tableres = Flatten[tableresult];
tableelem = Identity;
tableres
]

-
"Note that there is no iteration variable in the call which I could read." - so, how does one use Monitor[] here? –  Ｊ. Ｍ. Feb 8 '12 at 10:47
As I said, it's a calculation which already started (had I had an idea that it could take that long, I would have instrumented it accordingly). That is, I cannot wrap it any more into a monitor or modify the expression. However a +1 for the idea to use a linked list instead of a built-in list for speed. –  celtschk Feb 8 '12 at 11:39
@celtschk So you are running the calculation right now? You're out of luck then. I've been there. –  Szabolcs Feb 8 '12 at 12:05
@celtschk For a few more custom Table functions (conditional Table and Abortable table), see my posts here: stackoverflow.com/questions/6367932/…, and here: stackoverflow.com/questions/6470625/mathematica-table-function/…. They are based on Reap and Sow and don't allow the "real-time" monitoring, but can be modified using @Szabolcs's suggestion, to allow that. They however accept a general syntax of Table, including the multi-dimensional case. –  Leonid Shifrin Feb 8 '12 at 12:27
@celtschk In this context, the reason I gave you that link is not the memoization part but the part of backing things up (persistency), via SQL or disk, so that you don't lose the data in case of Aborts, crashes, etc. –  Leonid Shifrin Feb 8 '12 at 14:12

Similar, with a progress bar:

tEnd = 10;
ProgressIndicator[Dynamic[i], {0, tEnd}]
Table[Pause[1]; i, {i, tEnd}]

-
As I said, the calculation is already started and does not have an iteration variable. But thanks anyway. –  celtschk Feb 8 '12 at 11:32
@celtschk, you can execute a FrontEndTokenExecute with "SubsessionEvaluateCells" and incrementing and i=0 via \$Post. ;-) But you know what: stop the execution and add a i in your table. –  user21 Feb 8 '12 at 12:30
If with "stop" you mean "interrupt", then telling how to add an i after the fact would be exactly an answer to my question. If by "stop" you mean "abort" then that's exactly what I'd like to avoid. I'd hate to lose more than half a day of calculation while thinking that maybe it might have finished the next minute if I didn't kill it. Of course if the answer to my question should be "it's not possible" then of course that is what I'll have to do (actually with a new run I'll then make sure to get the actual data early). –  celtschk Feb 8 '12 at 13:01

If you're using a function that doesn't use iterators such as Scan or Fold you could consider using PrintTemporary, for example

f[x_] := (Pause[2]; Sin[x]);
res = {};
Module[{i = 0, temp},
Scan[(NotebookDelete[temp];
temp = PrintTemporary[++i];
AppendTo[res, f[#]]) &, Range[20]]]

-
If we're already setting a variable to a value, then we can Monitor again... –  Szabolcs Feb 8 '12 at 12:08
@Szabolcs I tried that but couldn't get it to work. –  Heike Feb 8 '12 at 12:24
@Szabolcs Ignore that last comment, I just realized I made a mistake when I tried to Monitor before –  Heike Feb 8 '12 at 12:26

Suppose you're evaluating something slightly different:

Table[Pause[0.1](*SuperSlowExpression*), {i, 1000}]


You can create a cell containing i, and use Evaluation > Evaluate in Subsession to see the current value of i without interrupting the main evaluation. In the case where you don't have an iterator variable to inspect, then the question is whether there is some state or property in your SuperSlowExpression that could be evaluated to get an idea of how much progress has been made.

-
I suggested this as well. But a question, since I don't really know anyone else who uses this actively: is this safe when doing parallel evaluations, especially in the case when TCPIP links are used between kernels? (Is there a chance that something will time out in a MathLink connection, messing up things irreparably?) –  Szabolcs Feb 8 '12 at 16:55