How to keep building list until manually stopped?

Question

I would like to run a numerical experiment for as much as it can run in a reasonable time.

This is a sketch of how I currently do it:

results = Reap[
 Do[
  If[testQ @ k,
   Sow[transformation @ k]
   ]
  , {k, 2, 1000}]]

I would like to get as much data as possible. If I set the upper bound to 1000, I'll say "that was fast". I set it to 10000 - ok, a bit longer. I set it to 100000 - "hmm, is it ever going to finish or should I kill it?"

What I would like is to set the upper bound to something really big or infinite and just press some Alt+. or other key combination to kill it when I'm tired of waiting.

How should I do it? Should I write in a file instead of Sowing? Or is there something intended for this purpose?

Answer 1

You can use Sow for this in conjunction with CheckAbort:

Reap@CheckAbort[Sow[...], ...]

You can read more about this here

Other than that there is bags. It will be as fast as Sow since Sow is implemented in terms of bags, but you will not lose the content of the bag if you interrupt the computation.

Quick recap:

bag = Internal`Bag[{}] (* Create an empty bag *)
Internal`StuffBag[bag, value] (* Add a value to the bag *)
Internal`BagLength[bag] (* The number of values in the bag *)
Internal`BagPart[bag, part] (* Retrieve values from bag *)

Another option is to use linked lists.

Answer 2

Writing into a globally accessable Association is also an option. I would write to file only

• if there is too much data to keep in memory or
• if the computations take really long and if the risk of a kernel quit is not negliable.

Answer 3

I'm not a big fan of using abort for this..here is a simple idea for a clean exit, create a file and let deleting the file end the loop.

Export["lockfile", 0, "Text"]
i = 0
Reap[While[ FileExistsQ["lockfile"] ,  Pause[1]; Sow[++i] ]][[2, 1]]

when you get bored go into the os and delete the file. Of course this only makes sense if the code itself is slow enough that the time associated with the file check is not significant.