Specifying an array that may have an unknown number of entries [duplicate]

Question

Say I'm running some process of unknown duration, and during the process, I need to periodically record the state of some variable. If we knew exactly how long the process took, we could simply specify an array of appropriate size, and know that when the array is filled, we'll be finished. However, if this is not the case, it seems one has to resort to operations like:

testArray = Append[testArray,dataPoint];

Which becomes both computationally intensive and memory intensive at the limit of large array sizes. Is there a better way to proceed? Or should I simply specify a very large version of testArray and then chop it down after I've finished my procedure and filled it to some limited extent?

Let me provide an example of the kind of process I'm thinking of (using the horrible Append strategy just to make things clear):

randVar=0;
storage={};

While[randVar<=0.99,
randVar=RandomReal[];
storage = Append[storage,randVar];
];
Length[storage];

How do we use e.g. Sow/Reap to deal with the array updates?

Answer 1

Use Sow and Reap. Here's a contrived example:

{result, reaped} = Reap[Map[(Sow[#]; #!) &, {1, 2, 3, 4, 5}]]

So, we are going through the list of numbers 1 to 5, computing factorial. We want the result, and what the number was. The Sow puts each into the queue as we do our work, the Reap gets the results, along with whatever was Sown. If you removed the Reap, just the results come back. Quite useful for debugging!

Very powerful construct, I urge you to study it, it has flexibility such as labeling items, etc. and is quite fast and cheap on resources.

Per your update:

randVar = 0

{nothing, storage} = 
  Reap[While[randVar <= 0.99, randVar = RandomReal[];
     Sow[randVar]];];

Length[storage[[1]]]

Note that if something like this example is really what you're doing, much more efficient to generate variates in bulk and grab until criteria is met:

storage = TakeWhile[RandomReal[1, 1000000], # <= .99 &]

Answer 2

Instead of var = Append[var, element] you can use AppendTo[var,element] which is a little bit terser, but not less intensive. Indeed it is recommended to use Sow and Reap instead of Append/AppendTo. A third option would be to use linked lists since with that method you do not copy the entire list every time and so it does not have the same problems as Append does. Append basically makes a copy of the list to add something to it. At the end you can simply flatten the list to get it the way you want it. For example:

randVar = 0;
storage = {};
While[randVar <= .99,
 randVar = RandomReal[];
 storage = {randVar, storage};
 ]
Flatten@storage