Say I'm running some process of unknown duration, and during the process, I need to periodically record the state of some variable. If we knew exactly how long the process took, we could simply specify an array of appropriate size, and know that when the array is filled, we'll be finished. However, if this is not the case, it seems one has to resort to operations like:

testArray = Append[testArray,dataPoint];

Which becomes both computationally intensive and memory intensive at the limit of large array sizes. Is there a better way to proceed? Or should I simply specify a very large version of testArray and then chop it down after I've finished my procedure and filled it to some limited extent?

Let me provide an example of the kind of process I'm thinking of (using the horrible Append strategy just to make things clear):


storage = Append[storage,randVar];

How do we use e.g. Sow/Reap to deal with the array updates?

  • $\begingroup$ Just build a linked list, e.g., start with myList={}, then at each update do myList={update,myList}. You can flatten/manipulate as needed to collect to final result. Or, use Sow and Reap. $\endgroup$
    – ciao
    Commented Jan 24, 2014 at 4:45
  • $\begingroup$ @rasher Great, do you know which method is fastest? $\endgroup$
    – 110110
    Commented Jan 24, 2014 at 4:47
  • $\begingroup$ @rasher And how can I avoid nesting behavior? Should I flatten at every step or just not worry about this? It seems wasteful. And does repeatedly applying Flatten lead to a slowdown? $\endgroup$
    – 110110
    Commented Jan 24, 2014 at 4:48
  • $\begingroup$ Probably Sow/Reap, but not a huge difference. For your question, Sow/Reap makes the most sense. As far as the second query, whenever possible, do list operations as one, so one flatten will beat repeated. $\endgroup$
    – ciao
    Commented Jan 24, 2014 at 4:51
  • $\begingroup$ @rasher If you wouldn't mind, could you show me an example of using Sow/Reap? I'm reading through the manual now, and I'm not 100% sure I know what you have in mind. $\endgroup$
    – 110110
    Commented Jan 24, 2014 at 4:53

2 Answers 2


Use Sow and Reap. Here's a contrived example:

{result, reaped} = Reap[Map[(Sow[#]; #!) &, {1, 2, 3, 4, 5}]]

So, we are going through the list of numbers 1 to 5, computing factorial. We want the result, and what the number was. The Sow puts each into the queue as we do our work, the Reap gets the results, along with whatever was Sown. If you removed the Reap, just the results come back. Quite useful for debugging!

Very powerful construct, I urge you to study it, it has flexibility such as labeling items, etc. and is quite fast and cheap on resources.

Per your update:

randVar = 0

{nothing, storage} = 
  Reap[While[randVar <= 0.99, randVar = RandomReal[];


Note that if something like this example is really what you're doing, much more efficient to generate variates in bulk and grab until criteria is met:

storage = TakeWhile[RandomReal[1, 1000000], # <= .99 &]
  • $\begingroup$ This is actually quite cool. However, how would I use this to periodically enter an unspecified number of data points into an array structure of flexible size? $\endgroup$
    – 110110
    Commented Jan 24, 2014 at 5:02
  • $\begingroup$ Perhaps I'm not understanding your original question - if so, I'll delete my answer, but that is precisely what Sow and Reap do. Are you implying you need to examine the array contents during the process? If so, making a linked-list is a more plausible solution. $\endgroup$
    – ciao
    Commented Jan 24, 2014 at 5:07
  • $\begingroup$ Probably this is just me being slow, let me provide an example... $\endgroup$
    – 110110
    Commented Jan 24, 2014 at 5:16
  • $\begingroup$ I provided an example of what I meant in the question posting. $\endgroup$
    – 110110
    Commented Jan 24, 2014 at 5:20
  • $\begingroup$ @110110: see updated answer. $\endgroup$
    – ciao
    Commented Jan 24, 2014 at 7:14

Instead of var = Append[var, element] you can use AppendTo[var,element] which is a little bit terser, but not less intensive. Indeed it is recommended to use Sow and Reap instead of Append/AppendTo. A third option would be to use linked lists since with that method you do not copy the entire list every time and so it does not have the same problems as Append does. Append basically makes a copy of the list to add something to it. At the end you can simply flatten the list to get it the way you want it. For example:

randVar = 0;
storage = {};
While[randVar <= .99,
 randVar = RandomReal[];
 storage = {randVar, storage};

Not the answer you're looking for? Browse other questions tagged or ask your own question.