I have a loop which computes a table, tab, of results at each iteration. This table is usually very large and I would like to use PutAppendor anything of the sort to save the data in a file and therefore saving memory. An example would look like the following:


Do[tab=RandomReal[{0,1},10];PutAppend[tab, "test.csv"],{5}]

When I import the data in test.csv for later use I do this via data=Flatten[Import[".../test.csv","Data"],1]. However, the list data is not ready to use because typical elements look e.g. like

In[45]:= data[[1]]

Out[45]= "{0.3105121460958056"

In[43]:= data[[2]]

Out[43]= 0.0657094

In[44]:= data[[4]]

Out[44]= " "

Which is a mixture of strings and reals.

So my question is: how can I append the table tab at each iteration of the loop to a file which later is ready to use as a list of rank two of reals?

Thanks for your help.


1 Answer 1


This appears to be working:

Do[tab = RandomReal[{0, 1}, 10]; PutAppend[tab, "test123.csv"], {5}]

ReadList["test123.csv", Expression]

You may also wish to look at WriteString which will allow for complete control of what is written.

In a comment Ajasja makes an excellent point regarding performance. If your application is anything like the dummy above, with little data being written in each pass of a fast loop, you should be using streams. This is somewhat less convenient in that you must remember to Close the stream when you are done but it is much faster.

You can use either PutAppend or WriteString with streams, though use with PutAppend is not well documented. I'll show both and perform a three-way timing for comparison.

Do[tab = RandomReal[{0, 1}, 10]; PutAppend[tab, "testAppend1.csv"], {5000}] // AbsoluteTiming

{4.8828086, Null}

str = OpenAppend["testAppend2.csv"];
Do[tab = RandomReal[{0, 1}, 10]; PutAppend[tab, str], {5000}] // AbsoluteTiming

{0.2340004, Null}

str = OpenAppend["testAppend3.csv"];
Do[tab = RandomReal[{0, 1}, 10]; WriteString[str, ToString[tab, InputForm], "\n"], {5000}] //

{0.2028004, Null}

Notice that with WriteString I needed to use ToString -- this is because WriteString will convert into OutputForm by default, which truncates digits.

For the sake of convenience I would use PutAppend myself, despite the lack of documentation and the slightly slower speed.

  • $\begingroup$ Thanks a lot. I really appreciate it! $\endgroup$
    – chris
    Oct 19, 2012 at 11:02
  • $\begingroup$ @chris Glad I could help. $\endgroup$
    – Mr.Wizard
    Oct 19, 2012 at 11:23
  • 2
    $\begingroup$ @chris Also using WriteString and opening the file stream only once is likely to be much faster. $\endgroup$
    – Ajasja
    Oct 19, 2012 at 12:10
  • $\begingroup$ @Ajasja I updated my answer in response to your comment. Please take a look and tell me if there is anything you would add. $\endgroup$
    – Mr.Wizard
    Oct 19, 2012 at 12:39
  • $\begingroup$ @Mr.Wizard I think you included the necessary material. I'd give you another +1 if I could. $\endgroup$
    – Ajasja
    Oct 19, 2012 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.