Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I have a loop which computes a table, tab, of results at each iteration. This table is usually very large and I would like to use PutAppendor anything of the sort to save the data in a file and therefore saving memory. An example would look like the following:

Put["test.csv"];

Do[tab=RandomReal[{0,1},10];PutAppend[tab, "test.csv"],{5}]

When I import the data in test.csv for later use I do this via data=Flatten[Import[".../test.csv","Data"],1]. However, the list data is not ready to use because typical elements look e.g. like

In[45]:= data[[1]]

Out[45]= "{0.3105121460958056"

In[43]:= data[[2]]

Out[43]= 0.0657094

In[44]:= data[[4]]

Out[44]= " "

Which is a mixture of strings and reals.

So my question is: how can I append the table tab at each iteration of the loop to a file which later is ready to use as a list of rank two of reals?

Thanks for your help.

share|improve this question
add comment

1 Answer

up vote 5 down vote accepted

This appears to be working:

Do[tab = RandomReal[{0, 1}, 10]; PutAppend[tab, "test123.csv"], {5}]

ReadList["test123.csv", Expression]

You may also wish to look at WriteString which will allow for complete control of what is written.


In a comment Ajasja makes an excellent point regarding performance. If your application is anything like the dummy above, with little data being written in each pass of a fast loop, you should be using streams. This is somewhat less convenient in that you must remember to Close the stream when you are done but it is much faster.

You can use either PutAppend or WriteString with streams, though use with PutAppend is not well documented. I'll show both and perform a three-way timing for comparison.

Do[tab = RandomReal[{0, 1}, 10]; PutAppend[tab, "testAppend1.csv"], {5000}] // AbsoluteTiming

{4.8828086, Null}

str = OpenAppend["testAppend2.csv"];
Do[tab = RandomReal[{0, 1}, 10]; PutAppend[tab, str], {5000}] // AbsoluteTiming
Close[str];

{0.2340004, Null}

str = OpenAppend["testAppend3.csv"];
Do[tab = RandomReal[{0, 1}, 10]; WriteString[str, ToString[tab, InputForm], "\n"], {5000}] //
  AbsoluteTiming
Close[str];

{0.2028004, Null}

Notice that with WriteString I needed to use ToString -- this is because WriteString will convert into OutputForm by default, which truncates digits.

For the sake of convenience I would use PutAppend myself, despite the lack of documentation and the slightly slower speed.

share|improve this answer
    
Thanks a lot. I really appreciate it! –  chris Oct 19 '12 at 11:02
    
@chris Glad I could help. –  Mr.Wizard Oct 19 '12 at 11:23
2  
@chris Also using WriteString and opening the file stream only once is likely to be much faster. –  Ajasja Oct 19 '12 at 12:10
    
@Ajasja I updated my answer in response to your comment. Please take a look and tell me if there is anything you would add. –  Mr.Wizard Oct 19 '12 at 12:39
    
@Mr.Wizard I think you included the necessary material. I'd give you another +1 if I could. –  Ajasja Oct 19 '12 at 13:01
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.