# Running out of memory when filtering a large data set

My goal is to read list0.dat, filter the data with a function f, and then write the transformed data to list1.dat, as in the following:

input = <<list0.dat;
output = Map[f[#]&, input];
output >> list1.dat;


However, input and output are both very large lists. (The format is just the usual anlytical expression.) So "No more memory available" always appears. How can I deal with this problem? I need some suggestions for memory control and how to export the transformed data to an external file in real time.

-
Are you sure that f[ ] functions correctly? If you calculate the above lines with a list0.dat that has only 5 entries, does it work? Check the MemoryInUse function and see how it scales with the length of your input list. –  bill s Mar 14 '13 at 8:40
Possible duplicate of this question? –  m_goldberg Mar 14 '13 at 9:56
Here I provide an explicit worked-out example of solving a similar problem. Not sure if this one is a duplicate of that or other mentioned candidates. –  Leonid Shifrin Mar 14 '13 at 11:28
@LeonidShifrin Thanks! In my case, essentially, I just transform a large math expression to another a large expressions. Can you give me some suggestions？ –  Orders Mar 14 '13 at 12:30
If your expression is very large, it may make sense to have some serialization mechanism, so that its parts can be read separately. In general, doing this in chunks, similar to what was suggested by Mr.Wizard, on in my linked answer, seems to be the way to go. –  Leonid Shifrin Mar 14 '13 at 12:47

This is merely a skeleton answer. I will leave you to implement these ideas as without knowing your data format I cannot give usable code.

Alternatively you could use PutAppend rather than an output stream; this may be more convenient depending on what format you want.

-

If you are working in version 9, there is a function called ImageFileFilter which applies any function f to all the pixels of an image, without reading the whole image into memory. So if your data happened to be an image your problem would be solved. But of course an image is just an array of numbers, so another strategy you could consider would be to convert your data into one of the standard image formats, apply ImageFileFilter to the converted data, and then convert back into your preferred data format.

-
I like this idea - have you ever demonstrated or got this working - say creating an array of numbers and processing it as PNG or TIFF or JPEG? –  cormullion Apr 21 '13 at 13:20
@cormullion -- Not in the offline setting... meaning I often translate between Image format and Imagedata format in Mathematica's working memory, but have never actually tried it on disk. –  bill s Apr 21 '13 at 13:30