1
$\begingroup$

My goal is to read list0.dat, filter the data with a function f, and then write the transformed data to list1.dat, as in the following:

input = <<list0.dat;
output = Map[f[#]&, input];
output >> list1.dat;

However, input and output are both very large lists. (The format is just the usual anlytical expression.) So "No more memory available" always appears. How can I deal with this problem? I need some suggestions for memory control and how to export the transformed data to an external file in real time.

Improve this question
$\endgroup$
5
  • $\begingroup$ Are you sure that f[ ] functions correctly? If you calculate the above lines with a list0.dat that has only 5 entries, does it work? Check the MemoryInUse function and see how it scales with the length of your input list. $\endgroup$ – bill s Mar 14 '13 at 8:40
  • $\begingroup$ Possible duplicate of this question? $\endgroup$ – m_goldberg Mar 14 '13 at 9:56
  • $\begingroup$ Here I provide an explicit worked-out example of solving a similar problem. Not sure if this one is a duplicate of that or other mentioned candidates. $\endgroup$ – Leonid Shifrin Mar 14 '13 at 11:28
  • $\begingroup$ @LeonidShifrin Thanks! In my case, essentially, I just transform a large math expression to another a large expressions. Can you give me some suggestions？ $\endgroup$ – Orders Mar 14 '13 at 12:30
  • $\begingroup$ If your expression is very large, it may make sense to have some serialization mechanism, so that its parts can be read separately. In general, doing this in chunks, similar to what was suggested by Mr.Wizard, on in my linked answer, seems to be the way to go. $\endgroup$ – Leonid Shifrin Mar 14 '13 at 12:47
5
$\begingroup$

This is merely a skeleton answer. I will leave you to implement these ideas as without knowing your data format I cannot give usable code.

  • Open an input stream with OpenRead, and an output stream with OpenWrite or OpenAppend.

  • Read a block of a manageable size from the input stream using ReadList

  • Process this data and apply your function

  • Format the data, if necessary, and write with Write or WriteString

  • Detect the end of the input file; see EndOfFile.

  • Close the input and output streams.

Alternatively you could use PutAppend rather than an output stream; this may be more convenient depending on what format you want.

Improve this answer
$\endgroup$
1
$\begingroup$

If you are working in version 9, there is a function called ImageFileFilter which applies any function f to all the pixels of an image, without reading the whole image into memory. So if your data happened to be an image your problem would be solved. But of course an image is just an array of numbers, so another strategy you could consider would be to convert your data into one of the standard image formats, apply ImageFileFilter to the converted data, and then convert back into your preferred data format.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I like this idea - have you ever demonstrated or got this working - say creating an array of numbers and processing it as PNG or TIFF or JPEG? $\endgroup$ – cormullion Apr 21 '13 at 13:20
  • $\begingroup$ @cormullion -- Not in the offline setting... meaning I often translate between Image format and Imagedata format in Mathematica's working memory, but have never actually tried it on disk. $\endgroup$ – bill s Apr 21 '13 at 13:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.