3
$\begingroup$

I have a handful of files (some of which are very large), and I would like to loop through them one line at a time, run a calculation, and then write the result of the calculation to a new output file:

Table[
file=OpenRead[fileName];
line=ReadLine[file];
While[Not[SameQ[line,EndOfFile]],
    newLine=ComputeSomething[line];
    WriteLine[outputFileName,newLine];
    line=ReadLine[file]];
Close[file],
{fileName,listOfFileNames}]

After running this code for a couple of days (on a system with 128GB of RAM) it crashes my system. I get the following message:

No more memory available.
Mathematica kernel has shut down.
Try quitting other applications and then retry.

I'm trying to figure out what is eating up all of the memory. The function ComputeSomething[] rewrites over all of its internal variables, so I don't think it can be that. What I'm guessing is that I have the output file open the whole time, and as it gets larger I'm filling up my RAM?

Either that, or the results of ReadLine[file] get cached somewhere, and never actually cleaned out.

Any suggestions to avoid this problem?

Also, a related question. Is there a way to call a ReadLine[] type function, but specifying a line I want it to read (instead of just reading in the next line)?

$\endgroup$
  • 1
    $\begingroup$ Have you tried the For function to implement the loop? $\endgroup$ – Please Correct GrammarMistakes Jan 30 at 23:28
  • $\begingroup$ Do you mean instead of the 'While' loop? I haven't tried that. Would you expect a 'For' loop to handle memory management differently? $\endgroup$ – Mark Hughes Jan 31 at 0:17
  • $\begingroup$ I mean replacing Table with For saving memory. $\endgroup$ – Please Correct GrammarMistakes Jan 31 at 0:21
  • 6
    $\begingroup$ @PleaseCorrectGrammarMistakes that is never a good idea. Use Do instead of For in all situations. $\endgroup$ – b3m2a1 Jan 31 at 0:40
  • 1
    $\begingroup$ @pl Using For or Do instead of Table won't save memory, I'm afraid. Table is essentially a loop, and the output of Table almost counts nothing in this case. Then, it's hard to give more advice without a minimal example that reproduces the issue, so I'm voting to close this question. $\endgroup$ – xzczd Jan 31 at 4:01

Browse other questions tagged or ask your own question.