Serializing File Output

Question

Right now my code is something like

Put[
    Table[
        *some calculations*
        {SparseArray[matrix], coordinates}, 

        {i, 1, numIterations}
    ],
    destinationFile
]

As far as I can tell, this requires that the Table finish evaluating in full, which ends up storing a list of numIterations copies of my matrices and coordinates, before it can write those to a file en mass. The issue with this is that I'm working on a research system with very tight memory usage restrictions. This tends to run over the memory cap before it finishes executing. What I would like to be able to do is serialize that output (replacing Table by Do) such that it never has to store more than one copy of the matrix and the coordinates in memory.

Using PutAppend doesn't work because I later need to be able to read this back in as a list in another notebook, and PutAppend places newlines after everything so I can't actually re-create the output format from Put manually. Are there ways to do this that aren't too horrible?

Answer 1

You can (without much trouble) incrementally construct this via OpenWrite, ToString and WriteString / WriteLine.

Unfortunately I'm not sure whether this will actually solve your memory problem, but here you are:

In[91]:= mxStep[i_] := RandomReal[1, {i, 25}]

In[92]:= $file = CreateFile[];

In[93]:= $steps = 2;
$outputStream = OpenWrite@$file;
WriteLine[$outputStream, "{"];
Do[
  WriteString[$outputStream, ToString[mxStep[i], InputForm]];
  If[i < $steps, WriteLine[$outputStream, ","]],
  {i, $steps}
  ];
WriteLine[$outputStream, "}"];
Close@$outputStream;
Get@$file

Out[99]= {{{0.0710782, 0.295713, 0.358965, 0.403495, 0.91221, 
   0.069269, 0.017493, 0.545457, 0.809002, 0.0733453, 0.531084, 
   0.0770646, 0.910339, 0.383475, 0.543017, 0.866425, 0.198109, 
   0.211237, 0.0802018, 0.444478, 0.9013, 0.47711, 0.348436, 0.822508,
    0.124759}}, {{0.440658, 0.00978242, 0.979155, 0.98225, 0.705378, 
   0.00294301, 0.774715, 0.354214, 0.391704, 0.34864, 0.467632, 
   0.960777, 0.47898, 0.910341, 0.174031, 0.267939, 0.855774, 
   0.113104, 0.102345, 0.92824, 0.488277, 0.28265, 0.73727, 0.345047, 
   0.0844829}, {0.50581, 0.5225, 0.648322, 0.289525, 0.617664, 
   0.420004, 0.343535, 0.028053, 0.620158, 0.107122, 0.445789, 
   0.43033, 0.0933331, 0.733211, 0.161032, 0.873127, 0.974476, 
   0.131961, 0.543887, 0.736284, 0.686062, 0.525942, 0.46038, 
   0.613206, 0.836906}}}