I am running an iterative routine that I want to export to a file while each iteration is computed, instead of storing everything in memory and then exporting to a file.

My solution is to write to an "m" file that saves the values in the usual array format that mathematica understands (e.g. {{2,1},{3,1}} for a 2x2 matrix with the obvious contents). To do that though, I also need to write the "commas" and the brackets "{","}" manually.

In any case, here is a sample code that achieves that but in a, quite likely, not very clever, efficient and readable way:

sm = 3;
rm = 3;
prior = 0;
stream = OpenAppend["test.m"];

Do[next = prior + s + r;
If[r == 1 && s == 1, WriteString[stream, "{"]];
If[r == 1, WriteString[stream, "{"]];
WriteString[stream, ToString[next]];
prior = next;
If[s < sm, If[r == rm, WriteString[stream, "},"]; prior = 0, WriteString[stream, ","]], If[r == rm, WriteString[stream, "}"], WriteString[stream, ","]]];
If[r == rm && s == sm, WriteString[stream, "}"]] ;, {s, 1, sm}, {r, 1, sm}]

This generates an "m" file that when I open I can immediately process by defining a matrix with the written data to make further analysis later. It looks like this for the above code:

enter image description here

The problem is that my actual code includes three iterating indices (and the actual expression for calculation is much more complex) so the situation becomes very complicated with this simple solution (mainly, too many IF commands that need to be introduced).

So, the question is, is there a way to make this code sorter, more elegant, clever, efficient and readable so that it is easily generalised and debugged?

Note that this question is also related to this question I asked a few days ago.


  • 1
    $\begingroup$ What is the reason that you do not or cannot generate entire blocks of output at once? $\endgroup$
    – Mr.Wizard
    Jun 30, 2018 at 7:07
  • $\begingroup$ I am not entirely sure I completely understand what you mean. Let me explain what is actually happening in my real code and see. I have three indices (one goes to very big values and this is the reason I do not want to store everything to memory) that are being iterated and at each iteration two quantities are being calculated. The values of the quantities are evaluated after some random process being simulated (some random numbers being generated) and an optimisation of a quantity. Also, the value of the quantities at the previous step are necessary for all evaluations at the current step. $\endgroup$
    – AG1123
    Jun 30, 2018 at 7:48

2 Answers 2


Something like this?

The strategy is to pre-generate the delimiters and the coordinates to then have a single loop that writes the value and the next delimiter, whatever that is in each iteration and for whatever the number of dimensions.

  sm = 3,
  rm = 3,
  stream = OpenWrite["test.m"],
  delim, coord, wf,
  prev = 0,
 wf = WriteString[stream, #] &; (* write function *)
 expensivefunc = Function[{x, y}, prev + x + y]; (* heavy task *)
 delim = StringSplit[ExportString[Table["%", {sm}, {rm}], "String"],"%"]; (* pre-calculated delimiters *)
 coord = Flatten[Table[{s, r}, {s, sm}, {r, rm}], 1]; (* pre-calculated parameters *)
 wf[delim[[1]]];  (* Writes first delimiter *)
  prev = Apply[expensivefunc, coord[[k]]]; (* calculates *)
  wf[ToString[prev]]; (* Writes value *)
  wf[delim[[k + 1]]]; (* Writes delimiter *)
  , {k, Length[coord]}
  • $\begingroup$ It's not shorter, but I think it's easier to read. $\endgroup$
    – rhermans
    Jun 30, 2018 at 14:41

Assuming one row will fit in memory but the matrix will not, use Table to construct each row and supply the necessary delimiters with the minimum calculation, storage and If.

sm = 3;
rm = 3;
stream = OpenAppend["test.m"];
WriteString[stream, "{"];
  prior = 0; (*initialization for each row*)
  WriteString[stream, ToString[Table[prior = prior + s + r, {s, 1, sm}]]]; (*one row*)
  If[r < rm, WriteString[stream, ",\n"]] (*one row per line*)
  ,{r, 1, rm}
WriteString[stream, "}"];

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.