How can I make this code simpler, readable and more efficient?

Question

I am running an iterative routine that I want to export to a file while each iteration is computed, instead of storing everything in memory and then exporting to a file.

My solution is to write to an "m" file that saves the values in the usual array format that mathematica understands (e.g. {{2,1},{3,1}} for a 2x2 matrix with the obvious contents). To do that though, I also need to write the "commas" and the brackets "{","}" manually.

In any case, here is a sample code that achieves that but in a, quite likely, not very clever, efficient and readable way:

sm = 3;
rm = 3;
prior = 0;
SetDirectory[NotebookDirectory[]];
DeleteFile["test.m"]
stream = OpenAppend["test.m"];

Do[next = prior + s + r;
If[r == 1 && s == 1, WriteString[stream, "{"]];
If[r == 1, WriteString[stream, "{"]];
WriteString[stream, ToString[next]];
prior = next;
If[s < sm, If[r == rm, WriteString[stream, "},"]; prior = 0, WriteString[stream, ","]], If[r == rm, WriteString[stream, "}"], WriteString[stream, ","]]];
If[r == rm && s == sm, WriteString[stream, "}"]] ;, {s, 1, sm}, {r, 1, sm}]
Close[stream]

This generates an "m" file that when I open I can immediately process by defining a matrix with the written data to make further analysis later. It looks like this for the above code:

The problem is that my actual code includes three iterating indices (and the actual expression for calculation is much more complex) so the situation becomes very complicated with this simple solution (mainly, too many IF commands that need to be introduced).

So, the question is, is there a way to make this code sorter, more elegant, clever, efficient and readable so that it is easily generalised and debugged?

Note that this question is also related to this question I asked a few days ago.

Thanks.

Answer 1

Something like this?

The strategy is to pre-generate the delimiters and the coordinates to then have a single loop that writes the value and the next delimiter, whatever that is in each iteration and for whatever the number of dimensions.

Module[
 {
  sm = 3,
  rm = 3,
  stream = OpenWrite["test.m"],
  delim, coord, wf,
  prev = 0,
  expensivefunc
  },
 wf = WriteString[stream, #] &; (* write function *)
 expensivefunc = Function[{x, y}, prev + x + y]; (* heavy task *)
 delim = StringSplit[ExportString[Table["%", {sm}, {rm}], "String"],"%"]; (* pre-calculated delimiters *)
 coord = Flatten[Table[{s, r}, {s, sm}, {r, rm}], 1]; (* pre-calculated parameters *)
 wf[delim[[1]]];  (* Writes first delimiter *)
 Table[
  prev = Apply[expensivefunc, coord[[k]]]; (* calculates *)
  wf[ToString[prev]]; (* Writes value *)
  wf[delim[[k + 1]]]; (* Writes delimiter *)
  , {k, Length[coord]}
  ];
 Close[stream]
 ]

Answer 2

Assuming one row will fit in memory but the matrix will not, use Table to construct each row and supply the necessary delimiters with the minimum calculation, storage and If.

sm = 3;
rm = 3;
SetDirectory[NotebookDirectory[]];
DeleteFile["test.m"];
stream = OpenAppend["test.m"];
WriteString[stream, "{"];
Do[
  prior = 0; (*initialization for each row*)
  WriteString[stream, ToString[Table[prior = prior + s + r, {s, 1, sm}]]]; (*one row*)
  If[r < rm, WriteString[stream, ",\n"]] (*one row per line*)
  ,{r, 1, rm}
];
WriteString[stream, "}"];
Close[stream]