1
$\begingroup$

I am running an iterative routine that I want to export to a file while each iteration is computed, instead of storing everything in memory and then exporting to a file.

My solution is to write to an "m" file that saves the values in the usual array format that mathematica understands (e.g. {{2,1},{3,1}} for a 2x2 matrix with the obvious contents). To do that though, I also need to write the "commas" and the brackets "{","}" manually.

In any case, here is a sample code that achieves that but in a, quite likely, not very clever, efficient and readable way:

sm = 3;
rm = 3;
prior = 0;
SetDirectory[NotebookDirectory[]];
DeleteFile["test.m"]
stream = OpenAppend["test.m"];

Do[next = prior + s + r;
If[r == 1 && s == 1, WriteString[stream, "{"]];
If[r == 1, WriteString[stream, "{"]];
WriteString[stream, ToString[next]];
prior = next;
If[s < sm, If[r == rm, WriteString[stream, "},"]; prior = 0, WriteString[stream, ","]], If[r == rm, WriteString[stream, "}"], WriteString[stream, ","]]];
If[r == rm && s == sm, WriteString[stream, "}"]] ;, {s, 1, sm}, {r, 1, sm}]
Close[stream]

This generates an "m" file that when I open I can immediately process by defining a matrix with the written data to make further analysis later. It looks like this for the above code:

enter image description here

The problem is that my actual code includes three iterating indices (and the actual expression for calculation is much more complex) so the situation becomes very complicated with this simple solution (mainly, too many IF commands that need to be introduced).

So, the question is, is there a way to make this code sorter, more elegant, clever, efficient and readable so that it is easily generalised and debugged?

Note that this question is also related to this question I asked a few days ago.

Thanks.

$\endgroup$
  • 1
    $\begingroup$ What is the reason that you do not or cannot generate entire blocks of output at once? $\endgroup$ – Mr.Wizard Jun 30 '18 at 7:07
  • $\begingroup$ I am not entirely sure I completely understand what you mean. Let me explain what is actually happening in my real code and see. I have three indices (one goes to very big values and this is the reason I do not want to store everything to memory) that are being iterated and at each iteration two quantities are being calculated. The values of the quantities are evaluated after some random process being simulated (some random numbers being generated) and an optimisation of a quantity. Also, the value of the quantities at the previous step are necessary for all evaluations at the current step. $\endgroup$ – AG1123 Jun 30 '18 at 7:48
2
$\begingroup$

Something like this?

The strategy is to pre-generate the delimiters and the coordinates to then have a single loop that writes the value and the next delimiter, whatever that is in each iteration and for whatever the number of dimensions.

Module[
 {
  sm = 3,
  rm = 3,
  stream = OpenWrite["test.m"],
  delim, coord, wf,
  prev = 0,
  expensivefunc
  },
 wf = WriteString[stream, #] &; (* write function *)
 expensivefunc = Function[{x, y}, prev + x + y]; (* heavy task *)
 delim = StringSplit[ExportString[Table["%", {sm}, {rm}], "String"],"%"]; (* pre-calculated delimiters *)
 coord = Flatten[Table[{s, r}, {s, sm}, {r, rm}], 1]; (* pre-calculated parameters *)
 wf[delim[[1]]];  (* Writes first delimiter *)
 Table[
  prev = Apply[expensivefunc, coord[[k]]]; (* calculates *)
  wf[ToString[prev]]; (* Writes value *)
  wf[delim[[k + 1]]]; (* Writes delimiter *)
  , {k, Length[coord]}
  ];
 Close[stream]
 ]
| improve this answer | |
$\endgroup$
  • $\begingroup$ It's not shorter, but I think it's easier to read. $\endgroup$ – rhermans Jun 30 '18 at 14:41
1
$\begingroup$

Assuming one row will fit in memory but the matrix will not, use Table to construct each row and supply the necessary delimiters with the minimum calculation, storage and If.

sm = 3;
rm = 3;
SetDirectory[NotebookDirectory[]];
DeleteFile["test.m"];
stream = OpenAppend["test.m"];
WriteString[stream, "{"];
Do[
  prior = 0; (*initialization for each row*)
  WriteString[stream, ToString[Table[prior = prior + s + r, {s, 1, sm}]]]; (*one row*)
  If[r < rm, WriteString[stream, ",\n"]] (*one row per line*)
  ,{r, 1, rm}
];
WriteString[stream, "}"];
Close[stream]
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.