Exporting result of calculation to file at each step of iterative routine

I am performing a calculation that involves a triple "For" command. My goal is to export the table of all values of some function (which depends on the three indices) to a data file for statistical analysis at a later stage.

The problem: if I first execute the code, storing everything into a matrix which is afterwards exported to a file, it often makes my computer crash or run out of memory, since it has to store all values into memory. However, the calculation of the iterative routine only depends on the previous value of the function and thus I would like to modify my code so that I write the result of the calculation at each iteration to a file and then continue without storing much into memory.

So, what I would like some advice with, is how can I write into the file at each step of the iterative routine in such a way that the resulting file in the end is identical to the one that would have been produced by keeping all values into some matrix and then saving the whole matrix into a file.

Here is a very simplistic code, which however captures the essence of my actual code:

SetDirectory[NotebookDirectory[]];
For[s = 0, s < 6, s++, Test[s, 0] = 0];
For[s = 1, s < 6, s++, For[r = 1, r < 6, r++, Test[s, r] = Test[s, r - 1] + s + r; ]; ]
Export["Test.dat", Table[Test[x, y], {x, 1, 5}, {y, 1, 5}]];


What I would like to do is write the result of my calculation at each iteration, thus dropping the need to store every value into the matrix "Test[s,r]" of the example. However, I would like to do it in such a way so that the resulting files are identical (assuming this is possible).

Any help appreciated. Thanks!

This uses your style of coding and should produce a file identical to your Test.dat without constructing the entire data structure in memory.

You must ensure that the file Test2.dat does not exist in the directory prior to running this because this code will append to any existing Test2.dat file.

SetDirectory[NotebookDirectory[]];
stream = OpenAppend["Test2.dat"];
For[s = 1, s < 6, s++,
prior = 0;
For[r = 1, r < 6, r++,
next = prior + s + r;
WriteString[stream, ToString[next]];
If[r < 5, WriteString[stream, "\t"]];
prior = next
];
If[s < 5, WriteString[stream, "\n"]]
];
Close[stream]


Test this carefully before you depend on it.

• Thanks for taking the time to respond! I will examine your code and if I am convinced that this is what I was looking for I will accept your answer. Cheers! – AG1123 Jun 27 '18 at 4:40

First a side note: Test[s, 0] = 0 etc. does not define a matrix but a hash table. You can allocate memory with Test=ConstantArray[0,{5,5}] and access the data with double brackets [[ ]] (see also the documentation of Part).

Towards your question. Your algorithm as an iterative structure, so I suggest to use Nest instead of For (really, everyting is better than using For! There are various post on this site that will explain to you why.)

This is the function for each iteration. It needs the iteration counter r and the current state (the current matrix row). Write writes to a file specified by the global variable stream that is yet to be defined.

step[{r_, state_}] := With[{newstate = state + Range[Length[state]] + r},
Write[stream, newstate];
{r + 1, newstate}
]


This opens the stream, processes step in a Nested way and finally closes the stream.

stream = OpenWrite["a.dat"];
Nest[step, {1, ConstantArray[0, 5]}, 5]
Close[stream]


This is a very crude way to write to file and it produces a file that Mathematica cannot be imported directly with Import. But you can import line by line with

stream = OpenRead["a.dat"];


Of course, you have to execute Read[stream] once for each line you want to get.