3
$\begingroup$

Suppose I do some big experiment, presumably in a Table[], or a ParallelTable[]. Experience shows that there is a non-zero probability of Mathematica (or the computer) crashing part-way through (usually through excess of ambition by the user, but whatever), so it would be nice to export the results of the individual computations as soon as they become available. Is there any preferred way of doing that? (Reap[] and Sow[] seem like they could do the right thing, but it seems likely that they wait for the whole computation to finish). (and, by the way, I have no idea what the right tag for this is, maybe someone can help).

$\endgroup$
1
$\begingroup$

You can use Streams and Low‐Level Input and Output to achieve this:
Define an export function

exportResults[fileName_String, results_] := Module[{strm = OpenAppend[fileName]},
  Export[strm, {results}, "Table"];
  WriteString[strm, "\n"];
  Close[strm];
 ]

and then make it a part of your code (executing it at the end of each computation task with the result as its second input) or do something like

file = "D:\\MmaExperiment_Results.txt"
Table[exportResults[file, {n, m, n + m, n m}], {n, 10}, {m, 10}];

For this simple calculation ParallelTabel does not work properly, as multiple kernels are writing to the same file at the same time. However, I never encountered that issue in my more calculation heavy parallelized tasks.
If you want to make sure, that there are no conflicts between different kernels, you can have each kernel writing to its own file, e.g. with

ParallelTable[
  exportResults[
    StringJoin[{"D:\\MmaExperiment_Results_", "Kernel-", ToString[$KernelID], ".txt"}],   
    {n, m, n + m, n m}], 
  {n, 100}, {m, 100}];
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.