Export to file from ParallelDo loop

Question

I currently have a Mathematica code similar to:

ParallelDo[
 Quiet@Check[
   Print[var1, "\t", var2, "\t", integral1[var1,var2], "\t",
         integral2[var1, var2]], 
   err
 ], 
 {var1, 10, 50, 2}, {var2, 10, 100,1}
]

And, I get the results I want in the GUI. However, I can't seem to figure out the best way to export the data I want, while keeping the ParallelDo loop.

Ideally I would like a comma separated format, similar to what the above outputs.

Any ideas? Many thanks in advance!

Answer 1

Although I'd generally suggest using ParallelTable instead, I can imagine a scenario where you would like to use ParallelDo with a function that is really lengthy and could potentially hang or crash. Then you would want to save the results of each successful parallel computation to avoid losing it in case of a crash.

But to do that, it's safest to create a separate file for each result, because we don't usually have control over the order and timing with which ParallelDo produces results.

Since the loop can always be parameterized by integer indices, we can use these indices to label the individual result files, too. Then these files can later be combined into a single file when all the results have been written out:

ParallelDo[
 Export["out-" <> 
   ToString@NumberForm[i, 2, NumberPadding -> {"0", " "}] <> "-" <> 
   ToString@NumberForm[j, 2, NumberPadding -> {"0", " "}], 
  ToString[Row[{i, ",", j, ",", integral1[i, j]}]] <> "\n", 
  "Text"], {i, 10}, {j, 1, 10}]

Here, I used NumberForm with 2 digits and leading 0 as padding, to produce file names in which the integers i and j appear with constant length. You would have to change the 2 to something larger if the integers range over more digits.

I've inserted a ToString before the integral1 function because I don't know what that function is.

When the loop is done, you'll have files ranging from out-001-001 to out-010-010. Under Unix, these can be combined easily by issuing the command cat out* > results which creates a file called results where you have everything in the desired format.