# Export to file from ParallelDo loop

I currently have a Mathematica code similar to:

ParallelDo[
Quiet@Check[
Print[var1, "\t", var2, "\t", integral1[var1,var2], "\t",
integral2[var1, var2]],
err
],
{var1, 10, 50, 2}, {var2, 10, 100,1}
]


And, I get the results I want in the GUI. However, I can't seem to figure out the best way to export the data I want, while keeping the ParallelDo loop.

Ideally I would like a comma separated format, similar to what the above outputs.

Any ideas? Many thanks in advance!

-
Is there any reason why you can't collect the results in a ParallelTable first and export the results later? –  Jens Mar 28 '12 at 15:37
Or, if you absolutely need ParallelDo and not ParallelTable, can't you simply transmit the data back to the main kernel using a shared variable? –  acl Mar 28 '12 at 15:52

Although I'd generally suggest using ParallelTable instead, I can imagine a scenario where you would like to use ParallelDo with a function that is really lengthy and could potentially hang or crash. Then you would want to save the results of each successful parallel computation to avoid losing it in case of a crash.

But to do that, it's safest to create a separate file for each result, because we don't usually have control over the order and timing with which ParallelDo produces results.

Since the loop can always be parameterized by integer indices, we can use these indices to label the individual result files, too. Then these files can later be combined into a single file when all the results have been written out:

ParallelDo[
Export["out-" <>
ToString@NumberForm[i, 2, NumberPadding -> {"0", " "}] <> "-" <>
ToString@NumberForm[j, 2, NumberPadding -> {"0", " "}],
ToString[Row[{i, ",", j, ",", integral1[i, j]}]] <> "\n",
"Text"], {i, 10}, {j, 1, 10}]


Here, I used NumberForm with 2 digits and leading 0 as padding, to produce file names in which the integers i and j appear with constant length. You would have to change the 2 to something larger if the integers range over more digits.

I've inserted a ToString before the integral1 function because I don't know what that function is.

When the loop is done, you'll have files ranging from out-001-001 to out-010-010. Under Unix, these can be combined easily by issuing the command cat out* > results which creates a file called results where you have everything in the desired format.

-
"Although I'd generally suggest using ParallelTable instead, I can imagine a scenario where you would like to use ParallelDo with a function that is really lengthy and could potentially hang or crash. Then you would want to save the results of each successful parallel computation to avoid losing it in case of a crash." That is exactly what I want, but I don't want several files. The order in which ParallelDo gives me the results does not matter, since it's part of a large 2D density plot. So what I seek is the easiest/simplest way for a Mathematica-novice to have the output in a TSV format. –  Bjarke Mar 28 '12 at 17:06
But if you're worried about crashes then you'd also have to worry about corrupted output files. That's another reason why separate files are safer. You're letting the operating system take care of storing your data without collisions. Otherwise, you'd have to re-invent file-locking mechanisms to deal with concurrent file access etc. –  Jens Mar 28 '12 at 17:35