Writing to file with ParallelDo

I have been experiencing difficulties writing to file in MMA_11.3 when using ParallelDo. The structure of the code is quite simple: a main script runs a ParallelDo block in which a function (parTestFn) is called that returns a variable alpha. I want to write the alpha values to a (csv) file. There are over 100,000 such calculations to be done, which is why i want to parallelize the task.

The code works fine with a standard Do loop.

I planned to do the run in batches of 10,000.

The first time I did this successfully by using alpha>>>"test10000.csv" in the main script. All the alpha values were appended to the file, as anticipated. I then changed the line to alpha>>>"test20000.csv", adjusted the start and end loop points from {i, 1, 10000} to {i,10001,20000} and reran. Nothing was written to file.

I tried the following:

1) changing the new filename to something else (shorter) 2) changing the file directory 3) exiting MMA and rebooting the system 4) using filestream with OpenAppend. 5) setting the variable alpha as a shared variable.
6) Not sharing alpha. 7) Setting the filename as a shared variable.
8) Not sharing the filename. 9) putting the file write code in the parTestfn called from the main loop.

None of these changes had any effect.

In the end I abandoned the attempt to write to file and ran the code, saving the results in a variable. Predictably, MMA failed to terminate gracefully, would not interrupt and I lost all of the output.

• Hmmm ... parallel computations writing to a file. Somewhere along the line something is going to have to serialise the output of the computations into that file. It's almost always best (in the absence of a parallel file system) that the programmer take responsibility for that serialisation. Personally I'd start experimenting with parallel computations writing to a shared data structure, and having a single process/thread responsible for writing to disk. – High Performance Mark Oct 31 '18 at 9:47
• Furthermore, in the absence of any statements to the effect that ParallelDo can take care of multiple computations writing to a single file, I'd bet my best hat that it can't and, while I can't speak to the failures reported, I'm not surprised by them. – High Performance Mark Oct 31 '18 at 12:12