1
$\begingroup$

Why does

fileName = $TemporaryDirectory <> "/" <> CreateUUID[]
ParallelTable[buffer = i -> i^2;
  CriticalSection[{outFile}, 
  outFile = OpenAppend[fileName, BinaryFormat -> True]; 
  Write[outFile, buffer]; Close@outFile];, {i, 1, 10}];
NestWhileList[Function[{}, Read[fileName]], 
 Read[fileName], # =!= EndOfFile & , 1, Infinity, -1]
DeleteFile@fileName

work, but

fileName = $TemporaryDirectory <> "/" <> CreateUUID[]
outFile = OpenWrite[fileName, BinaryFormat -> True];
SetSharedVariable[outFile]
ParallelTable[buffer = i -> i^2;
  CriticalSection[{outFile}, Write[outFile, buffer]];, {i, 1, 10}];
NestWhileList[Function[{}, Read[fileName]], 
 Read[fileName], # =!= EndOfFile & , 1, Infinity, -1]
DeleteFile@fileName

does not?

Also as a side note (Why) do i have to specify m and mmax for NestWhileList to specify n? Aswell: should i use a database to save such information? Mathematica likes to run out of memory for me.

Edit: Instead of ParallelTable ParallelDo should have worked too. It's a relic from doing it without intermediate saving.

$\endgroup$
  • $\begingroup$ Do you expect the data in your file to be correctly sorted: Table[Print[i -> i^2], {i, 1, 10}]; or to be in chronological order: ParallelTable[Print[i -> i^2], {i, 1, 10}]; ? $\endgroup$ – Shadowray May 2 at 20:39
  • $\begingroup$ @Shadowray I do not care for any order. I specifically wonder why the one example works while the other does not. $\endgroup$ – Gladaed May 3 at 12:06
  • $\begingroup$ I think the problem is the use of SetSharedVariable[outFile] because you end up with a bunch of processes that all get in each others way trying to use the same stream to write something to the file. Have you tried giving each parallel process its own output stream? E.g., ParallelEvaluate[outFile = OpenWrite[...]]. $\endgroup$ – Sjoerd Smit May 3 at 12:59
  • $\begingroup$ @SjoerdSmit Well the processes should not do that since i use the CriticalSection function for critical sections. It works in the upper example. $\endgroup$ – Gladaed May 3 at 16:33
  • $\begingroup$ @Gladaed But in the first code section the variable outFile only lives in the parallel kernels and not in the master kernel. Each kernel uses a separate stream to write to the file. Also note the remark in the Details section of CriticalSection: "Variables used as locks should not have a value in the master kernel.". Your code does not follow this rule. (See also the Possible Issues section). $\endgroup$ – Sjoerd Smit May 3 at 21:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.