I want to know how to write and read a same text file (for example, external.txt ) without conflicts in parallelization. The following is just a small idea, but the problem is there seems a conflicts in writing to the same text file -- ObtainedData sometimes are only part written in the external.txt.

The following is the code just for describe the idea.


 data = Import["external.txt"];
 ExternalMemory = ImportString[data, "List"];

(*get data from some operations, here i ignore the codes using ...... *)

(*check whether the ObtainedData is in the external file, if yes skip, 
 if not then write to the file and also write to corresponding results file.*)

If[MemberQ[ExternalMemory,ObtainedData]==False , 

 WriteLine[currentdirectory<>"\\"<>"external.txt", ToString[ObtainedData]];
 WriteLine[currentdirectory<>"\\"<>"results"<>ToString[$KernelID]<>".txt", ObtainedData];

 ,{iilsts, 1, 1000}];


One thing I thought is to use the master (KernelID=0) or one of other Kernels to read all the results files and write to external.txt, but I'm not sure how and whether it can solve.

Is anybody familiar with such problems or similar cases? Thank you all in advance!

  • 3
    $\begingroup$ Trying to have multiple programs writing in the same file without coordination seems to be calling for problems. What about a database? If not, depending on the overheads, how frequent the queries and heavy the data you can use SetSharedVariable to keep a light index synchronized on all kernels. $\endgroup$ – rhermans Jul 20 '19 at 10:27
  • $\begingroup$ I'm not sure how frequent the queries. What do you mean use SetSharedVariable to keep a light index? do you mean on all kernels I try to fix the index? Thank you very much! $\endgroup$ – Xuemei Gu Jul 20 '19 at 10:32
  • 1
    $\begingroup$ Did you read the docs for SetSharedVariable? It "declares symbols as shared variables whose values are synchronized among all parallel kernels.". If there is a lightweight way to identify your data, you can keep an index in a shared variable or a database instead of searching and writing a file each time. And then create one data file per kernel, so there no possibility of collision. $\endgroup$ – rhermans Jul 20 '19 at 10:37
  • $\begingroup$ I just read the docs for SetShareVariable. The way I do is always to write and read the file, I don't know to the lightweight way to identify data yet. I will have a try. Thank you! $\endgroup$ – Xuemei Gu Jul 20 '19 at 15:13
  • 5
    $\begingroup$ Probably the best answer is that you just shouldn't. Think twice if you really need to do this, and then think again. If you still want it after that, create a separate function for writing to the file and use SetSharedFunction on it. There's also CriticalSection. $\endgroup$ – Szabolcs Jul 20 '19 at 16:44