3
$\begingroup$

After I post the question writing data without erasing exist data when quit kernel , I can successfully use OpenAppend to realize it. But if I work with ParallelTable and write in two different ways, one works good and another seems does not work.

If I write in the following way, it seems not work. When I quit Kernels or close and run the code again, all the exist data are replaced by new data.

not working Case:

currentdirectory = NotebookDirectory[];
foldername = "StoreFolder";
pSite = FileNameJoin[{currentdirectory, foldername}]; 
If[DirectoryQ[pSite] == False, folderpath = CreateDirectory[pSite], 
  folderpath = pSite];

kernelNumber = 2;  (*number of Kernels in my computer *)
storePath = {};
For[Kernelnum = 1, Kernelnum <= kernelNumber, Kernelnum++,
  AppendTo[storePath, FileNameJoin[{folderpath, "store" <> ToString[Kernelnum] <> ".txt"}]];
  OpenAppend[storePath[[Kernelnum]]];
  ];

Steps = 10; 
testlist = {0, 0, 0, 0};
ParallelTable[
  WriteLine[storePath[[$KernelID]], ToString[testlist]];,{ii, 1, Steps}];

For[Kernelnum = 1, Kernelnum <= kernelNumber, Kernelnum++,
  Close[storePath[[Kernelnum]]];
  ];

If I write in the following way, it seems work. When I quit Kernels or close and run the code again, all the exist data are not erased.

working Case:

currentdirectory = NotebookDirectory[];
foldername = "StoreFolder";
pSite = FileNameJoin[{currentdirectory, foldername}]; 
If[DirectoryQ[pSite] == False, folderpath = CreateDirectory[pSite], 
  folderpath = pSite];

kernelNumber = 2;  (*number of Kernels in my computer *)
storePath = {};
For[Kernelnum = 1, Kernelnum <= kernelNumber, Kernelnum++,
  AppendTo[storePath, FileNameJoin[{folderpath, "store" <> ToString[Kernelnum] <> ".txt"}]];
  ];

Steps = 10;
testlist = {0, 0, 0, 0};
ParallelTable[OpenAppend[storePath[[$KernelID]]];
  WriteLine[storePath[[$KernelID]], ToString[testlist]];
  Close[storePath[[$KernelID]]]; nn = iiLists, {iiLists, 1, Steps}];

I don't know why one work and another do not work, or did I miss something? Thank you very much!

$\endgroup$
3
$\begingroup$

The issue is that the stream is not opened on the parallel kernels: To see this, we can use the following code:

checkOpen[s_] := Print@MemberQ[Streams[], s]

checkOpen@s
(* True *)

ParallelEvaluate[checkOpen@s]
(* (kernel 1) False *)
(* (kernel 2) False *)
(* {Null, Null} *)

As you can see, the stream is not in the list of open streams on the sub-kernels.

I would suggest you adapt the first approach something like this:

currentdirectory = NotebookDirectory[];
foldername = "StoreFolder";
pSite = FileNameJoin[{currentdirectory, foldername}]; 
folderpath = If[DirectoryQ[pSite], pSite, CreateDirectory[pSite]];

(* open each kernels file on that kernel *)
ParallelEvaluate[
 (* use $KernelID to identify each kernel *)
 storePath = FileNameJoin[{folderpath, "store" <> ToString[$KernelID] <> ".txt"}];
 (* open the stream. Since storeStream is not shared between kernels, 
    each one will have its own stream stored here*)
 storeStream = OpenAppend@storePath;
 ]

Steps = 10;
testlist = {0, 0, 0, 0};
(* write whatever you want to the streams. Again, each kernel has its own stream *)
ParallelTable[WriteLine[storeStream, ToString[testlist]];, {ii, 1, Steps}];

(* close the streams *)
ParallelEvaluate[Close@storeStream]

You'll also note that I've used the OutputStream object returned by OpenAppend to refer to the stream instead of the path. This has two advantages in my opinion:

  • It is clearer in intent: Looking at the code, it is directly visible that the file has already been opened. (otherwise, you rely on the fact that WriteLine and similar use any existing stream for that file)
  • It will fail if the file is not open. If you had used that in your first example, you would have gotten many "OutputStream[...] is not open" error messages from the kernels trying to write to the stream opened on the master kernel (see example below):

This example shows how using streams instead of filenames would have made the problem more visible (I tried to keep the changes to the code in the question minimal):

currentdirectory = NotebookDirectory[];
foldername = "StoreFolder";
pSite = FileNameJoin[{currentdirectory, foldername}]; 
If[DirectoryQ[pSite] == False, folderpath = CreateDirectory[pSite], 
  folderpath = pSite];

kernelNumber = 2;  (*number of Kernels in my computer *)
storeStream= {};
For[Kernelnum = 1, Kernelnum <= kernelNumber, Kernelnum++,
  AppendTo[
    storeStream,
    OpenAppend@FileNameJoin[{folderpath, "store" <> ToString[Kernelnum] <> ".txt"}]
   ];
  ];

Steps = 10; 
testlist = {0, 0, 0, 0};
ParallelTable[
  WriteLine[storeStream[[$KernelID]], ToString[testlist]];,{ii, 1, Steps}];

For[Kernelnum = 1, Kernelnum <= kernelNumber, Kernelnum++,
  Close[storeStream[[Kernelnum]]];
  ];

Here, I changed the storing of the file path (storePath) so that the stream returned by OpenAppend is stored instead (storeStream). Running this code will produce error messages about the stream not being opened.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thank you very much! When I run the first example, I didn't have any “OutputStream[...]” error messages, which means all the files are already open. Maybe the subkernels does didn't realize the file was open and only does WriteLine. What do you think if I do all the operation within ParallelTable, such as ParallelTable[ storePath = FileNameJoin[{folderpath, "store" <> ToString[$KernelID] <> ".txt"}]; storeStream = OpenAppend[storePath]; WriteLine[storeStream, ToString[testlist]]; Close[storeStream], {ii, 1, Steps}]; is it good to open and close the file very often? @LukasLang $\endgroup$ – Xuemei Gu Aug 30 '19 at 17:37
  • $\begingroup$ @XuemeiGu Regarding the missing errors: What I meant to say is that if you would have used the stream objects instead of the filename, then you would have gotten an error (see the updated example in the answer). As noted at the bottom of the answer, if the file is already open (e.g. via OpenAppend), then WriteLine will use that stream. If not, it will open the file. If you give it a stream instead of the filename, then it will complain if the stream is not open. $\endgroup$ – Lukas Lang Aug 30 '19 at 20:43
  • $\begingroup$ ha, now I understand what do you mean :) Thank you very much! @LukasLang $\endgroup$ – Xuemei Gu Aug 30 '19 at 20:48
  • $\begingroup$ @XuemeiGu Glad I could help :) - regarding your question about opening the files every iteration: In most cases, it will probably not matter too much, assuming the filesystem is fast. However, if your file is on a (slow) network drive for example, each OpenAppend&Close might take a noticeable amount of time. To be extra safe, I would therefore stick to something like outlined in the answer, where you open each file once on the respective kernel. But remember one of the most important rules in performance optimization: Measure performance first, and only optimize it if it really is too slow $\endgroup$ – Lukas Lang Aug 30 '19 at 20:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.