3
$\begingroup$

I just realized the following problem. I tried to speed up a program that reads hundreds of files each of size up to 200+ MB and processes them, therefore I changed a Do to ParallelDo . The code (a part of the whole program) is as follows:

out = Reap[
   ParallelDo[
    data = Import[currentFile];
    data = Mean /@ (Through[data[#]] & /@ features);
    Sow[{getNumber[currentFile], data}] 
    {currentFile, files}]
   ];

This delivers an empty output. When changing back from ParallelDo to "normal" Do everything is fine. My conjecture is, that the Reap/ Sow mechanism gets "upset" here. Any other ideas?

$\endgroup$
  • 1
    $\begingroup$ In this simple case, ParallelTable should work. In general, I would suggest to create one list or association per kernel and merge them in the end. $\endgroup$ – Henrik Schumacher Jun 10 at 7:22
  • 3
    $\begingroup$ I think it is because expressions are Sowed in subkernels, which doesn't affect the state of the master kernel. So, in the master kernel, there is nothing to Reap. $\endgroup$ – Anton.Sakovich Jun 10 at 7:24
  • 1
    $\begingroup$ @HenrikSchumacher: Yes, this could be a good approach! I used this in other programs -but here I didn't had this idea. This code is somewhat older so I got stuck with the Reap/ Sow construction. I rewrite this to ParallelTable- Thanks for the hint. $\endgroup$ – mgamer Jun 10 at 7:29
  • $\begingroup$ @Anton.Sakovich I think you are right the sub kernels could be the problem. Thank you. $\endgroup$ – mgamer Jun 10 at 7:29
  • 2
    $\begingroup$ There are countless threads on this on this site. Parallel "threads" are really different processes, they don't share memory. $\endgroup$ – Szabolcs Jun 10 at 8:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.