0
$\begingroup$

I am trying to read a stream and perform some operations in parallel. Here is the logic of the code inside the ParallelDo:

1- Open stream

2- Read chunk of data

3- Close stream

4- Process the data

5- Store final result for that chunk of data in the variable gI

6- Increment the progress variable that is shared between kernels

7- Repeat with the next chunk of data

Note that this code works when using Do instead of ParallelDo. It also works for small files using ParallelDo but returns errors:

Cannot take positions 4812 through 5062 in \
{1,2,3,4,4,3,4,3,3,2,1,0,0,0,-1,-1,0,-1,0,-1,-1,-2,-2,-3,-2,-2,-2,-3,-\
2,-2,-2,-2,-1,-2,-1,0,1,1,1,1,2,2,3,3,3,3,2,2,1,0,<<4951>>}.

I managed to find that the error comes from the following line in the code:

 If[firstPeaks > 0, dG = data[[firstPeaks + xGS ;; firstPeaks + xGS + xGL]], dG = {0}];

Therefore, I would have two questions:

1- Why is the code working with Do and not with ParallelDo? My only guess is that the variable data containing the chunk of information from the stream is somehow affected, but I do not see how.

2- How can I fix this issue and use ParallelDo in my code? The performance improvement appears to be quite significant.

Thank you in advance!

SetSharedVariable[pr]
pr = 0;

Monitor[

ParallelDo[
  str = OpenRead[FileNameJoin[{path, fileName <> fileExtension}], BinaryFormat -> True];
  data = f1 /@ Flatten[readlines[i, dataLength][[start ;; length + start]]];
  Close[str];

  allPeaks = findPeaks[data, 0.1, .01, maxHeight];
  firstPeaks = If[Length[allPeaks] > 0, First[allPeaks][[1]] + 1, 0];
  If[firstPeaks > 0, dG = data[[firstPeaks + xGS ;; firstPeaks + xGS + xGL]], dG = {0}];
  gI[[i]] = Abs[Total[dG]];
  pr++;
  , {i, 1, width*height}
  ];
, ProgressIndicator[pr/(width*height)]

  ] // AbsoluteTiming
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.