# Problems with ParallelDo

As example I have the following two image sets (10 images each, png format, 8 bit gray scale) in two different directories and want to determine the mean intensity of each image:

The code below produces an error. What is the reason that the files in dir_2 are not found?

ClearAll["Global*"];

dirs = {"F:\\dir_1\\", "F:\\dir_2\\"};

ndir = Length[dirs];

SetSharedVariable[meanIntensity];

Do[

SetDirectory[dirs[[j]]];

dirBase = FileNameTake[dirs[[j]]];

fNames = FileNames["*.png"];
numFiles = Length[fNames];

meanIntensity = Array[0 &, numFiles];

ParallelDo[

meanIntensity[[i]] =
ComponentMeasurements[Import[fNames[[i]], "png"],
"MeanIntensity"][[All, 2]][[1]];

Print["j=", j, " i=", i, " meanIntensity=", meanIntensity[[i]]];

, {i, 1, numFiles}

];

, {j, 1, ndir}

];


The output is:

(kernel 2) j=1 i=5 meanIntensity=0.0228285

(kernel 4) j=1 i=1 meanIntensity=0.0225379

(kernel 3) j=1 i=3 meanIntensity=0.0227294

(kernel 1) j=1 i=6 meanIntensity=0.0227178

(kernel 4) j=1 i=2 meanIntensity=0.0225431

(kernel 3) j=1 i=4 meanIntensity=0.0224934

(kernel 2) j=1 i=7 meanIntensity=0.0223063

(kernel 1) j=1 i=8 meanIntensity=0.0227366

(kernel 4) j=1 i=9 meanIntensity=0.0227438

(kernel 3) j=1 i=10 meanIntensity=0.0225731

(kernel 4) ComponentMeasurements::invarg1 :  Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of $Failed. (kernel 3) ComponentMeasurements::invarg1 : Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of$Failed.

(kernel 4) Part::partd :  Part specification ComponentMeasurements[$Failed,MeanIntensity][[All,2]] is longer than depth of object. (kernel 3) Part::partd : Part specification ComponentMeasurements[$Failed,MeanIntensity][[All,2]] is longer than depth of object.

(kernel 2) ComponentMeasurements::invarg1 :  Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of $Failed. (kernel 1) ComponentMeasurements::invarg1 : Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of$Failed.

During evaluation of In[4]:= ComponentMeasurements::invarg1: Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of $Failed. During evaluation of In[4]:= ComponentMeasurements::invarg1: Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of$Failed.

During evaluation of In[4]:= ComponentMeasurements::invarg1: Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of $Failed. During evaluation of In[4]:= General::stop: Further output of ComponentMeasurements::invarg1 will be suppressed during this calculation. (kernel 2) Part::partd : Part specification ComponentMeasurements[$Failed,MeanIntensity][[All,2]] is longer than depth of object.

(kernel 1) Part::partd :  Part specification ComponentMeasurements[$Failed,MeanIntensity][[All,2]] is longer than depth of object. (kernel 4) ComponentMeasurements::invarg1 : Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of$Failed.

(kernel 3) ComponentMeasurements::invarg1 :  Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of $Failed. (kernel 2) ComponentMeasurements::invarg1 : Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of$Failed.

(kernel 1) ComponentMeasurements::invarg1 :  Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of $Failed. (kernel 4) ComponentMeasurements::invarg1 : Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of$Failed.

(kernel 3) ComponentMeasurements::invarg1 :  Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of $Failed. (kernel 4) General::stop : Further output of ComponentMeasurements::invarg1 will be suppressed during this calculation. (kernel 3) General::stop : Further output of ComponentMeasurements::invarg1 will be suppressed during this calculation. (kernel 2) ComponentMeasurements::invarg1 : Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of$Failed.

(kernel 1) ComponentMeasurements::invarg1 :  Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of $Failed. (kernel 4) j=2 i=1 meanIntensity=ComponentMeasurements[$Failed,MeanIntensity]

(kernel 3) j=2 i=3 meanIntensity=ComponentMeasurements[$Failed,MeanIntensity] (kernel 2) General::stop : Further output of ComponentMeasurements::invarg1 will be suppressed during this calculation. (kernel 1) General::stop : Further output of ComponentMeasurements::invarg1 will be suppressed during this calculation. (kernel 4) Import::nffil : File not found during Import. (kernel 3) Import::nffil : File not found during Import. (kernel 2) j=2 i=5 meanIntensity=ComponentMeasurements[$Failed,MeanIntensity]

(kernel 1) j=2 i=6 meanIntensity=ComponentMeasurements[$Failed,MeanIntensity] (kernel 4) Part::partd : Part specification ComponentMeasurements[$Failed,MeanIntensity][[All,2]] is longer than depth of object.

(kernel 3) Part::partd :  Part specification ComponentMeasurements[$Failed,MeanIntensity][[All,2]] is longer than depth of object. (kernel 2) Import::nffil : File not found during Import. (kernel 1) Import::nffil : File not found during Import. (kernel 4) j=2 i=2 meanIntensity=ComponentMeasurements[$Failed,MeanIntensity]

(kernel 3) j=2 i=4 meanIntensity=ComponentMeasurements[$Failed,MeanIntensity] (kernel 2) ComponentMeasurements::invarg1 : Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of$Failed.

(kernel 1) ComponentMeasurements::invarg1 :  Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of $Failed. (kernel 2) Part::partd : Part specification ComponentMeasurements[$Failed,MeanIntensity][[All,2]] is longer than depth of object.

(kernel 1) Part::partd :  Part specification ComponentMeasurements[$Failed,MeanIntensity][[All,2]] is longer than depth of object. (kernel 4) Import::nffil : File not found during Import. (kernel 3) Import::nffil : File not found during Import. (kernel 4) ComponentMeasurements::invarg1 : Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of$Failed.

(kernel 3) ComponentMeasurements::invarg1 :  Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of $Failed. (kernel 2) ComponentMeasurements::invarg1 : Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of$Failed.

(kernel 1) ComponentMeasurements::invarg1 :  Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of $Failed. (kernel 4) Part::partd : Part specification ComponentMeasurements[$Failed,MeanIntensity][[All,2]] is longer than depth of object.

(kernel 3) Part::partd :  Part specification ComponentMeasurements[$Failed,MeanIntensity][[All,2]] is longer than depth of object. (kernel 2) ComponentMeasurements::invarg1 : Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of$Failed.

(kernel 1) ComponentMeasurements::invarg1 :  Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of $Failed. (kernel 4) ComponentMeasurements::invarg1 : Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of$Failed.

(kernel 3) ComponentMeasurements::invarg1 :  Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of $Failed. (kernel 2) General::stop : Further output of ComponentMeasurements::invarg1 will be suppressed during this calculation. (kernel 1) General::stop : Further output of ComponentMeasurements::invarg1 will be suppressed during this calculation. (kernel 2) j=2 i=7 meanIntensity=ComponentMeasurements[$Failed,MeanIntensity]

(kernel 1) j=2 i=8 meanIntensity=ComponentMeasurements[$Failed,MeanIntensity] (kernel 4) ComponentMeasurements::invarg1 : Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of$Failed.

(kernel 3) ComponentMeasurements::invarg1 :  Expecting a valid label matrix, an image, or a list of an image and a label matrix instead of $Failed. (kernel 4) General::stop : Further output of ComponentMeasurements::invarg1 will be suppressed during this calculation. (kernel 3) General::stop : Further output of ComponentMeasurements::invarg1 will be suppressed during this calculation. (kernel 4) j=2 i=9 meanIntensity=ComponentMeasurements[$Failed,MeanIntensity]

(kernel 3) j=2 i=10 meanIntensity=ComponentMeasurements[\$Failed,MeanIntensity]


When I use Do instead of ParallelDo everything is fine:

j=1 i=1 meanIntensity=0.0225379

j=1 i=2 meanIntensity=0.0225431

j=1 i=3 meanIntensity=0.0227294

j=1 i=4 meanIntensity=0.0224934

j=1 i=5 meanIntensity=0.0228285

j=1 i=6 meanIntensity=0.0227178

j=1 i=7 meanIntensity=0.0223063

j=1 i=8 meanIntensity=0.0227366

j=1 i=9 meanIntensity=0.0227438

j=1 i=10 meanIntensity=0.0225731

j=2 i=1 meanIntensity=0.0221354

j=2 i=2 meanIntensity=0.0223254

j=2 i=3 meanIntensity=0.0225109

j=2 i=4 meanIntensity=0.0220713

j=2 i=5 meanIntensity=0.0221103

j=2 i=6 meanIntensity=0.022274

j=2 i=7 meanIntensity=0.0221438

j=2 i=8 meanIntensity=0.0221835

j=2 i=9 meanIntensity=0.0223304

j=2 i=10 meanIntensity=0.0222357

• I assume you would need to set the current directory on each subkernel as necessary. – ilian Sep 23 '16 at 16:25
• How would that be done? – mrz Sep 23 '16 at 16:49
• @mrz try to replace SetDirectory[dirs[j]] by With[{dirj = dirs[[j]]}, ParallelEvaluate@SetDirectory@dirj ] – Rolf Mertig Sep 23 '16 at 18:39
• @Rolf Mertig: I tried your replacement but this even produces more errors: see the nb file: tinyurl.com/zsxxmy6 and the output: tinyurl.com/z4dynxg. – mrz Sep 24 '16 at 8:59

This works as expected. The key issue is to use the second argument of FileNames.

dirs={"F:\\dir_1\\","F:\\dir_2\\"};
ndir=Length[dirs];
SetSharedVariable[meanIntensity];
Do[
fNames=FileNames["*.png",{dirs[[j]]}];
numFiles=Length[fNames];
meanIntensity=Array[0&,numFiles];
ParallelDo[ meanIntensity[[i]]  = ComponentMeasurements[
Import[fNames[[i]],"png"],
"MeanIntensity"][[All,2]][[1]];
Print["j=",j," i=",i," meanIntensity=",meanIntensity[[i]] ]
,
{i,1,numFiles}
]
,
{j,1,ndir}
];

• Great. Thanks a lot. Can you explain why this small change solves the problem? – mrz Sep 24 '16 at 9:22
• It works because you give FileNames the real full file name and all Global symbols are distributed over the parallel kernels. Why SetDirectory on parallel subkernels does not work, I do not know. Please ask or better call Wolfram support. They will certainly know more about the mysteries of differences of subkernels from the master kernel. – Rolf Mertig Sep 24 '16 at 13:23