7
$\begingroup$

While developing a parallel version of some code, I noticed that looking up an item in an association seemed much slower than in the serial version of the code. To show the effect, I created the following example :

LaunchKernels[]; (* four kernels will be launched *)
assoc= Association[Table[i-> {i,i^2},{i,1,100}]];
array=Table[{i,i^2},{i,1,100}];
DistributeDefinitions[assoc,array];

To make sure the association and the table are known by the sub-kernels :

In[6]:= ParallelTable[Names["Global`*"], {4}]

Out[6]= {{"array", "assoc"}, {"array", "assoc"}, {"array", "assoc"}, {"array", "assoc"}}

First I run a serial version of some example code :

In[7]:= AbsoluteTiming[Table[Table[assoc[10], {100000}];, {4}];]

Out[7]= {0.106069, Null}

Next I run the parallel version of the same code :

In[8]:= AbsoluteTiming[ParallelTable[Table[assoc[10], {100000}];, {4}, Method -> "CoarsestGrained"];]

Out[8]= {4.37772, Null}

I also tried using the Lookup command explicitly:

In[9]:= AbsoluteTiming[ParallelTable[Table[Lookup[assoc, 10], {100000}];, {4}, Method -> "CoarsestGrained"];]

Out[9]= {4.58422, Null}

In both cases, the parallel version runs much slower although there is (I think) no call-back to the main kernel from the sub-kernels during the evaluation which might slow things down. There is also no passing of data into our out off the sub-kernels.

When I use a table lookup instead of an association lookup in the code , the slow down is not present :

In[13]:= AbsoluteTiming[ParallelTable[Table[array[[10]], {100000}];, {4}, Method -> "CoarsestGrained"];]

Out[13]= {0.037059, Null}

I don't have a clue what is going on. I would be glad if someone can explain me why this slow down happens and how to avoid it. Or point me to a previous similar post in case I missed it.

$\endgroup$
3
$\begingroup$

Seems to be the same "bug" as for rules and dispatches. For some reason it helps to copy the variable in question in the local Kernel.

AbsoluteTiming[ParallelTable[assoc = assoc; Table[assoc[10], {100000}];, {4},Method -> "CoarsestGrained"];]


{0.0354219, Null}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.