# Dramatic increase in memory consumption and performance drop with ParallelMap

Consider the following toy problem:

Q = 10^9;
A = Table[RandomInteger[10], {Q}];
DeveloperPackedArrayQ@A
B = Map[N[Sin[#]] &, A]; // AbsoluteTiming
DeveloperPackedArrayQ@B
MemoryInUse[]
MaxMemoryUsed[]

True
{105.901936, Null}
True
16022521160
24022519456


But using ParallelMap even with custom memberQ gives the following:

Q = 10^9;
A = Table[RandomInteger[10], {Q}];
DeveloperPackedArrayQ@A
withModifiedMemberQ[expr_] :=
Module[{doneQ, unmatchable},
InternalInheritedBlock[{MemberQ}, Unprotect[MemberQ];
(*Can uncomment this if we want to print out the MemberQ calls:mq:
MemberQ[args___]/;(Print@HoldForm[mq];True):=mq;*)
MemberQ[list_, patt_Symbol, args___] /; ! TrueQ[doneQ] :=
Block[{doneQ = True},
MemberQ[Unevaluated[
list] /. _List?DeveloperPackedArrayQ -> {unmatchable},
Unevaluated[patt], args]];
Protect[MemberQ];
expr]];
SetAttributes[withModifiedMemberQ, HoldAllComplete];
B = withModifiedMemberQ@
ParallelMap[N[Sin[#]] &, A]; // AbsoluteTiming
DeveloperPackedArrayQ@B
MemoryInUse[]
MaxMemoryUsed[]

True
{533.782398, Null}
True
24027869336
48030873944


We see: 5x drop in performance, 2x increase in max memory usage. Why is it happening? How can it be avoided, while keeping the computation parallelized?

Edit: the code of the real life example can be found here. Same problems observed.

-
Take a look at this link. A short workaround is at the end of the question under "Solution" and a detailed explanation is in the answer. Let me know if this is not what's causing the problem in your case. –  Szabolcs Jun 17 '14 at 2:12
@Szabolcs I've spent this night checking that withModifiedMemberQ@ParallelMap does not help, nor does fix@ParallelMap. –  Yasha Gindikin Jun 17 '14 at 7:41
@Szabolcs I have edited the post to make clear that the custom MemberQ is not of any help here. –  Yasha Gindikin Jun 17 '14 at 8:26
Having checked a few things, my suspicion is that this is simply one of those cases in which the distribution overhead dominates the calculation time, and so for which one cannot achieve any meaningful performance improvement by running in parallel. Remember that MathLink isn't exactly fast for transferring large amounts of numerical data. But this isn't necessarily a robust conclusion; I didn't find the definitive cause, but only eliminated a few likely ones. –  Oleksandr R. Jun 17 '14 at 11:31
@OleksandrR. I was thinking about the parallelization overhead, hoping that a real life example with time-consuming functions instead of Sin[] would reveal the advantages of ParallelMap. However, all I observed was appr. 4x drop in performance and memory consumption as compared to Map[]. The code of the real-life example I am talking about is here: goo.gl/XlheF9 –  Yasha Gindikin Jun 17 '14 at 12:37