Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Consider the following toy problem:

Q = 10^9;
A = Table[RandomInteger[10], {Q}];
Developer`PackedArrayQ@A
B = Map[N[Sin[#]] &, A]; // AbsoluteTiming
Developer`PackedArrayQ@B
MemoryInUse[]
MaxMemoryUsed[]

True
{105.901936, Null}
True
16022521160
24022519456

But using ParallelMap even with custom memberQ gives the following:

Q = 10^9;
A = Table[RandomInteger[10], {Q}];
Developer`PackedArrayQ@A
withModifiedMemberQ[expr_] := 
  Module[{doneQ, unmatchable}, 
   Internal`InheritedBlock[{MemberQ}, Unprotect[MemberQ];
    (*Can uncomment this if we want to print out the MemberQ calls:mq:
    MemberQ[args___]/;(Print@HoldForm[mq];True):=mq;*)
    MemberQ[list_, patt_Symbol, args___] /; ! TrueQ[doneQ] := 
     Block[{doneQ = True}, 
      MemberQ[Unevaluated[
         list] /. _List?Developer`PackedArrayQ -> {unmatchable}, 
       Unevaluated[patt], args]];
    Protect[MemberQ];
    expr]];
SetAttributes[withModifiedMemberQ, HoldAllComplete];
B = withModifiedMemberQ@
    ParallelMap[N[Sin[#]] &, A]; // AbsoluteTiming
Developer`PackedArrayQ@B
MemoryInUse[]
MaxMemoryUsed[]

True
{533.782398, Null}
True
24027869336
48030873944

We see: 5x drop in performance, 2x increase in max memory usage. Why is it happening? How can it be avoided, while keeping the computation parallelized?

Edit: the code of the real life example can be found here. Same problems observed.

share|improve this question
    
Take a look at this link. A short workaround is at the end of the question under "Solution" and a detailed explanation is in the answer. Let me know if this is not what's causing the problem in your case. –  Szabolcs Jun 17 at 2:12
    
@Szabolcs I've spent this night checking that withModifiedMemberQ@ParallelMap does not help, nor does fix@ParallelMap. –  Yasha Gindikin Jun 17 at 7:41
    
@Szabolcs I have edited the post to make clear that the custom MemberQ is not of any help here. –  Yasha Gindikin Jun 17 at 8:26
    
Having checked a few things, my suspicion is that this is simply one of those cases in which the distribution overhead dominates the calculation time, and so for which one cannot achieve any meaningful performance improvement by running in parallel. Remember that MathLink isn't exactly fast for transferring large amounts of numerical data. But this isn't necessarily a robust conclusion; I didn't find the definitive cause, but only eliminated a few likely ones. –  Oleksandr R. Jun 17 at 11:31
    
@OleksandrR. I was thinking about the parallelization overhead, hoping that a real life example with time-consuming functions instead of Sin[] would reveal the advantages of ParallelMap. However, all I observed was appr. 4x drop in performance and memory consumption as compared to Map[]. The code of the real-life example I am talking about is here: goo.gl/XlheF9 –  Yasha Gindikin Jun 17 at 12:37

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.