How to avoid unpacking from Language`ExtendedFullDefinition

Question

Consider this code:

LaunchKernels[];
On["Packing"]

a = RandomReal[{0., 1.}, {64, 30000}];

ParallelMap[Fourier, a]; // AbsoluteTiming

DeveloperFromPackedArray::unpack: Unpacking array in call to LanguageExtendedFullDefinition. Developer`FromPackedArray::unpack: Unpacking array in call to MemberQ. >>

(*{1.853567, Null}*)

Map[Fourier, a]; // AbsoluteTiming

Developer`FromPackedArray::punpackl1: Unpacking array with dimensions {64,30000} to level 1.

(*{0.289122, Null}*)

How to avoid the unpacking and get some speedup by parallelization?

Update

By avoiding the MemberQ unpack (fix function copied from here), we can get about 2X speedup, but still slower than the non-parallel version :

memberQ[list_, form_] := Or @@ (MatchQ[#, form] & /@ list)
ClearAll[fix]
SetAttributes[fix, HoldAll]
fix[expr_] := Block[{MemberQ = memberQ}, expr]

fix@ParallelMap[Fourier, a]; // AbsoluteTiming

DeveloperFromPackedArray::unpack: Unpacking array in call to LanguageExtendedFullDefinition. >>

(*{0.564126, Null}*)

Update 2

Using the ParallelTable can eliminate unpacking and can actually get speedup

first run

fix[
   ParallelTable[
    Fourier[a[[n]]], {n, 1, Length[a]}]]; // AbsoluteTiming
(*{0.215288, Null}*)

second run

fix[
   ParallelTable[
    Fourier[a[[n]]], {n, 1, Length[a]}]]; // AbsoluteTiming
(*{0.092006, Null}*)

Questions:

1. What is this LanguageExtendedFullDefinition` and why I always get this warning? How to avoid unpacking from it? I'm using version 9.
2. Can you give more evidence on "Fourier is so fast that you loose any time you gain in the overhead of parallelism"?
3. If the slow is because of parallel over head, why ParallelTable is 5X faster than ParallelMap? Thanks a lot!

Answer 1

I don't think unpacking is the problem. Rather, I believe that Fourier is so fast that you loose any time you gain in the overhead of parallelism.

Consider using Identity as an example. I will use withModifiedMemberQ to deal with that bug.

I use List @@ to intentionally unpack to level one, which is not a problem as subarrays remain packed.

a = List @@ RandomReal[{0., 1.}, {640, 30000}];
On["Packing"]

withModifiedMemberQ[
  ParallelMap[Identity, a]; // AbsoluteTiming
]

Map[Identity, a]; // AbsoluteTiming

{0.4930282, Null}

{0.0850049, Null}

Note that there are no unpacking messages issued. However, I had to run the code above twice as the first pass I got an error:

list::shdw: Symbol list appears in multiple contexts {Parallel`Preferences`,Global`}; definitions in context Parallel`Preferences` may shadow or be shadowed by other definitions.)

This might be a v7 bug. Does anyone else see it with the code above?