# How to avoid unpacking from LanguageExtendedFullDefinition

Consider this code:

LaunchKernels[];
On["Packing"]

a = RandomReal[{0., 1.}, {64, 30000}];

ParallelMap[Fourier, a]; // AbsoluteTiming


DeveloperFromPackedArray::unpack: Unpacking array in call to LanguageExtendedFullDefinition. DeveloperFromPackedArray::unpack: Unpacking array in call to MemberQ. >>

(*{1.853567, Null}*)

Map[Fourier, a]; // AbsoluteTiming


DeveloperFromPackedArray::punpackl1: Unpacking array with dimensions {64,30000} to level 1.

(*{0.289122, Null}*)


How to avoid the unpacking and get some speedup by parallelization?

Update

By avoiding the MemberQ unpack (fix function copied from here), we can get about 2X speedup, but still slower than the non-parallel version :

memberQ[list_, form_] := Or @@ (MatchQ[#, form] & /@ list)
ClearAll[fix]
SetAttributes[fix, HoldAll]
fix[expr_] := Block[{MemberQ = memberQ}, expr]

fix@ParallelMap[Fourier, a]; // AbsoluteTiming


DeveloperFromPackedArray::unpack: Unpacking array in call to LanguageExtendedFullDefinition. >>

(*{0.564126, Null}*)


Update 2

Using the ParallelTable can eliminate unpacking and can actually get speedup

first run

fix[
ParallelTable[
Fourier[a[[n]]], {n, 1, Length[a]}]]; // AbsoluteTiming
(*{0.215288, Null}*)


second run

fix[
ParallelTable[
Fourier[a[[n]]], {n, 1, Length[a]}]]; // AbsoluteTiming
(*{0.092006, Null}*)


Questions:

1. What is this LanguageExtendedFullDefinition and why I always get this warning? How to avoid unpacking from it? I'm using version 9.
2. Can you give more evidence on "Fourier is so fast that you loose any time you gain in the overhead of parallelism"?
3. If the slow is because of parallel over head, why ParallelTable is 5X faster than ParallelMap? Thanks a lot!
-

I don't think unpacking is the problem. Rather, I believe that Fourier is so fast that you loose any time you gain in the overhead of parallelism.

Consider using Identity as an example. I will use withModifiedMemberQ to deal with that bug.

I use List @@ to intentionally unpack to level one, which is not a problem as subarrays remain packed.

a = List @@ RandomReal[{0., 1.}, {640, 30000}];
On["Packing"]

withModifiedMemberQ[
ParallelMap[Identity, a]; // AbsoluteTiming
]

Map[Identity, a]; // AbsoluteTiming

{0.4930282, Null}

{0.0850049, Null}


Note that there are no unpacking messages issued. However, I had to run the code above twice as the first pass I got an error:

list::shdw: Symbol list appears in multiple contexts {ParallelPreferences,Global}; definitions in context ParallelPreferences may shadow or be shadowed by other definitions.)

This might be a v7 bug. Does anyone else see it with the code above?

-
I do not see the shdw message. Version 9.0.1 on fresh kernel, screen shot: !Mathematica graphics on second call: !Mathematica graphics –  Nasser Sep 3 '13 at 2:37
@Nasser Thanks; I don't know why ParallelPreferences  would end up on the context path; that seems like a mistake. Good that it's apparently fixed. –  Mr.Wizard Sep 3 '13 at 2:40
I guess unpack maybe somewhat relevant because if I remove used the fixed MemberQ I get 2X speedup? See my update. –  xslittlegrass Sep 3 '13 at 2:40
@xslittlegrass Yes, there is a known MemberQ unpacking bug as Oleksandr described, and as I already addressed in my answer. Beyond that what is your point? I don't understand. –  Mr.Wizard Sep 3 '13 at 2:44
@Mr.Wizard I'm wonderring the unpack in calling to LanguageExtendedFullDefinition also has some effect. Since in your example, the a` is ten times larger than mine, does that make the overhead more significant in your case? And also 1 second over head seems quit large, isn't it? –  xslittlegrass Sep 3 '13 at 2:47