PackedArray and Dot results in undesired unpacking

General problem

I need to work with two vectors (List) of Real numbers that are quite large (each vector containing about $4\cdot 10^8$ elements). Afterwards I need to Dot them. Here the Kernel goes crazy consuming vast amounts of memory. I broke to problem down to a minimal example below.

Minimal example

On["Packing"];
a = Range[380000000];
b = a + 1; (* ensure the vector has to be stored completely *)
Print@DeveloperPackedArrayQ@a;
Print@MemoryInUse[];


leads to what I would naively expect with

(* Out:
True
6117811920 *)


Now I would assume that multiplying and summing up my two PackedArrays wouldn't make it necessary to unpack either of the two. This isn't the case as

Dot[a,b];


gives me a warning

(** Out: DeveloperFromPackedArray::punpack1:
Unpacking array with dimensions {380000000}. *)


It's absolutely impossible to do this calculation on my machine now due to vast amounts of memory being consumed. It costs about 6 gig to store both vectors of the Dot product so in my opinion I shouldn't need more than about 9 gig of memory to do the whole calculation. This isn't the case. Mathematica needs 21 gig of RAM during the calculation - and this when I stopped the kernel. So the real value might be even higher…

So the question for me is: What did I do wrong here? Is the a possibility to avoid unpacking? Why does Dot even need to unpack my data?

Is it important to do the computation with integers?

I have no problem computing this with reals (using N):

In[1] :=  On["Packing"];
a = N@Range[380000000];
b = a + 1;(*ensure the vector has to be stored completely*)
Print@DeveloperPackedArrayQ@a;
Print@MemoryInUse[];

True

6124572848

In[5] := c = Dot[a, b];

In[6] := Print@MemoryInUse[];

6124520224


I would conjecture that some elements of Dot[a,b] are too big to be contained as machine integers, hence the unpacking. This can be verified with the following computations:

In[11]:= b2 = ConstantArray[1, Length[a]];
DeveloperPackedArrayQ[b2]

Out[12]= True

In[15]:= a2 = Range[380000000];
DeveloperPackedArrayQ[a2]

Out[16]= True

In[17]:= c2 = Dot[a2, b2]
Out[17] = 72200000190000000
`
• You just made my day. After one has been told so, it is always as plain as the nose on ones face. Thanks especially for providing a look behind the scenes of M.'s kernel :-) – pbx Jan 12 '17 at 14:18
• @pbx :) no problem, I am glad I helped. – Anton Antonov Jan 12 '17 at 15:03