6
$\begingroup$

General problem

I need to work with two vectors (List) of Real numbers that are quite large (each vector containing about $4\cdot 10^8$ elements). Afterwards I need to Dot them. Here the Kernel goes crazy consuming vast amounts of memory. I broke to problem down to a minimal example below.


Minimal example

On["Packing"];
a = Range[380000000];
b = a + 1; (* ensure the vector has to be stored completely *)
Print@Developer`PackedArrayQ@a;
Print@MemoryInUse[];

leads to what I would naively expect with

(* Out: 
   True
   6117811920 *)

Now I would assume that multiplying and summing up my two PackedArrays wouldn't make it necessary to unpack either of the two. This isn't the case as

Dot[a,b];

gives me a warning

(** Out: Developer`FromPackedArray::punpack1: 
    Unpacking array with dimensions {380000000}. *)

It's absolutely impossible to do this calculation on my machine now due to vast amounts of memory being consumed. It costs about 6 gig to store both vectors of the Dot product so in my opinion I shouldn't need more than about 9 gig of memory to do the whole calculation. This isn't the case. Mathematica needs 21 gig of RAM during the calculation - and this when I stopped the kernel. So the real value might be even higher…


So the question for me is: What did I do wrong here? Is the a possibility to avoid unpacking? Why does Dot even need to unpack my data?

$\endgroup$
10
$\begingroup$

Is it important to do the computation with integers?

I have no problem computing this with reals (using N):

In[1] :=  On["Packing"];
a = N@Range[380000000];
b = a + 1;(*ensure the vector has to be stored completely*)
Print@Developer`PackedArrayQ@a;
Print@MemoryInUse[];

True

6124572848

In[5] := c = Dot[a, b];

In[6] := Print@MemoryInUse[];

6124520224

I would conjecture that some elements of Dot[a,b] are too big to be contained as machine integers, hence the unpacking. This can be verified with the following computations:

In[11]:= b2 = ConstantArray[1, Length[a]];
Developer`PackedArrayQ[b2]

Out[12]= True

In[15]:= a2 = Range[380000000];
Developer`PackedArrayQ[a2]

Out[16]= True

In[17]:= c2 = Dot[a2, b2]
Out[17] = 72200000190000000
$\endgroup$
  • $\begingroup$ You just made my day. After one has been told so, it is always as plain as the nose on ones face. Thanks especially for providing a look behind the scenes of M.'s kernel :-) $\endgroup$ – pbx Jan 12 '17 at 14:18
  • $\begingroup$ @pbx :) no problem, I am glad I helped. $\endgroup$ – Anton Antonov Jan 12 '17 at 15:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.