Building sparse array from packed arrays

Question

Suppose I have two large packed arrays a and b, that respectively contain the position of the elements of a square matrix (i.e. a has the explicit form {{1,2},{2,3},{3,4},...}) and the value for each matrix element (i.e. b has the explicit form {0.2938,0.091,-0.4984,...}).

Now, is there a way to build a sparse array from a and b without having unpacking in the internal process of creating the sparse array ?

Refining the question + example below: The usual SparseArray[a->b] command seems have some unpacking in the process although a and b are unaffected. Really, what I'm concerned about is : suppose you have memA=ByteCount[a] and memB=ByteCount[b].

In the process of constructing c=SparseArray[a->b] I do not want my kernel to suck up all my memory by unpacking a or b (or doing some other unpacking in the internal code), i.e. the kernel memory required during the process should not exceed something like memA+memB, which seems like a very natural requirement to me... In eldo's answer below, it seems there is no unpacking since the arrays considered are too small. Considering larger arrays will lead to the above mentioned problem.

Is there a simple solution (or not so simple) to this issue ?

*Specific example:

On["Packing"] 

a=Table[{i, i + 1}, {i, 1, 10000000}]; (* Position of the elements *)
b=Table[RandomReal[], {i, 1, 10000000}]; (* Value of the elements *)
c=SparseArray[{a -> b}]; (* Generating the sparse array *)
ByteCount[a]
ByteCount[b]
ByteCount[c]

80000168

80000168

160000952

Plus you get the following messages :

FromPackedArray::punpack: Unpacking array with dimensions {10000000,2} in call to Rule. >> FromPackedArray::punpack: Unpacking array with dimensions {10000000} in call to Rule. >> FromPackedArray::punpack1: Unpacking array with dimensions {10000000}.>>

So it seems there is some unpacking in the internal process. In this specific examples the kernel memory during the process goes up by 1 GB, although the final results takes only 160 MB...

$Version

8.0 for Mac OS X x86 (64-bit) (October 5, 2011)

Answer 1

In version 10.1 under Windows I find that a subtle syntax difference is critical to this issue.

The form SparseArray[{a -> b}] causes unpacking while SparseArray[a -> b] does not. Further this affects the output of SparseArray itself, not only its memory consumption during creation.

On["Packing"]

a = {#, # + 1}\[Transpose] & @ Range @ 1*^6;
b = RandomReal[1, 1*^6];
c = SparseArray[a -> b];

Developer`PackedArrayQ /@ {a,b}

MaxMemoryUsed[]
ByteCount[c]

{True, True}

85279216

24000792

In a fresh session the exact same code as above adding only the brackets around a -> b

On["Packing"]

a = {#, # + 1}\[Transpose] & @ Range @ 1*^6;
b = RandomReal[1, 1*^6];
c = SparseArray[{a -> b}];    (* <-------- CRITICAL SYNTAX DIFFERENCE *)

Developer`PackedArrayQ /@ {a,b}

MaxMemoryUsed[]
ByteCount[c]

Developer`FromPackedArray::punpack: Unpacking array with dimensions {1000000,2} in call to Rule. >>

Developer`FromPackedArray::punpack: Unpacking array with dimensions {1000000} in call to Rule. >>

{True, True}

286566696

40000728

The two objects are functionally equivalent despite the size difference:

c1 = SparseArray[a -> b];
c2 = SparseArray[{a -> b}];

ByteCount /@ {c1, c2}

c1 === c2

{24000792, 40000728}

True

I surmise that specific optimizations exist in SparseArray for exactly your application, however the syntax is brittle and the optimization is not triggered for {a -> b}.

Answer 2

Needs["Developer`"];

a = {{1, 2}, {3, 4}, {99, 2}} // ToPackedArray;

b = {1.1, 1.2, 1.3} // ToPackedArray;

c = SparseArray[a -> b];

PackedArrayQ /@ {a, b, c}

{True, True, False}

Hence, a and b were NOT unpacked.

UPDATE

Needs["Developer`"];

a = Table[{i, i + 1}, {i, 1, 10000000}];
b = Table[RandomReal[], {i, 1, 10000000}];
c = SparseArray[{a -> b}];

ByteCount /@ {a, b, c}

{160000152, 80000144, 400000728}

PackedArrayQ /@ {a, b, c}

{True, True, False}

No Warnings.

$Version

"9.0 for Microsoft Windows (64-bit) (January 25, 2013)"