Memory used by SparseArray objects

In dealing with a quantum mechanics problem, I am representing some many-body states in two different ways in Mathematica. For two parameters n and l the representations are

1. A list of length n which is a partition of l into integers, 0 included, sorted in decreasing order.
2. A list of length l+1 counting how many times each integer 0,...,l occurs in representation 1.

An example: for n=4,l=5, a state could be either represented as

1. {3,1,1,0}
2. {1,2,0,1,0,0}

where the latter list is interpreted as "one 0, two 1's, zero 2's etc".

Typically n is small, 10 or so, while l is maximally l=n(n-1). This means that representation 2 is often very sparse when all integer partitions are considered, so I'm using SparseArray's to store them. However today I did the followig:

Sum[ByteCount[c2[a]], {a, dim}]
(* 4 087 776 *)

Sum[ByteCount[Normal@c2[a]], {a, dim}]
(* 2 009 328 *)


where c2[a] is representaton 2 of state number a and dim is the number of states. What gives??

EDIT: I generate the set of all representation-1 lists with

c1 = IntegerPartitions[l, {n}, Range[0, l]];
dim = Length[c1];


Then, the following sets up representation 2:

counter[i_, l_] :=
Block[{tally},
tally = Tally@Reverse@c1[[i]];
c2[i] = SparseArray[{#1 + 1} -> #2 & @@@ tally, l + 1]
];
Scan[counter[#,l]&,Range[dim]]

• Probably ByteCount doesn't count correctly. Try MaxMemoryUsed[<definition of c2>] instead. (It is undocumented, so the argument will be highlighted in red. But it works.) It would be helpful for the question if you could provide the code to generate some test arrays. – Oleksandr R. May 1 '15 at 11:45
• @OleksandrR., thanks, see my edit. How in this case would I try your suggestion...? – Marius Ladegård Meyer May 1 '15 at 12:10
• There's no free lunch on sparsearrays - with the exception of MMA functions where elements (even when "small") are themselves represented sparsely and appear to be optimized for space "under the covers", it probably makes no sense to create a swarm of relatively tiny sparsearrays - create an empty one (SparseArray[{},10]) and do a bytecount to see the overhead (WRI staff chime in if I'm spouting nonsense). Why not create an empty SA large enough, then populate it in chunks (so c2[[]] to ref. instead of c2[])? Worked fine on old netbook for n,l = 10,90 test with expected space savings... – ciao May 2 '15 at 7:25
• @ciao, you are right. I created a single SparseArray and checked the ByteCount, and it was indeed smaller than the Normal matrix. However, I need to modify copies of the c2[a] and do lookups of which new index a' I end up at. In the process I modify lots of the default elements, and due to what Leonid wrote here I'm starting to think SparseArray is not the tool for this job after all. – Marius Ladegård Meyer May 2 '15 at 10:22