# How to relate memory usage with occupied positions of SparseArrays?

What is the relation of memory usage of a SparseArray
and the number of its occupied positions?

Let's say you build a 100.000.000 by 10 element SparseArray.
And fill the two position 1/1 and 100.000.000/10 with a value.

num = 999999;
idSparse = SparseArray[{{1, 1} -> num, {100000000, 10} -> num}]


There are two elements in the Array.

Memory usage is:

ByteCount[idSparse]
400000968


Disk usage is:

Export["idSparse.rsa", idSparse];
FileByteCount["idSparse.rsa"]
380

-

Looking at the InputForm or FullForm of the expression which, while not equivalent to the internal data format, shows something of the structure and what is stored:

SparseArray[{{1, 1} -> 999, {5, 100} -> 999}] // InputForm

SparseArray[Automatic, {5, 100}, 0, {1, {{0, 1, 1, 1, 1, 2}, {{1}, {100}}}, {999, 999}}]


versus:

SparseArray[{{1, 1} -> 999, {100, 5} -> 999}] // InputForm

SparseArray[Automatic, {100,
5}, 0, {1, {{0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2}, {{1}, {5}}}, {999, 999}}]


shows that some limited data is stored for every row in the array.

Therefore your expression will take up much less space if it is entered as:

num = 999999;
idSparse = SparseArray[{{1, 1} -> num, {10, 100000000} -> num}];
ByteCount[idSparse]


784

Of course your program will need to account for the changed orientation.

-
Good find, didn't know that. –  Sjoerd C. de Vries May 22 '13 at 8:57
OK. . .Very good. . .We may add that in the first way we get large a RAM and a small file, the second way we get a smal RAM usage and a large file. –  Hp Radojewski Schäfer Von May 23 '13 at 15:12
@HpR that's good information, but not very convenient. I wonder if there is another sparse array file format that matches Mathematica's behavior. Using Transpose on the small-type SparseArray makes it large (as expected), so that is not a solution. –  Mr.Wizard May 23 '13 at 16:26