7
$\begingroup$

What is the relation of memory usage of a SparseArray and the number of its occupied positions?

Let's say you build a $100000000 \times 10$ SparseArray and fill the two positions $(1,1)$ and $(100000000,10)$ with a value:

num = 999999;  
idSparse = SparseArray[{{1, 1} -> num, {100000000, 10} -> num}]  

There are two elements in the array.

Memory usage is:

ByteCount[idSparse]  
  400000968  

Disk usage is:

Export["idSparse.rsa", idSparse];  
FileByteCount["idSparse.rsa"]  
  380  
$\endgroup$
8
$\begingroup$

Looking at the InputForm or FullForm of the expression which, while not equivalent to the internal data format, shows something of the structure and what is stored:

SparseArray[{{1, 1} -> 999, {5, 100} -> 999}] // InputForm
SparseArray[Automatic, {5, 100}, 0, {1, {{0, 1, 1, 1, 1, 2}, {{1}, {100}}}, {999, 999}}]

versus:

SparseArray[{{1, 1} -> 999, {100, 5} -> 999}] // InputForm
SparseArray[Automatic, {100, 
  5}, 0, {1, {{0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
     1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2}, {{1}, {5}}}, {999, 999}}]

shows that some limited data is stored for every row in the array.

Therefore your expression will take up much less space if it is entered as:

num = 999999;
idSparse = SparseArray[{{1, 1} -> num, {10, 100000000} -> num}];
ByteCount[idSparse]

784

Of course your program will need to account for the changed orientation.

$\endgroup$
  • $\begingroup$ Good find, didn't know that. $\endgroup$ – Sjoerd C. de Vries May 22 '13 at 8:57
  • $\begingroup$ OK. . .Very good. . .We may add that in the first way we get large a RAM and a small file, the second way we get a smal RAM usage and a large file. $\endgroup$ – Hp Radojewski Schäfer Von May 23 '13 at 15:12
  • $\begingroup$ @HpR that's good information, but not very convenient. I wonder if there is another sparse array file format that matches Mathematica's behavior. Using Transpose on the small-type SparseArray makes it large (as expected), so that is not a solution. $\endgroup$ – Mr.Wizard May 23 '13 at 16:26
  • 1
    $\begingroup$ N.B. peering through the internal form of SparseArray[] quickly reveals that the storage format is the so-called "row-major" or Yale format. $\endgroup$ – J. M. is away Jul 27 '16 at 16:52
  • $\begingroup$ @J.M. Thank you! $\endgroup$ – Mr.Wizard Jul 27 '16 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.