Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

What is the relation of memory usage of a SparseArray
and the number of its occupied positions?

Let's say you build a 100.000.000 by 10 element SparseArray.
And fill the two position 1/1 and 100.000.000/10 with a value.

num = 999999;  
idSparse = SparseArray[{{1, 1} -> num, {100000000, 10} -> num}]  

There are two elements in the Array.

Memory usage is:

ByteCount[idSparse]  
400000968  

Disk usage is:

Export["idSparse.rsa", idSparse];  
FileByteCount["idSparse.rsa"]  
380  
share|improve this question
add comment

1 Answer

Looking at the InputForm or FullForm of the expression which, while not equivalent to the internal data format, shows something of the structure and what is stored:

SparseArray[{{1, 1} -> 999, {5, 100} -> 999}] // InputForm
SparseArray[Automatic, {5, 100}, 0, {1, {{0, 1, 1, 1, 1, 2}, {{1}, {100}}}, {999, 999}}]

versus:

SparseArray[{{1, 1} -> 999, {100, 5} -> 999}] // InputForm
SparseArray[Automatic, {100, 
  5}, 0, {1, {{0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
     1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2}, {{1}, {5}}}, {999, 999}}]

shows that some limited data is stored for every row in the array.

Therefore your expression will take up much less space if it is entered as:

num = 999999;
idSparse = SparseArray[{{1, 1} -> num, {10, 100000000} -> num}];
ByteCount[idSparse]

784

Of course your program will need to account for the changed orientation.

share|improve this answer
    
Good find, didn't know that. –  Sjoerd C. de Vries May 22 '13 at 8:57
    
OK. . .Very good. . .We may add that in the first way we get large a RAM and a small file, the second way we get a smal RAM usage and a large file. –  Hp Radojewski Schäfer Von May 23 '13 at 15:12
    
@HpR that's good information, but not very convenient. I wonder if there is another sparse array file format that matches Mathematica's behavior. Using Transpose on the small-type SparseArray makes it large (as expected), so that is not a solution. –  Mr.Wizard May 23 '13 at 16:26
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.