5
$\begingroup$

I have a compiled function cfun that produces a very large array of data. I learned that compiled functions automatically use PackedArrays. I just wonder whether there is a more efficient way in Mathematica to store large arrays of numerical data.

Is there somewhere a table where I can see how much memory a PackedArray consisting of numerical data with MachineSize precision should consume?

For an output array of length 10^8 ByteCount[] returns memory consumption of 1.6 GB on my machine. Would be great if somebody could tell me whether this is a normal value and whether there is a way to store the array in a more efficient way (since for the purpose of my program I need to store a list of length 10^9-10^10 which would correspond to 16-160 GB RAM).

$\endgroup$
  • $\begingroup$ Not an anwser, but could you use a stream to write the data to a file, using Openwrite ,Write etc? Or create an enourmous (~100gb) swap file, like I did today? $\endgroup$ – Ruud3.1415 Feb 9 '18 at 16:50
  • $\begingroup$ 10^8 elements will take up about 0.75 GB if the elements are real, or 1.5 GB if they are complex. Note that a gigabyte is 1024^3 bytes, not 1000^3. $\endgroup$ – Szabolcs Feb 9 '18 at 16:54
  • $\begingroup$ Ruud3: My problem is that I need to do computational stuff with the data I create. I'm afraid, writing it to a file and then import it again won't be helpful here. Szabolcs: You are right, of course. Thank you for the reminder. Perhaps the meta data makes it 1.6 GB. I need to check again on this. $\endgroup$ – Display Name Feb 9 '18 at 17:03
  • $\begingroup$ Maybe your data has certain redundancies or some "smoothness" that allow a (nearly) lossless compression... $\endgroup$ – Henrik Schumacher Feb 9 '18 at 17:40
  • $\begingroup$ @HenrikSchumacher: How can I check for redundancies or "smoothness"? Could you please provide me with an example? $\endgroup$ – Display Name Feb 9 '18 at 17:52
6
$\begingroup$

Real-type packed arrays are stored as a contiguous block of double-precision values, plus a small bit of metadata.

This means that:

  • If you have n elements, it takes n*8 bytes
  • There is no reasonable way to store data of the same precision more efficiently

An integer takes up 8 bytes on a 64-bit system or 4 byte on a 32-bit system in a packed array. A complex number takes up 16 bytes (both the real and imaginary parts are double-precision).

$\endgroup$
  • $\begingroup$ Thank you for your explanation. In fact, my numbers are complex, which explains why my array occupies 1.6 GB. Is your statement true for any programming language? Or do other languages have smarter ways of storing data? I am afraid, that my local memory will be the bottleneck for my computation then if there is no other reasonable way. :( $\endgroup$ – Display Name Feb 9 '18 at 16:57
  • $\begingroup$ @DisplayName It is true for any language. If you use a low-level language like C, you can use single-precision numbers (~7 dgitis of precision), which take up half the space. 10^10 is almost never a reasonable size of an array. Consider carefully if you really need to store this much data simultaneously. Maybe you can generate and process the data piece by piece. $\endgroup$ – Szabolcs Feb 9 '18 at 17:00
  • $\begingroup$ Ok interesting, wasn't aware of this. I guess I cannot reduce the precision that easily with Mathematica? In principle, generating and processing the data piece by piece is a possibility. I have already thought about this but first wanted to know whether there is a more efficient way of storing the data. Essentially, it is stochastic data and I need to compute some quantities like moments etc. As it seems these averaged quantites seem to converge very slowly. This just as background information why I am aiming for such a large array. $\endgroup$ – Display Name Feb 9 '18 at 17:07
  • $\begingroup$ @DisplayName It is not necessary to store a full list to compute its average (and all moments are averages, so the same applies). Just do it piece by piece. Keep track of how many numbers you have already averaged. $\endgroup$ – Szabolcs Feb 9 '18 at 17:13
  • 1
    $\begingroup$ @DisplayName Mathematica encourages working with entire arrays. You generate the whole array, then you do operations on the whole array. In performance-oriented languages you would not do this. You would process those random numbers one by one as they are generated. I assume you are generating them and not reading from a file. This way memory is not an issue. Example $\endgroup$ – Szabolcs Feb 9 '18 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.