3
$\begingroup$

How do I reduce the memory, reserved for a number?

In[23]:= ByteCount[RandomReal[{1, 9}, 1]]
ByteCount[RandomInteger[{1, 9}, 1]]
ByteCount[Round[RandomReal[{1, 9}, 1], 0.01]]
ByteCount[Round[RandomInteger[{1, 9}, 1], 0.01]]
ByteCount[Chop[RandomReal[{1, 9}, 1]]]
ByteCount[Chop[RandomInteger[{1, 9}, 1]]]

Out[23]= 112

Out[24]= 112

Out[25]= 112

Out[26]= 112

Out[27]= 64

Out[28]= 112

In this form a billion number takes all the RAM. 4 decimal place accuracy is enough for me. How do I reduce the memory usage per number so that I can upload few billion numbers?

In[30]:= ByteCount[RandomReal[{1, 9}, 10^9]]/(1024.^3)

Out[30]= 7.45058073103

Any suggestions?

$\endgroup$
9
  • 3
    $\begingroup$ The standard Real is a 64-bit floating-point number (doubles). There's NumericArray[array, "Real32"], which will convert them to 32-bit FP (singles). Of course you can't convert a whole array of doubles without storing it as an array of doubles. You'd have to chunk the conversion somehow. I haven't played with this type, but you can use Join, Flatten, Part etc on them. If fixed-point will work for you, you could store them in 16-bit "Integer16" or "UnsignedInteger16" numeric arrays. $\endgroup$
    – Michael E2
    Jan 11, 2021 at 16:55
  • $\begingroup$ I tried your suggestion, without success. Would you please give me an explicit example how to use it properly? $\endgroup$
    – csk 7
    Jan 11, 2021 at 19:04
  • $\begingroup$ NumericArray[RandomReal[1, {5, 5}], "Real32"]? (Adapted from the second example in the documentation of NumericArray.) $\endgroup$
    – Michael E2
    Jan 11, 2021 at 19:10
  • $\begingroup$ Allow me to back up one step. Why do you need to store that many numbers all at once? Maybe there is another approach to solve your actual problem that might not require this step. $\endgroup$
    – MarcoB
    Jan 11, 2021 at 23:05
  • $\begingroup$ @MarcoB: I'm looking to compute correlation and build model from close to a 100 GB data. Beyond 4th decimal place accuracy is not important. $\endgroup$
    – csk 7
    Jan 11, 2021 at 23:31

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.