Create huge RandomInteger list

Question

For testing purposes, I'd like to create very large amounts of Random Integers, let's say between $-1000$ and $1000$. I'm using the usual

RandomInteger[{-1000, 1000}, 5]

command to create $5$-tuples. On my machine (8GB of RAM), this process becomes quite slow once in the

  Table[
   RandomInteger[{-1000, 1000}, 5]
   ,
   {100000000}]

(one hundred million) region. It takes 17 seconds of Timing[] for this list to be made, and I haven't even had any calculations on it.

Does this mean I should restrain using lists till somewhat lower as this threshold, or is there any other way of handling huge lists? Does MMA input the whole list in RAM before doing calculations with it? Is it possible to handle huge lists without including the whole list in RAM, for speeding purposes?

I thought of dividing the list, perhaps exporting to different files (CSV for example), and then doing calculations with every file individually. This is probably the same as Partition[] does (without exporting)?

Thanks for any more suggestions!

Answer 1

Usually, there are efficiency advantages to generating all of your random numbers in one go, but in this instance, it makes little difference if you generate all in one go, or in tuples of 5 (million of times). I am not sure what you are doing with them, but usually one is doing something like computing a sample mean, or adding them up or similar:

ALL IN ONE

Map[ Total, RandomInteger[{-1000, 1000}, {10000000, 5}]]; // AbsoluteTiming

{2.954751, Null}

10 million tuples of 5:

Table[ Total[RandomInteger[{-1000, 1000}, 5]], {10000000}] ; // AbsoluteTiming 

{2.678985, Null}

And, having to use virtual memory, and your hard drive thrashing ... and your system slowing to a crawl is not exactly fun or efficient ... so working with individual tuples of 5 makes good sense here.

---

The advantage of working with individual tuples of 5 is that you can reduce the memory used to about 1/4:

ALL IN ONE

Map[ Total, RandomInteger[{-1000, 1000}, {10000000, 5}]]; // MaxMemoryUsed

560000824

10 million tuples of 5:

Table[ Total[RandomInteger[{-1000, 1000}, 5]], {10000000}] ; // MaxMemoryUsed 

160000480

[ To illustrate, I am using 10 million tuples here --- not the 100 million in your example. Depending on the example, you can probably break it up into smaller units of 10 million, and then average your results etc ]