Create huge RandomInteger list

For testing purposes, I'd like to create very large amounts of Random Integers, let's say between $-1000$ and $1000$. I'm using the usual

RandomInteger[{-1000, 1000}, 5]


command to create $5$-tuples. On my machine (8GB of RAM), this process becomes quite slow once in the

  Table[
RandomInteger[{-1000, 1000}, 5]
,
{100000000}]


(one hundred million) region. It takes 17 seconds of Timing[] for this list to be made, and I haven't even had any calculations on it.

Does this mean I should restrain using lists till somewhat lower as this threshold, or is there any other way of handling huge lists? Does MMA input the whole list in RAM before doing calculations with it? Is it possible to handle huge lists without including the whole list in RAM, for speeding purposes?

I thought of dividing the list, perhaps exporting to different files (CSV for example), and then doing calculations with every file individually. This is probably the same as Partition[] does (without exporting)?

Thanks for any more suggestions!

-
Little remark: RandomInteger[{-1000, 1000}, {100000000, 5}] is about 5 times faster. – Kuba Nov 6 '13 at 6:14
You don't need Table for this. See @Kuba suggestion. – RunnyKine Nov 6 '13 at 6:15
Like the big-list tag: funny. – wolfies Nov 6 '13 at 6:31

Usually, there are efficiency advantages to generating all of your random numbers in one go, but in this instance, it makes little difference if you generate all in one go, or in tuples of 5 (million of times). I am not sure what you are doing with them, but usually one is doing something like computing a sample mean, or adding them up or similar:

ALL IN ONE

Map[ Total, RandomInteger[{-1000, 1000}, {10000000, 5}]]; // AbsoluteTiming


{2.954751, Null}

10 million tuples of 5:

Table[ Total[RandomInteger[{-1000, 1000}, 5]], {10000000}] ; // AbsoluteTiming


{2.678985, Null}

And, having to use virtual memory, and your hard drive thrashing ... and your system slowing to a crawl is not exactly fun or efficient ... so working with individual tuples of 5 makes good sense here.

The advantage of working with individual tuples of 5 is that you can reduce the memory used to about 1/4:

ALL IN ONE

Map[ Total, RandomInteger[{-1000, 1000}, {10000000, 5}]]; // MaxMemoryUsed


560000824

10 million tuples of 5:

Table[ Total[RandomInteger[{-1000, 1000}, 5]], {10000000}] ; // MaxMemoryUsed


160000480

[ To illustrate, I am using 10 million tuples here --- not the 100 million in your example. Depending on the example, you can probably break it up into smaller units of 10 million, and then average your results etc ]

-
I can't reproduce your Timings on my machine. All in one seems to be faster. Also it becomes even faster if you use the second argument of Total instead of Map as follows: Total[RandomInteger[{-1000, 1000}, {1*^7, 5}], {2}] – RunnyKine Nov 6 '13 at 6:53
Also, if you ignore the Total part of the calculation and just generate the List. The All in one method uses half as much memory on my PC. – RunnyKine Nov 6 '13 at 6:59
Table is faster for me, 1.17 versus 1.23. OS X, ver 9.0. – C. E. Nov 6 '13 at 6:59
@Anon Well, I'm using Windows 8.1 Mathematica v 9.0.1 and I get 0.76 for All in one vs 1.17 for Table – RunnyKine Nov 6 '13 at 7:12
@Wolfies: thanks for your comment, it's interesting. For my actual program, I'm making a huge list of tuples of 6, and am doing rather large calculations with each tuple. Until now, I did the "all in one" method. I will try the other method! – Gabriel Nov 6 '13 at 8:26