Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

For testing purposes, I'd like to create very large amounts of Random Integers, let's say between $-1000$ and $1000$. I'm using the usual

RandomInteger[{-1000, 1000}, 5]

command to create $5$-tuples. On my machine (8GB of RAM), this process becomes quite slow once in the

   RandomInteger[{-1000, 1000}, 5]

(one hundred million) region. It takes 17 seconds of Timing[] for this list to be made, and I haven't even had any calculations on it.

Does this mean I should restrain using lists till somewhat lower as this threshold, or is there any other way of handling huge lists? Does MMA input the whole list in RAM before doing calculations with it? Is it possible to handle huge lists without including the whole list in RAM, for speeding purposes?

I thought of dividing the list, perhaps exporting to different files (CSV for example), and then doing calculations with every file individually. This is probably the same as Partition[] does (without exporting)?

Thanks for any more suggestions!

share|improve this question
Little remark: RandomInteger[{-1000, 1000}, {100000000, 5}] is about 5 times faster. – Kuba Nov 6 '13 at 6:14
You don't need Table for this. See @Kuba suggestion. – RunnyKine Nov 6 '13 at 6:15
Like the big-list tag: funny. – wolfies Nov 6 '13 at 6:31

Usually, there are efficiency advantages to generating all of your random numbers in one go, but in this instance, it makes little difference if you generate all in one go, or in tuples of 5 (million of times). I am not sure what you are doing with them, but usually one is doing something like computing a sample mean, or adding them up or similar:


Map[ Total, RandomInteger[{-1000, 1000}, {10000000, 5}]]; // AbsoluteTiming

{2.954751, Null}

10 million tuples of 5:

Table[ Total[RandomInteger[{-1000, 1000}, 5]], {10000000}] ; // AbsoluteTiming 

{2.678985, Null}

And, having to use virtual memory, and your hard drive thrashing ... and your system slowing to a crawl is not exactly fun or efficient ... so working with individual tuples of 5 makes good sense here.

The advantage of working with individual tuples of 5 is that you can reduce the memory used to about 1/4:


Map[ Total, RandomInteger[{-1000, 1000}, {10000000, 5}]]; // MaxMemoryUsed


10 million tuples of 5:

Table[ Total[RandomInteger[{-1000, 1000}, 5]], {10000000}] ; // MaxMemoryUsed 


[ To illustrate, I am using 10 million tuples here --- not the 100 million in your example. Depending on the example, you can probably break it up into smaller units of 10 million, and then average your results etc ]

share|improve this answer
I can't reproduce your Timings on my machine. All in one seems to be faster. Also it becomes even faster if you use the second argument of Total instead of Map as follows: Total[RandomInteger[{-1000, 1000}, {1*^7, 5}], {2}] – RunnyKine Nov 6 '13 at 6:53
Also, if you ignore the Total part of the calculation and just generate the List. The All in one method uses half as much memory on my PC. – RunnyKine Nov 6 '13 at 6:59
Table is faster for me, 1.17 versus 1.23. OS X, ver 9.0. – Pickett Nov 6 '13 at 6:59
@Anon Well, I'm using Windows 8.1 Mathematica v 9.0.1 and I get 0.76 for All in one vs 1.17 for Table – RunnyKine Nov 6 '13 at 7:12
@Wolfies: thanks for your comment, it's interesting. For my actual program, I'm making a huge list of tuples of 6, and am doing rather large calculations with each tuple. Until now, I did the "all in one" method. I will try the other method! – Gabriel Nov 6 '13 at 8:26

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.