For testing purposes, I'd like to create very large amounts of Random Integers, let's say between $-1000$ and $1000$. I'm using the usual

RandomInteger[{-1000, 1000}, 5]

command to create $5$-tuples. On my machine (8GB of RAM), this process becomes quite slow once in the

   RandomInteger[{-1000, 1000}, 5]

(one hundred million) region. It takes 17 seconds of Timing[] for this list to be made, and I haven't even had any calculations on it.

Does this mean I should restrain using lists till somewhat lower as this threshold, or is there any other way of handling huge lists? Does MMA input the whole list in RAM before doing calculations with it? Is it possible to handle huge lists without including the whole list in RAM, for speeding purposes?

I thought of dividing the list, perhaps exporting to different files (CSV for example), and then doing calculations with every file individually. This is probably the same as Partition[] does (without exporting)?

Thanks for any more suggestions!

  • 2
    $\begingroup$ Little remark: RandomInteger[{-1000, 1000}, {100000000, 5}] is about 5 times faster. $\endgroup$
    – Kuba
    Nov 6, 2013 at 6:14
  • $\begingroup$ You don't need Table for this. See @Kuba suggestion. $\endgroup$
    – RunnyKine
    Nov 6, 2013 at 6:15
  • $\begingroup$ Like the big-list tag: funny. $\endgroup$
    – wolfies
    Nov 6, 2013 at 6:31

1 Answer 1


Usually, there are efficiency advantages to generating all of your random numbers in one go, but in this instance, it makes little difference if you generate all in one go, or in tuples of 5 (million of times). I am not sure what you are doing with them, but usually one is doing something like computing a sample mean, or adding them up or similar:


Map[ Total, RandomInteger[{-1000, 1000}, {10000000, 5}]]; // AbsoluteTiming

{2.954751, Null}

10 million tuples of 5:

Table[ Total[RandomInteger[{-1000, 1000}, 5]], {10000000}] ; // AbsoluteTiming 

{2.678985, Null}

And, having to use virtual memory, and your hard drive thrashing ... and your system slowing to a crawl is not exactly fun or efficient ... so working with individual tuples of 5 makes good sense here.

The advantage of working with individual tuples of 5 is that you can reduce the memory used to about 1/4:


Map[ Total, RandomInteger[{-1000, 1000}, {10000000, 5}]]; // MaxMemoryUsed


10 million tuples of 5:

Table[ Total[RandomInteger[{-1000, 1000}, 5]], {10000000}] ; // MaxMemoryUsed 


[ To illustrate, I am using 10 million tuples here --- not the 100 million in your example. Depending on the example, you can probably break it up into smaller units of 10 million, and then average your results etc ]

  • $\begingroup$ I can't reproduce your Timings on my machine. All in one seems to be faster. Also it becomes even faster if you use the second argument of Total instead of Map as follows: Total[RandomInteger[{-1000, 1000}, {1*^7, 5}], {2}] $\endgroup$
    – RunnyKine
    Nov 6, 2013 at 6:53
  • $\begingroup$ Also, if you ignore the Total part of the calculation and just generate the List. The All in one method uses half as much memory on my PC. $\endgroup$
    – RunnyKine
    Nov 6, 2013 at 6:59
  • $\begingroup$ Table is faster for me, 1.17 versus 1.23. OS X, ver 9.0. $\endgroup$
    – C. E.
    Nov 6, 2013 at 6:59
  • $\begingroup$ @Anon Well, I'm using Windows 8.1 Mathematica v 9.0.1 and I get 0.76 for All in one vs 1.17 for Table $\endgroup$
    – RunnyKine
    Nov 6, 2013 at 7:12
  • $\begingroup$ @Wolfies: thanks for your comment, it's interesting. For my actual program, I'm making a huge list of tuples of 6, and am doing rather large calculations with each tuple. Until now, I did the "all in one" method. I will try the other method! $\endgroup$
    – Gabriel
    Nov 6, 2013 at 8:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.