Random Choice: List Format & Minimizing Memory Use

Question

This is a two part question, and for that I apologize.. but they're related!

Here's what I'm working with:

d1 = Import["file.CSV", "List"]

size = Length[d1]

dis1 = RandomChoice[{d1}, {100, size}]

• Q1: Length views d1 as $300,000$ individual elements but RandomChoice views it as a single element. Thus when I execute I get $100$ repetitions of d1 in the exact order d1 is presented in.

  Have I made an annotation error in assigning the RandomChoice number pool as {d1}? If I replace {d1} with a hand written list of values it executes perfectly, so I assume it is a presentation issue.. but I can't tell what I've done wrong from the documentation of the function.

• Q2: Asking to make $100$ random lists of size ~ $300,000$ elements requires a lot of memory. The thing is I don't need to save each RandomChoice output, I just need the mean/median/SD/SEM for each of the $100$ sets I've tasked RandomChoice with.

  Is there a way to tell the program to spit out those end-point values and dump the accumulated list after each one is generated?

Answer 1

Q1: You don't need the curly-braces around d1 in the RandomChoice: That turns it into a list with one element - d1.

Q2: If memory utilization is more important than speed (because it is usually far better to generate samples/variates/etc. en masse), you can do something like this:

myBigList = RandomInteger[100, 20000];
listLen = Length[myBigList];

results = 
 Reap[Do[picks = RandomChoice[myBigList, listLen]; 
    Sow[{Mean[#], Median[#], StandardDeviation[#]} &@picks], {100}]][[2,
    1]]

(Replace my RandomInteger with your import).

Answer 2

For Q1, your syntax mistake is an extra pair of brackets. The following code should work properly:

d1 = Import["file.CSV", "List"]
size = Length[d1]
dis1 = RandomChoice[d1, {100, size}]