# Random Choice: List Format & Minimizing Memory Use

This is a two part question, and for that I apologize.. but they're related!

Here's what I'm working with:

d1 = Import["file.CSV", "List"]

size = Length[d1]

dis1 = RandomChoice[{d1}, {100, size}]

• Q1: Length views d1 as $300,000$ individual elements but RandomChoice views it as a single element. Thus when I execute I get $100$ repetitions of d1 in the exact order d1 is presented in.

Have I made an annotation error in assigning the RandomChoice number pool as {d1}? If I replace {d1} with a hand written list of values it executes perfectly, so I assume it is a presentation issue.. but I can't tell what I've done wrong from the documentation of the function.

• Q2: Asking to make $100$ random lists of size ~ $300,000$ elements requires a lot of memory. The thing is I don't need to save each RandomChoice output, I just need the mean/median/SD/SEM for each of the $100$ sets I've tasked RandomChoice with.

Is there a way to tell the program to spit out those end-point values and dump the accumulated list after each one is generated?

Q1: You don't need the curly-braces around d1 in the RandomChoice: That turns it into a list with one element - d1.

Q2: If memory utilization is more important than speed (because it is usually far better to generate samples/variates/etc. en masse), you can do something like this:

myBigList = RandomInteger[100, 20000];
listLen = Length[myBigList];

results =
Reap[Do[picks = RandomChoice[myBigList, listLen];
Sow[{Mean[#], Median[#], StandardDeviation[#]} &@picks], {100}]][[2,
1]]


(Replace my RandomInteger with your import).

• Aha! Thank you very much for both answers. The Reap/Sow usage is excellent. And yes, memory is more important currently. I'll be iterating this procedure quite a lot. – Justin Jan 29 '14 at 5:39
• A follow up query.. if anyone has input. Let's say instead of Sowing the Mean/Median/SD I directly take #1 and #2 lists from "picks" and perform a T-test on those and punch out the p-value, then repeat for the next 2 randomchoice sets. Is that possible? I didn't realize Mathematica would compute p-values for whole data sets. – Justin Jan 29 '14 at 6:07
• @justin: Sure. Just replace listLen in the RandomChoice with {2,listLen}, and the whole {Mean[#], Median[#], StandardDeviation[#]} & part with TTest and you're set. Note that you'll get results for pairs of sampled lists, so adjust the iterator if needed. – ciao Jan 29 '14 at 6:36
• Thanks. I appreciate it! I gave that a whirl but was coming up with very large p-values. I suspect the default of TTest is to compare the query set to 0. I made an attempt to recode as follows: results = Reap[Do[pick1 = RandomChoice[d1, {1, listLen}]; pick2 = RandomChoice[d1, {1, listLen}]; Sow[{TTest[pick1, pick2 , "PValue"]}], {1}]][[2, 1]] The goal being to generate the first list as the query set, and the second list as the comparison set. But it keeps telling me my array is the wrong size/shape. Thanks for the input folks! – Justin Jan 29 '14 at 6:49
• @Justin: take the {1,listLen} to just listLen - the way you have it written, you're asking for a list of one element that is a list of listLen choices. You'll also then need to bracket the pick1 and pick2 in the TTest. There's no real need to pick these separately though, since TTest does not "favor" either list - it simply compares if the means are the same, so doing it as I showed is (a bit) faster. – ciao Jan 29 '14 at 7:03

For Q1, your syntax mistake is an extra pair of brackets. The following code should work properly:

d1 = Import["file.CSV", "List"]
size = Length[d1]
dis1 = RandomChoice[d1, {100, size}]