Return to Answer

slightly shortened code

Source Link

edited May 1, 2013 at 5:57

2.3k
18
20

You'll want to use Algorithm R for reservoir sampling to get a truly random sample (as opposed to periodic) with only one scan of the input and no need to know the length of the input in advance. Basically, you take the first items and then randomly replace them with decreasing frequency as you complete the scan.

sampleSize = 100;

stream = OpenRead["data.txt"];
read[] := Read[stream, {Number, Number}]

result = ConstantArray[0, sampleSize];

Do[result[[i]] = read[]ReadList[stream, {iNumber, sampleSizeNumber}];, sampleSize];

i = sampleSize;
While[(next = read[]) =!= EndOfFile, 
 i++; j = RandomInteger[{1, i++i}]; 
 If[j <= sampleSize, result[[j]] = next]]

sampleSize = 100;

stream = OpenRead["data.txt"];
read[] := Read[stream, {Number, Number}]

result = ConstantArray[0, sampleSize];

Do[result[[i]] = read[], {i, sampleSize}];
i = sampleSize;
While[(next = read[]) =!= EndOfFile, 
 i++; j = RandomInteger[{1, i}]; 
 If[j <= sampleSize, result[[j]] = next]]

sampleSize = 100;

stream = OpenRead["data.txt"];
read[] := Read[stream, {Number, Number}]

result = ReadList[stream, {Number, Number}, sampleSize];

i = sampleSize;
While[(next = read[]) =!= EndOfFile, j = RandomInteger[{1, ++i}];
 If[j <= sampleSize, result[[j]] = next]]

Source Link

created May 1, 2013 at 5:30

Michael Hale

2.3k
18
20

sampleSize = 100;

stream = OpenRead["data.txt"];
read[] := Read[stream, {Number, Number}]

result = ConstantArray[0, sampleSize];

Do[result[[i]] = read[], {i, sampleSize}];
i = sampleSize;
While[(next = read[]) =!= EndOfFile, 
 i++; j = RandomInteger[{1, i}]; 
 If[j <= sampleSize, result[[j]] = next]]