# How to Import random elements of huge data files

I am calculating huge data files with an external program. I would like then to import the data into Mathematica for analysis. The files are 2 columns and up to many millions of rows.

So for small data files I have just been using:

dataTable = Import["data.txt", "Table"];


However once the files gets so large (into the millions), Mathematica and my computer slow down considerably. Now, I don't really care about most of the data in these files and in Mathematica I end up only using several thousand entries. So my question is, can I import a random sampling from the large data file (say 10000 elements only) instead of importing the entire file? So if I had a file with 10^6 rows, I would like to import just 10^4 of those rows (preferably randomly).

-
– cormullion Apr 24 '13 at 20:30
This may be also relevant. – Leonid Shifrin Apr 24 '13 at 21:17
Isn't there some way to do this within the Import statement, by specifying elements? Unfortunately the documentation on Import-elements is not good, so I am also wondering how to use this. thanks, – user9432 Sep 9 '13 at 11:53

Via @chuy and @rcollyer with sequential skipping and reading of records, and without backtracking.

(
If[# != 1, Skip[s, Record, # - 1], Null];
ReadList[s, Number, 2]) & /@ Differences[p~Prepend~0]


List p of lines you want to read, sorted; eg. some random ten from a hundred:

p = Sort[RandomSample[Range[100], 10]];


Your file opened as a stream, eg. s = OpenRead["data.txt"].

-
You don't need to explicitly put Null in. If is perfectly fine accepting only two arguments, and will return Null, if the answer is False. The only time If will return unevaluated is if the condition's truth value cannot be determined and a fourth argument was not supplied. – rcollyer Apr 25 '13 at 12:29

Surely not the fastest or ideal solution, but this should work (it assumes that you have two numbers in each row):

randomline[str_InputStream, num_] := (
SetStreamPosition[ str, 0];
Skip[str, Record, num-1];



Now, if you have 2000 lines and want 100 random samples:

Map[randomline[str, #] &, RandomSample[Range[2000], 100]]


This might provide a jumping-off point for a more sophisticated answer.

-
I would sort the line numbers, first, that way you don't have to backtrack. If you need them back in the original, unsorted order, use Ordering to capture it. – rcollyer Apr 24 '13 at 20:59

You'll want to use Algorithm R for reservoir sampling to get a truly random sample (as opposed to periodic) with only one scan of the input and no need to know the length of the input in advance. Basically, you take the first items and then randomly replace them with decreasing frequency as you complete the scan.

sampleSize = 100;