Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I am calculating huge data files with an external program. I would like then to import the data into Mathematica for analysis. The files are 2 columns and up to many millions of rows.

So for small data files I have just been using:

dataTable = Import["data.txt", "Table"]; 

However once the files gets so large (into the millions), Mathematica and my computer slow down considerably. Now, I don't really care about most of the data in these files and in Mathematica I end up only using several thousand entries. So my question is, can I import a random sampling from the large data file (say 10000 elements only) instead of importing the entire file? So if I had a file with 10^6 rows, I would like to import just 10^4 of those rows (preferably randomly).

share|improve this question
    
    
This may be also relevant. –  Leonid Shifrin Apr 24 '13 at 21:17
    
Isn't there some way to do this within the Import statement, by specifying elements? Unfortunately the documentation on Import-elements is not good, so I am also wondering how to use this. thanks, –  Mark Sep 9 '13 at 11:53
add comment

3 Answers

up vote 5 down vote accepted

Via @chuy and @rcollyer with sequential skipping and reading of records, and without backtracking.

(
   If[# != 1, Skip[s, Record, # - 1], Null];
   ReadList[s, Number, 2]) & /@ Differences[p~Prepend~0]

List p of lines you want to read, sorted; eg. some random ten from a hundred:

p = Sort[RandomSample[Range[100], 10]];

Your file opened as a stream, eg. s = OpenRead["data.txt"].

share|improve this answer
4  
You don't need to explicitly put Null in. If is perfectly fine accepting only two arguments, and will return Null, if the answer is False. The only time If will return unevaluated is if the condition's truth value cannot be determined and a fourth argument was not supplied. –  rcollyer Apr 25 '13 at 12:29
add comment

Surely not the fastest or ideal solution, but this should work (it assumes that you have two numbers in each row):

randomline[str_InputStream, num_] := (
  SetStreamPosition[ str, 0];
  Skip[str, Record, num-1];
  Read[str, {Number, Number}])

str = OpenRead["data.txt"];

Now, if you have 2000 lines and want 100 random samples:

Map[randomline[str, #] &, RandomSample[Range[2000], 100]]

This might provide a jumping-off point for a more sophisticated answer.

share|improve this answer
4  
I would sort the line numbers, first, that way you don't have to backtrack. If you need them back in the original, unsorted order, use Ordering to capture it. –  rcollyer Apr 24 '13 at 20:59
add comment

You'll want to use Algorithm R for reservoir sampling to get a truly random sample (as opposed to periodic) with only one scan of the input and no need to know the length of the input in advance. Basically, you take the first items and then randomly replace them with decreasing frequency as you complete the scan.

sampleSize = 100;

stream = OpenRead["data.txt"];
read[] := Read[stream, {Number, Number}]

result = ReadList[stream, {Number, Number}, sampleSize];

i = sampleSize;
While[(next = read[]) =!= EndOfFile, j = RandomInteger[{1, ++i}];
 If[j <= sampleSize, result[[j]] = next]]
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.