15
$\begingroup$

I am calculating huge data files with an external program. I would like then to import the data into Mathematica for analysis. The files are 2 columns and up to many millions of rows.

So for small data files I have just been using:

dataTable = Import["data.txt", "Table"];

However once the files gets so large (into the millions), Mathematica and my computer slow down considerably. Now, I don't really care about most of the data in these files and in Mathematica I end up only using several thousand entries. So my question is, can I import a random sampling from the large data file (say 10000 elements only) instead of importing the entire file? So if I had a file with 10^6 rows, I would like to import just 10^4 of those rows (preferably randomly).

Improve this question
$\endgroup$
3
  • $\begingroup$ Related: mathematica.stackexchange.com/questions/14656/… $\endgroup$
    – cormullion
    Commented Apr 24, 2013 at 20:30
  • $\begingroup$ This may be also relevant. $\endgroup$
    – Leonid Shifrin
    Commented Apr 24, 2013 at 21:17
  • $\begingroup$ Isn't there some way to do this within the Import statement, by specifying elements? Unfortunately the documentation on Import-elements is not good, so I am also wondering how to use this. thanks, $\endgroup$
    – user9432
    Commented Sep 9, 2013 at 11:53

3 Answers 3

Reset to default
5
$\begingroup$

Via @chuy and @rcollyer with sequential skipping and reading of records, and without backtracking.

(
   If[# != 1, Skip[s, Record, # - 1], Null];
   ReadList[s, Number, 2]) & /@ Differences[p~Prepend~0]

List p of lines you want to read, sorted; eg. some random ten from a hundred:

p = Sort[RandomSample[Range[100], 10]];

Your file opened as a stream, eg. s = OpenRead["data.txt"].

Improve this answer
$\endgroup$
1
  • 4
    $\begingroup$ You don't need to explicitly put Null in. If is perfectly fine accepting only two arguments, and will return Null, if the answer is False. The only time If will return unevaluated is if the condition's truth value cannot be determined and a fourth argument was not supplied. $\endgroup$
    – rcollyer
    Commented Apr 25, 2013 at 12:29
4
$\begingroup$

Surely not the fastest or ideal solution, but this should work (it assumes that you have two numbers in each row):

randomline[str_InputStream, num_] := (
  SetStreamPosition[ str, 0];
  Skip[str, Record, num-1];
  Read[str, {Number, Number}])

str = OpenRead["data.txt"];

Now, if you have 2000 lines and want 100 random samples:

Map[randomline[str, #] &, RandomSample[Range[2000], 100]]

This might provide a jumping-off point for a more sophisticated answer.

Improve this answer
$\endgroup$
1
  • 4
    $\begingroup$ I would sort the line numbers, first, that way you don't have to backtrack. If you need them back in the original, unsorted order, use Ordering to capture it. $\endgroup$
    – rcollyer
    Commented Apr 24, 2013 at 20:59
3
$\begingroup$

You'll want to use Algorithm R for reservoir sampling to get a truly random sample (as opposed to periodic) with only one scan of the input and no need to know the length of the input in advance. Basically, you take the first items and then randomly replace them with decreasing frequency as you complete the scan.

sampleSize = 100;

stream = OpenRead["data.txt"];
read[] := Read[stream, {Number, Number}]

result = ReadList[stream, {Number, Number}, sampleSize];

i = sampleSize;
While[(next = read[]) =!= EndOfFile, j = RandomInteger[{1, ++i}];
 If[j <= sampleSize, result[[j]] = next]]
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.