Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I have a *.dat file in which the contents are basically one giant list:

{1,2,3,4} {5,6,7,8} {9,10,11,12}

and so on. I'd like to read in the first 500 members { } .... { } in to do some computations. The problem is that the file is about 1.5 GB. How does one deal with large files of this type in Mathematica?

share|improve this question
2  
Actually, this SO discussion "Way to deal with large data files in Wolfram Mathematica" looks pretty key for this. stackoverflow.com/questions/2370570/… –  programming_historian Nov 15 '12 at 14:59
4  
this answer may be relevant. –  Silvia Nov 15 '12 at 16:50
2  
There was a talk about this in the 2011 conference: "BigData: Demystifying Large Datasets in Mathematica with Nick Lariviere" wolfram.com/events/technology-conference/2011/videos.html –  Gustavo Delfino Nov 15 '12 at 17:16
1  
My framework has not been yet optimized to work with lists where every part is very small but the number of parts is just huge. OTOH, it is relatively straightforward to do. The framework itself need not be modified, since this scenario can be addressed by writing some code on top of it. I will try to work out some example tomorrow. –  Leonid Shifrin Nov 15 '12 at 18:55
1  
For large files I find it nice to monitor the progress of the Read, see my answer at: mathematica.stackexchange.com/questions/4640/… –  s0rce Nov 15 '12 at 20:00

1 Answer 1

You can read in the first 500 elements like this:

data = ToExpression@ReadList["myfile.txt", Record, 500, RecordSeparators -> " "];

On Linux/OS X, items 500 to 1000 can be read in this way:

n = 500; m = 1000;

data =ReadList["!cat myfile.txt | tr ' ' '\n'  | head -" <> 
          ToString@m <>" | tail -" <> ToString[m - (n - 1) ]];

Assuming your record separator is a space, and that you might want to ignore records which are not of the form, {...}, you could use the following to find records n through m:

n = 1; m = 500;

data = ReadList["!cat myfile.txt | tr ' ' '\n' | grep '{.*}' | head -" <> 
  ToString@m <> " | tail -" <> ToString[m - (n - 1) ]];
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.