19
$\begingroup$

I have a *.dat file in which the contents are basically one giant list: 

{1,2,3,4} {5,6,7,8} {9,10,11,12}

and so on. I'd like to read in the first 500 members { } .... { } in to do some computations. The problem is that the file is about 1.5 GB. How does one deal with large files of this type in Mathematica?

Improve this question
$\endgroup$
13
  • 2
    $\begingroup$ Actually, this SO discussion "Way to deal with large data files in Wolfram Mathematica" looks pretty key for this. stackoverflow.com/questions/2370570/… $\endgroup$
    – canadian_scholar
    Commented Nov 15, 2012 at 14:59
  • 4
    $\begingroup$ this answer may be relevant. $\endgroup$
    – Silvia
    Commented Nov 15, 2012 at 16:50
  • 2
    $\begingroup$ There was a talk about this in the 2011 conference: "BigData: Demystifying Large Datasets in Mathematica with Nick Lariviere" wolfram.com/events/technology-conference/2011/videos.html $\endgroup$
    – Gustavo Delfino
    Commented Nov 15, 2012 at 17:16
  • 1
    $\begingroup$ My framework has not been yet optimized to work with lists where every part is very small but the number of parts is just huge. OTOH, it is relatively straightforward to do. The framework itself need not be modified, since this scenario can be addressed by writing some code on top of it. I will try to work out some example tomorrow. $\endgroup$
    – Leonid Shifrin
    Commented Nov 15, 2012 at 18:55
  • 1
    $\begingroup$ For large files I find it nice to monitor the progress of the Read, see my answer at: mathematica.stackexchange.com/questions/4640/… $\endgroup$
    – s0rce
    Commented Nov 15, 2012 at 20:00

2 Answers 2

Reset to default
13
$\begingroup$

You can read in the first 500 elements like this:

data = ToExpression@ReadList["myfile.txt", Record, 500, RecordSeparators -> " "];

On Linux/OS X, items 500 to 1000 can be read in this way:

n = 500; m = 1000;

data =ReadList["!cat myfile.txt | tr ' ' '\n'  | head -" <> 
          ToString@m <>" | tail -" <> ToString[m - (n - 1) ]];

Assuming your record separator is a space, and that you might want to ignore records which are not of the form, {...}, you could use the following to find records n through m:

n = 1; m = 500;

data = ReadList["!cat myfile.txt | tr ' ' '\n' | grep '{.*}' | head -" <> 
  ToString@m <> " | tail -" <> ToString[m - (n - 1) ]];
Improve this answer
$\endgroup$
1
  • $\begingroup$ It would be great if you explain what is the meaning of the argument inside ReadList in the second and third examples. I have another separator and I am looking to extract the last record. $\endgroup$
    – Basheer Algohi
    Commented Sep 15, 2017 at 19:56
6
$\begingroup$

Depending which kind of processing you need, you can try to digest the information as you read, line by line. I wrote this to answer another question about sorting long files. Its based on OpenRead and Read. In this case it reads one line, stores only one number from the file and the StreamPosition in a table for indexing.

strR = OpenRead["UnSorted.txt"]

nList = Table[
   With[{sp = StreamPosition[strR], 
     line = Read[strR, {"Word", "Number", "String"}]}, {sp, line[[2]]}]
   , {i, 10^6}
   ];

Close[strR]
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.