# Dealing with large files

I have a *.dat file in which the contents are basically one giant list:

{1,2,3,4} {5,6,7,8} {9,10,11,12}


and so on. I'd like to read in the first 500 members { } .... { } in to do some computations. The problem is that the file is about 1.5 GB. How does one deal with large files of this type in Mathematica?

• Actually, this SO discussion "Way to deal with large data files in Wolfram Mathematica" looks pretty key for this. stackoverflow.com/questions/2370570/… – canadian_humanist Nov 15 '12 at 14:59
• this answer may be relevant. – Silvia Nov 15 '12 at 16:50
• There was a talk about this in the 2011 conference: "BigData: Demystifying Large Datasets in Mathematica with Nick Lariviere" wolfram.com/events/technology-conference/2011/videos.html – Gustavo Delfino Nov 15 '12 at 17:16
• My framework has not been yet optimized to work with lists where every part is very small but the number of parts is just huge. OTOH, it is relatively straightforward to do. The framework itself need not be modified, since this scenario can be addressed by writing some code on top of it. I will try to work out some example tomorrow. – Leonid Shifrin Nov 15 '12 at 18:55
• For large files I find it nice to monitor the progress of the Read, see my answer at: mathematica.stackexchange.com/questions/4640/… – s0rce Nov 15 '12 at 20:00

You can read in the first 500 elements like this:

data = ToExpression@ReadList["myfile.txt", Record, 500, RecordSeparators -> " "];


On Linux/OS X, items 500 to 1000 can be read in this way:

n = 500; m = 1000;

data =ReadList["!cat myfile.txt | tr ' ' '\n'  | head -" <>
ToString@m <>" | tail -" <> ToString[m - (n - 1) ]];


Assuming your record separator is a space, and that you might want to ignore records which are not of the form, {...}, you could use the following to find records n through m:

n = 1; m = 500;

data = ReadList["!cat myfile.txt | tr ' ' '\n' | grep '{.*}' | head -" <>
ToString@m <> " | tail -" <> ToString[m - (n - 1) ]];

• It would be great if you explain what is the meaning of the argument inside ReadList in the second and third examples. I have another separator and I am looking to extract the last record. – Algohi Sep 15 '17 at 19:56

Depending which kind of processing you need, you can try to digest the information as you read, line by line. I wrote this to answer another question about sorting long files. Its based on OpenRead and Read. In this case it reads one line, stores only one number from the file and the StreamPosition in a table for indexing.

strR = OpenRead["UnSorted.txt"]

nList = Table[
With[{sp = StreamPosition[strR],
line = Read[strR, {"Word", "Number", "String"}]}, {sp, line[[2]]}]
, {i, 10^6}
];

Close[strR]