Import strategies for large .txt files ~600MB

Question

I'm quite new to large data processing in Mathematica and got the task to Import and Plot some .txt measurement data. The files are very very large ~600 MB and have about 3*E7 lines, I can plot and process these sort of data if they were in small files. There I Import the file, read the lines, arrange a table and plot it using ListPlot or ListLinePlot, maybe I smoothen the plots with a slicing command of every n-th line (skip every n-th line of the data in the plot). My data processing now takes hours, about 27 minutes per file, and more comming soon! I import the files with the following command right now, where "names" are the selected files in the directory:

names = SystemDialogInput["FileOpen", FileNameJoin[{directory}]];

data = Import[names, {"TSV", "Data"}, NumberPoint -> ","];

In other posts I found the ReadList command... My problem is now that I can't deal with these Input streams. Further i would like to plot the data accurately, because i never used the ImportStream command. The files are composed out of one header line and 3 columns of values (using a "," as decimal separator) like shown below (i need to use the 2nd as the x-axis data and the 3rd column as the y-axis data). Here is some of the data to give you an idea what the structutre looks like:

t d delta_Tw

0,0200 0,0690 1,3296

0,0400 0,0694 1,4288

0,0600 0,0697 1,5167

0,0800 0,0698 1,5304

0,1000 0,0698 1,5200

0,1200 0,0698 1,5095 Due to the large amount of lines maybe a MovingAverage Line can be plotted as a "trend line", here the first few lines are shown in the "settle time" of the system, after that short time the system starts to oscillate.

I hope you can provide some help how i can import these huge files and give me some advice how to process these data as fast as possible. It would be nice if you can give me some advice but solely in Mathematica, which will be our tool of choice and help me with some specific code examples what would be very helpful :)!

Answer 1

In fact your files are not that large, simply txt format is inefficient for the data storage. You mention 30 millions lines containing 3 floating point numbers. If you store them in a binary format assuming double precision

$$3\cdot10^7\times 3 \times 8 /2^{20} = 687\text{MB}.$$

This is very close to 600MB that you mention. But reading your post I realize that your data does not have the precision of 15 digits (double precision). If one uses single precision for the storage,

$$3\cdot10^7\times 3 \times 4 /2^{20} = 343\text{MB}$$

required. Furthermore, I noticed that your data is equal spaced. This brings us to

$$3\cdot10^7\times 2 \times 4 /2^{20} = 229\text{MB}.$$

Thus, my recommendation is to store everything in a binary format.

a = {{0.0200, 0.0690, 1.3296}, 
     {0.0400, 0.0694, 1.4288}, 
     {0.0600, 0.0697, 1.5167}, 
     {0.0800, 0.0698, 1.5304}, 
     {0.1000, 0.0698, 1.5200}, 
     {0.1200, 0.0698, 1.5095}};

fileIn = "~/Documents/debug.dat"
(* prepare some binary file *)
(* this part should be done with some other program *)
str = OpenWrite[fileIn, BinaryFormat -> True]
BinaryWrite[str, Dimensions[a] // Last, "Integer32"]
Do[BinaryWrite[str, z, "Real32"], {z, a}];
Close[str]

It is assumed that at this point there is a large binary file ~/Documents/debug.dat with all the data. After that you can use the following strategy

(* open a data file in binary format *)
str = OpenRead[fileIn, BinaryFormat -> True]
(* read a header indicating the structure of the data, e.g. number of records and columns *)
(* depending on your needs you may or may not need such a header *)  
dim = BinaryRead[str, "Integer32"]
(* read all data at once *)
data = BinaryReadList[str, "Real32"];
(* verify if your data has correct dimensions*)
Dimensions[data]
(* finalize *)
Close[str]

Now do the plots

data2D = Partition[data, dim];
ListPlot[data2D[[All, {1, 2}]]]