# Import strategies for large .txt files ~600MB

I'm quite new to large data processing in Mathematica and got the task to Import and Plot some .txt measurement data. The files are very very large ~600 MB and have about 3*E7 lines, I can plot and process these sort of data if they were in small files. There I Import the file, read the lines, arrange a table and plot it using ListPlot or ListLinePlot, maybe I smoothen the plots with a slicing command of every n-th line (skip every n-th line of the data in the plot). My data processing now takes hours, about 27 minutes per file, and more comming soon! I import the files with the following command right now, where "names" are the selected files in the directory:

names = SystemDialogInput["FileOpen", FileNameJoin[{directory}]];

data = Import[names, {"TSV", "Data"}, NumberPoint -> ","];


In other posts I found the ReadList command... My problem is now that I can't deal with these Input streams. Further i would like to plot the data accurately, because i never used the ImportStream command. The files are composed out of one header line and 3 columns of values (using a "," as decimal separator) like shown below (i need to use the 2nd as the x-axis data and the 3rd column as the y-axis data). Here is some of the data to give you an idea what the structutre looks like:

t d delta_Tw

0,0200 0,0690 1,3296

0,0400 0,0694 1,4288

0,0600 0,0697 1,5167

0,0800 0,0698 1,5304

0,1000 0,0698 1,5200

0,1200 0,0698 1,5095 Due to the large amount of lines maybe a MovingAverage Line can be plotted as a "trend line", here the first few lines are shown in the "settle time" of the system, after that short time the system starts to oscillate.

I hope you can provide some help how i can import these huge files and give me some advice how to process these data as fast as possible. It would be nice if you can give me some advice but solely in Mathematica, which will be our tool of choice and help me with some specific code examples what would be very helpful :)!

• It seems you are dealing a large CSV file so this post might be useful: mathematica.stackexchange.com/a/249425/260 Nov 18, 2021 at 14:40
• What do you mean by "a slicing command of every x-th line" ? Let's say you can read in the first 100 lines of data, is it sufficient to plot something? Could you please add that minimum sized sample and create a representative plot to show what you exactly want? Right now the question is a bit vague.
– Syed
Nov 18, 2021 at 16:21
• Another option is to load the data files into a database. Mathematica works great with databases and often you can let the database do some of the initial processing. Nov 18, 2021 at 17:27
• Can you parallelize processing multiple files at once via the various Parallel options available assuming you have multi-core machine? This cut the time by about 4 on a quad-core. Maybe set up ParallelTable to do multiple files simultansously.
– josh
Nov 18, 2021 at 20:38
• some specific code examples would be very helpful Nov 19, 2021 at 7:21

In fact your files are not that large, simply txt format is inefficient for the data storage. You mention 30 millions lines containing 3 floating point numbers. If you store them in a binary format assuming double precision

$$3\cdot10^7\times 3 \times 8 /2^{20} = 687\text{MB}.$$

This is very close to 600MB that you mention. But reading your post I realize that your data does not have the precision of 15 digits (double precision). If one uses single precision for the storage,

$$3\cdot10^7\times 3 \times 4 /2^{20} = 343\text{MB}$$

required. Furthermore, I noticed that your data is equal spaced. This brings us to

$$3\cdot10^7\times 2 \times 4 /2^{20} = 229\text{MB}.$$

Thus, my recommendation is to store everything in a binary format.

a = {{0.0200, 0.0690, 1.3296},
{0.0400, 0.0694, 1.4288},
{0.0600, 0.0697, 1.5167},
{0.0800, 0.0698, 1.5304},
{0.1000, 0.0698, 1.5200},
{0.1200, 0.0698, 1.5095}};

fileIn = "~/Documents/debug.dat"
(* prepare some binary file *)
(* this part should be done with some other program *)
str = OpenWrite[fileIn, BinaryFormat -> True]
BinaryWrite[str, Dimensions[a] // Last, "Integer32"]
Do[BinaryWrite[str, z, "Real32"], {z, a}];
Close[str]


It is assumed that at this point there is a large binary file ~/Documents/debug.dat with all the data. After that you can use the following strategy

(* open a data file in binary format *)
str = OpenRead[fileIn, BinaryFormat -> True]
(* read a header indicating the structure of the data, e.g. number of records and columns *)
(* depending on your needs you may or may not need such a header *)
(* read all data at once *)
(* verify if your data has correct dimensions*)
Dimensions[data]
(* finalize *)
Close[str]


Now do the plots

data2D = Partition[data, dim];
ListPlot[data2D[[All, {1, 2}]]]

• You might be right, if there were just 3 columns, in fact there are also time formated columns, but our interest is just for 3 of them. And after that i can simply plot the data with the known Plot[data] command? Nov 23, 2021 at 10:46
• @ChristophBleicher Just update your post with a short example of your data in normal txt format. You can have just 5 rows instead of 30000000. This would be enough for illustration. One can further read txt in your way and save as binary for illustration. In real world you can save your data directly in binary format or use some scripting language (perl, python) to convert to the binary format. Nov 23, 2021 at 10:53
• should i upload it directly or is there a possibility to share it with you? Nov 23, 2021 at 11:24
• i thought it would not be possible to import a .txt file in the post Nov 23, 2021 at 11:41
• i see, now i gave a short example to copy into a notepad text file or whatever... Nov 23, 2021 at 11:53