# How to import only a small part from a large data file?

Let's assume that the data:

• are arranged in a table of $$n$$ rows by $$m$$ columns,

• are stored in a *.dat file or other textual format,

• number of rows is very large, say $$n=10^7$$.

Suppose we need to import only a 'small' part from the beginning of a data file, say the first $$k = 10^4$$ rows.

This is useful, among others at the stage of program development to make it scalable, and in particular to switch between a toy example and work on a full set of data.

Of course, I can import the whole file, get a table, and then select the first $$k$$ rows from it

Import["Data.dat"][[1;;k]]


The result is in line with expectations [illustrating content]:

{
{1 a, 1 b, 1 c},
{2 a, 2 b, 2 c},
...,
{10^4 a, 10^4 b, 10^4 c}
}


The thing is that the import takes a very long time and involves a lot of memory regardless of whether we use the entire data set or just a small part of it.

I would rather order the reading of data only to the row $$k$$ at the stage of the import procedure and not after it.

Does anyone from Dear Colleagues know how to achieve it?

• You can take a look at the documentation page for a specific import format. For "CSV" for example, the following works and is significantly faster than importing the whole file: Import["data.csv", {"Data", ;; 10}] Apr 27, 2019 at 12:57
• I found that the fallowing topics: mathematica.stackexchange.com/questions/14656/… and mathematica.stackexchange.com/questions/23980/… are related. As well as references therein. Apr 27, 2019 at 15:02
• CSV and TSV are indeed well optimized at this point, unfortunately Table is not yet. If CSV is not an option, ReadList[.., RecordLists -> True] may be your best bet for now with dat data. Apr 27, 2019 at 15:04
• Thank you both, Fellows. Esp. @GenericAccountName. Your suggestion concerning the TSV was exactly what I needed. It turned out that the DAT files generated in Mathematica are in fact internally formatted as TSV (at least when exporting a 2D table). After a simple change to the file extension, you can work with files created as DAT by Import["data.TSV", {"Data",1;;10^4}]. Apr 27, 2019 at 15:25
• Glad to hear it helped, I recommend exporting to CSV or TSV directly as well. Apr 27, 2019 at 15:58

Import["data.TSV",{"Data",1;;10^4}]

• You can always specify the format regardless of filename: Import["data.dat",{"TSV","Data",1;;10^4}] Apr 27, 2019 at 16:00