How to import only a small part from a large data file?

Question

Let's assume that the data:

• are arranged in a table of $n$ rows by $m$ columns,

• are stored in a *.dat file or other textual format,

• number of rows is very large, say $n=10^7$.

Suppose we need to import only a 'small' part from the beginning of a data file, say the first $k = 10^4$ rows.

This is useful, among others at the stage of program development to make it scalable, and in particular to switch between a toy example and work on a full set of data.

Of course, I can import the whole file, get a table, and then select the first $k$ rows from it

Import["Data.dat"][[1;;k]]

The result is in line with expectations [illustrating content]:

{
{1 a, 1 b, 1 c},
{2 a, 2 b, 2 c},
...,
{10^4 a, 10^4 b, 10^4 c}
}

The thing is that the import takes a very long time and involves a lot of memory regardless of whether we use the entire data set or just a small part of it.

I would rather order the reading of data only to the row $k$ at the stage of the import procedure and not after it.

Does anyone from Dear Colleagues know how to achieve it?

Answer 1

If the DAT files are themselves generated in Mathematica, they are in fact internally formated as TSV. It works that way at least in case of 2D table exported to DAT file without any aditional settings. They differ in the extension of the file name.

After a simple change to the filename extension DAT->TSV, one can work with files created as DAT by

Import["data.TSV",{"Data",1;;10^4}]