2
$\begingroup$

Let's assume that the data:

  • are arranged in a table of $n$ rows by $m$ columns,

  • are stored in a *.dat file or other textual format,

  • number of rows is very large, say $n=10^7$.

Suppose we need to import only a 'small' part from the beginning of a data file, say the first $k = 10^4$ rows.

This is useful, among others at the stage of program development to make it scalable, and in particular to switch between a toy example and work on a full set of data.

Of course, I can import the whole file, get a table, and then select the first $k$ rows from it

Import["Data.dat"][[1;;k]]

The result is in line with expectations [illustrating content]:

{
{1 a, 1 b, 1 c},
{2 a, 2 b, 2 c},
...,
{10^4 a, 10^4 b, 10^4 c}
}

The thing is that the import takes a very long time and involves a lot of memory regardless of whether we use the entire data set or just a small part of it.

I would rather order the reading of data only to the row $k$ at the stage of the import procedure and not after it.

Does anyone from Dear Colleagues know how to achieve it?

$\endgroup$
  • 6
    $\begingroup$ You can take a look at the documentation page for a specific import format. For "CSV" for example, the following works and is significantly faster than importing the whole file: Import["data.csv", {"Data", ;; 10}] $\endgroup$ – Lukas Lang Apr 27 at 12:57
  • $\begingroup$ I found that the fallowing topics: mathematica.stackexchange.com/questions/14656/… and mathematica.stackexchange.com/questions/23980/… are related. As well as references therein. $\endgroup$ – Druid Apr 27 at 15:02
  • 1
    $\begingroup$ CSV and TSV are indeed well optimized at this point, unfortunately Table is not yet. If CSV is not an option, ReadList[.., RecordLists -> True] may be your best bet for now with dat data. $\endgroup$ – GenericAccountName Apr 27 at 15:04
  • 1
    $\begingroup$ Thank you both, Fellows. Esp. @GenericAccountName. Your suggestion concerning the TSV was exactly what I needed. It turned out that the DAT files generated in Mathematica are in fact internally formatted as TSV (at least when exporting a 2D table). After a simple change to the file extension, you can work with files created as DAT by Import["data.TSV", {"Data",1;;10^4}]. $\endgroup$ – Druid Apr 27 at 15:25
  • $\begingroup$ Glad to hear it helped, I recommend exporting to CSV or TSV directly as well. $\endgroup$ – GenericAccountName Apr 27 at 15:58
1
$\begingroup$

If the DAT files are themselves generated in Mathematica, they are in fact internally formated as TSV. It works that way at least in case of 2D table exported to DAT file without any aditional settings. They differ in the extension of the file name.

After a simple change to the filename extension DAT->TSV, one can work with files created as DAT by

Import["data.TSV",{"Data",1;;10^4}]
$\endgroup$
  • 1
    $\begingroup$ You can always specify the format regardless of filename: Import["data.dat",{"TSV","Data",1;;10^4}] $\endgroup$ – GenericAccountName Apr 27 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.