I am importing a file which contains data consinsting of 8918347 lines and 6 columns of the following type:

1187.40   725.79     4.04    -7.51  0103753  0913
1363.31   724.57     0.00     0.00  0103754  0913
 564.34   725.77     0.00     0.00  0103755  0913
1174.40   726.31     0.00     0.00  0103756  0913
 527.56   728.31    -3.82     1.38  0103757  0913
 543.29   727.94    -0.80     0.24  0103758  0913
1510.04   729.10     0.00     0.00  0103760  0913
1565.91   738.97     1.80     1.37  0103762  0913
. . . 

input = Import["data.txt", "Table"];

The file is accessible here:

https://drive.google.com/open?id=19o4_YruFGJh3tBgV7GlClcbvcvl0MsNf

  • Before I import the data Wolfram Mathematica has allocated 431 MB (fresh started)

  • During the reading the allocated memory of Wolfram Mathematica goes up to about 8700 MB.

  • Then still during the line is executed the memory slowly goes down to 2330 MB and the line is finally executed.

The Import takes on my computer 135 sec.

I would like to understand:

  1. Why does this up and down of allocated memory occurs during importing.

  2. Why ist so much memory (2330 MB - 431 MB = 1899 MB) used?

If I assume that each number uses 8 Byte. Then this results in:

8*8918347*6 Bytes = 428080656 Bytes = 408.35 MB

My computer system is: Windows 10 Pro newest update, Ram: 32 GB, Notebook Dell Precision M4800, Intel Core i7-4940MX 3.1GHz

up vote 3 down vote accepted

I can only guess that the inner workings of the parsing of data with an unknown shape may be complicated. The memory used contains several copies of the data been processed.

If you want to have some fun seeing how Mathematica works inside, look at Trace. I suggest you start using trace in much simpler expressions. If you need to Trace more complex expressions see this questions, to trace to a file

An alternative that uses much less memory and it's faster:

path = "20180720_01_data.txt"

First@AbsoluteTiming[
  data2 = 
    ReadList[path, {Number, Number, Number, Number, Number, Number}];
  ]
(* 19.5111 *)

Compared with

First@AbsoluteTiming[
  data1 = Import[path, "Table"];
  ]
(* 172.437 *)

Check that the data is the same.

data1 == data2
(* True *)

Even better, with ReadList and family of function for Streams and Low-Level Input and Output you could choose which lines and columns to Read and what to Skip.

  • Thank you very much for the example how to load such a file much faster … I see this the first time. I will look also in Trace ... – mrz Jul 20 at 18:52
  • 2
    One cute thing I had missed but someone pointed out to me is we can just do ReadList[path, Number, RecordLists->True] and it will automatically determine the table width. – b3m2a1 Jul 20 at 19:18

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.