Memory consumption for importing numerical data from an ascii file

Question

I am importing a file which contains data consinsting of 8918347 lines and 6 columns of the following type:

1187.40   725.79     4.04    -7.51  0103753  0913
1363.31   724.57     0.00     0.00  0103754  0913
 564.34   725.77     0.00     0.00  0103755  0913
1174.40   726.31     0.00     0.00  0103756  0913
 527.56   728.31    -3.82     1.38  0103757  0913
 543.29   727.94    -0.80     0.24  0103758  0913
1510.04   729.10     0.00     0.00  0103760  0913
1565.91   738.97     1.80     1.37  0103762  0913
. . . 

input = Import["data.txt", "Table"];

The file is accessible here:

https://drive.google.com/open?id=19o4_YruFGJh3tBgV7GlClcbvcvl0MsNf

• Before I import the data Wolfram Mathematica has allocated 431 MB (fresh started)

• During the reading the allocated memory of Wolfram Mathematica goes up to about 8700 MB.

• Then still during the line is executed the memory slowly goes down to 2330 MB and the line is finally executed.

The Import takes on my computer 135 sec.

I would like to understand:

1. Why does this up and down of allocated memory occurs during importing.

2. Why ist so much memory (2330 MB - 431 MB = 1899 MB) used?

If I assume that each number uses 8 Byte. Then this results in:

8*8918347*6 Bytes = 428080656 Bytes = 408.35 MB

My computer system is: Windows 10 Pro newest update, Ram: 32 GB, Notebook Dell Precision M4800, Intel Core i7-4940MX 3.1GHz

Answer 1

I can only guess that the inner workings of the parsing of data with an unknown shape may be complicated. The memory used contains several copies of the data been processed.

If you want to have some fun seeing how Mathematica works inside, look at Trace. I suggest you start using trace in much simpler expressions. If you need to Trace more complex expressions see this questions, to trace to a file

An alternative that uses much less memory and it's faster:

path = "20180720_01_data.txt"

First@AbsoluteTiming[
  data2 = 
    ReadList[path, {Number, Number, Number, Number, Number, Number}];
  ]
(* 19.5111 *)

Compared with

First@AbsoluteTiming[
  data1 = Import[path, "Table"];
  ]
(* 172.437 *)

Check that the data is the same.

data1 == data2
(* True *)

Even better, with ReadList and family of function for Streams and Low-Level Input and Output you could choose which lines and columns to Read and what to Skip.