Decreasing memory consumption during Import

Question

We have 70 .dat (text) files each of which contains a column of numbers. This column can be assumed as a vector obtained from another program. The Length of each column is 1.67772*10^7. We wish to import them (from 1 to 70 text files) in Mathematica notebook in order to apply some procedures.

Do[
    vector[i] = 
     Flatten@Import["obtainedvector" <> ToString[i] <> ".dat", "Table"]
      , {i, 1,70}];

But when the importing process reaches i=20, Mathematica does not continue the importing process, and every thing in Ram vanishes as if nothing has been done previously!! There are two solutions: 1- A way to importing faster without encountering the problem (to which I am not familiar ) 2- There is not any necessity to import all vectors and we can do the procedures considered in a Mathematica notebook directly using a text file and without importing them (unfortunately I do not know its details).

I would be so grateful if someone helps me to overcome this problem.

Answer 1

Try ReadList. It's lower-level, faster, and less memory intensive.

@AlexyPopkov points out that we can do this:

ReadList["~/Desktop/bleh.dat", {Number, Number}] // 
  MaxMemoryUsed // AbsoluteTiming

{0.930234, 105096624}

And it's even better than my original proposal with Partition.

Export["~/Desktop/bleh.dat", RandomReal[{-1, 1}, {10000, 2}], "Table"];

Partition[ReadList["~/Desktop/bleh.dat", Number], 2] === 
 Import["~/Desktop/bleh.dat", "Table"]

True

Partition[ReadList["~/Desktop/bleh.dat", Number], 2] // MaxMemoryUsed

1120728

Import["~/Desktop/bleh.dat", "Table"] // MaxMemoryUsed

5741304

By the way, it might make sense to process at the time of import, rather than importing everything then processing. It'll decrease memory usage.