Decreasing memory consumption during Import

We have 70 .dat (text) files each of which contains a column of numbers. This column can be assumed as a vector obtained from another program. The Length of each column is 1.67772*10^7. We wish to import them (from 1 to 70 text files) in Mathematica notebook in order to apply some procedures.

Do[
vector[i] =
Flatten@Import["obtainedvector" <> ToString[i] <> ".dat", "Table"]
, {i, 1,70}];


But when the importing process reaches i=20, Mathematica does not continue the importing process, and every thing in Ram vanishes as if nothing has been done previously!! There are two solutions: 1- A way to importing faster without encountering the problem (to which I am not familiar ) 2- There is not any necessity to import all vectors and we can do the procedures considered in a Mathematica notebook directly using a text file and without importing them (unfortunately I do not know its details).

I would be so grateful if someone helps me to overcome this problem.

Try ReadList. It's lower-level, faster, and less memory intensive.

@AlexyPopkov points out that we can do this:

ReadList["~/Desktop/bleh.dat", {Number, Number}] //
MaxMemoryUsed // AbsoluteTiming

{0.930234, 105096624}


And it's even better than my original proposal with Partition.

Export["~/Desktop/bleh.dat", RandomReal[{-1, 1}, {10000, 2}], "Table"];

Import["~/Desktop/bleh.dat", "Table"]

True

1120728

Import["~/Desktop/bleh.dat", "Table"] // MaxMemoryUsed

5741304


By the way, it might make sense to process at the time of import, rather than importing everything then processing. It'll decrease memory usage.

• Very interesting!! – Inzo Babaria Aug 2 '17 at 5:46
• Your suggested title is very better than mine. Thanks a lot. – Inzo Babaria Aug 2 '17 at 6:41
• With ReadList["~/Desktop/bleh.dat", {Number, Number}] you don't need Partition. – Alexey Popkov Aug 2 '17 at 6:44
• @AlexeyPopkov Oh of course. I always forget the more complex specs. Let me update. – b3m2a1 Aug 2 '17 at 6:45
• Because I have just one column I did not considered to {Number, Number} but it is very useful – Inzo Babaria Aug 2 '17 at 6:47