We have 70 .dat (text) files each of which contains a column of numbers. This column can be assumed as a vector obtained from another program. The Length of each column is 1.67772*10^7. We wish to import them (from 1 to 70 text files) in Mathematica notebook in order to apply some procedures.

    vector[i] = 
     Flatten@Import["obtainedvector" <> ToString[i] <> ".dat", "Table"]
      , {i, 1,70}];

But when the importing process reaches i=20, Mathematica does not continue the importing process, and every thing in Ram vanishes as if nothing has been done previously!! There are two solutions: 1- A way to importing faster without encountering the problem (to which I am not familiar ) 2- There is not any necessity to import all vectors and we can do the procedures considered in a Mathematica notebook directly using a text file and without importing them (unfortunately I do not know its details).

I would be so grateful if someone helps me to overcome this problem.


Try ReadList. It's lower-level, faster, and less memory intensive.

@AlexyPopkov points out that we can do this:

ReadList["~/Desktop/bleh.dat", {Number, Number}] // 
  MaxMemoryUsed // AbsoluteTiming

{0.930234, 105096624}

And it's even better than my original proposal with Partition.

Export["~/Desktop/bleh.dat", RandomReal[{-1, 1}, {10000, 2}], "Table"];

Partition[ReadList["~/Desktop/bleh.dat", Number], 2] === 
 Import["~/Desktop/bleh.dat", "Table"]


Partition[ReadList["~/Desktop/bleh.dat", Number], 2] // MaxMemoryUsed


Import["~/Desktop/bleh.dat", "Table"] // MaxMemoryUsed


By the way, it might make sense to process at the time of import, rather than importing everything then processing. It'll decrease memory usage.

  • $\begingroup$ Very interesting!! $\endgroup$ Aug 2 '17 at 5:46
  • $\begingroup$ Your suggested title is very better than mine. Thanks a lot. $\endgroup$ Aug 2 '17 at 6:41
  • $\begingroup$ With ReadList["~/Desktop/bleh.dat", {Number, Number}] you don't need Partition. $\endgroup$ Aug 2 '17 at 6:44
  • $\begingroup$ @AlexeyPopkov Oh of course. I always forget the more complex specs. Let me update. $\endgroup$
    – b3m2a1
    Aug 2 '17 at 6:45
  • $\begingroup$ Because I have just one column I did not considered to {Number, Number} but it is very useful $\endgroup$ Aug 2 '17 at 6:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.