We have 70 .dat (text) files each of which contains a column of numbers. This column can be assumed as a vector obtained from another program. The Length of each column is 1.67772*10^7. We wish to import them (from 1 to 70 text files) in Mathematica notebook in order to apply some procedures.

    vector[i] = 
     Flatten@Import["obtainedvector" <> ToString[i] <> ".dat", "Table"]
      , {i, 1,70}];

But when the importing process reaches i=20, Mathematica does not continue the importing process, and every thing in Ram vanishes as if nothing has been done previously!! There are two solutions: 1- A way to importing faster without encountering the problem (to which I am not familiar ) 2- There is not any necessity to import all vectors and we can do the procedures considered in a Mathematica notebook directly using a text file and without importing them (unfortunately I do not know its details).

I would be so grateful if someone helps me to overcome this problem.


1 Answer 1


Try ReadList. It's lower-level, faster, and less memory intensive.

@AlexyPopkov points out that we can do this:

ReadList["~/Desktop/bleh.dat", {Number, Number}] // 
  MaxMemoryUsed // AbsoluteTiming

{0.930234, 105096624}

And it's even better than my original proposal with Partition.

Export["~/Desktop/bleh.dat", RandomReal[{-1, 1}, {10000, 2}], "Table"];

Partition[ReadList["~/Desktop/bleh.dat", Number], 2] === 
 Import["~/Desktop/bleh.dat", "Table"]


Partition[ReadList["~/Desktop/bleh.dat", Number], 2] // MaxMemoryUsed


Import["~/Desktop/bleh.dat", "Table"] // MaxMemoryUsed


By the way, it might make sense to process at the time of import, rather than importing everything then processing. It'll decrease memory usage.

  • $\begingroup$ Very interesting!! $\endgroup$ Aug 2, 2017 at 5:46
  • $\begingroup$ Your suggested title is very better than mine. Thanks a lot. $\endgroup$ Aug 2, 2017 at 6:41
  • $\begingroup$ With ReadList["~/Desktop/bleh.dat", {Number, Number}] you don't need Partition. $\endgroup$ Aug 2, 2017 at 6:44
  • $\begingroup$ @AlexeyPopkov Oh of course. I always forget the more complex specs. Let me update. $\endgroup$
    – b3m2a1
    Aug 2, 2017 at 6:45
  • $\begingroup$ Because I have just one column I did not considered to {Number, Number} but it is very useful $\endgroup$ Aug 2, 2017 at 6:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.