3
$\begingroup$

We have 70 .dat (text) files each of which contains a column of numbers. This column can be assumed as a vector obtained from another program. The Length of each column is 1.67772*10^7. We wish to import them (from 1 to 70 text files) in Mathematica notebook in order to apply some procedures.

Do[
    vector[i] = 
     Flatten@Import["obtainedvector" <> ToString[i] <> ".dat", "Table"]
      , {i, 1,70}];

But when the importing process reaches i=20, Mathematica does not continue the importing process, and every thing in Ram vanishes as if nothing has been done previously!! There are two solutions: 1- A way to importing faster without encountering the problem (to which I am not familiar ) 2- There is not any necessity to import all vectors and we can do the procedures considered in a Mathematica notebook directly using a text file and without importing them (unfortunately I do not know its details).

I would be so grateful if someone helps me to overcome this problem.

$\endgroup$
6
$\begingroup$

Try ReadList. It's lower-level, faster, and less memory intensive.

@AlexyPopkov points out that we can do this:

ReadList["~/Desktop/bleh.dat", {Number, Number}] // 
  MaxMemoryUsed // AbsoluteTiming

{0.930234, 105096624}

And it's even better than my original proposal with Partition.

Export["~/Desktop/bleh.dat", RandomReal[{-1, 1}, {10000, 2}], "Table"];

Partition[ReadList["~/Desktop/bleh.dat", Number], 2] === 
 Import["~/Desktop/bleh.dat", "Table"]

True

Partition[ReadList["~/Desktop/bleh.dat", Number], 2] // MaxMemoryUsed

1120728

Import["~/Desktop/bleh.dat", "Table"] // MaxMemoryUsed

5741304

By the way, it might make sense to process at the time of import, rather than importing everything then processing. It'll decrease memory usage.

$\endgroup$
  • $\begingroup$ Very interesting!! $\endgroup$ – Inzo Babaria Aug 2 '17 at 5:46
  • $\begingroup$ Your suggested title is very better than mine. Thanks a lot. $\endgroup$ – Inzo Babaria Aug 2 '17 at 6:41
  • $\begingroup$ With ReadList["~/Desktop/bleh.dat", {Number, Number}] you don't need Partition. $\endgroup$ – Alexey Popkov Aug 2 '17 at 6:44
  • $\begingroup$ @AlexeyPopkov Oh of course. I always forget the more complex specs. Let me update. $\endgroup$ – b3m2a1 Aug 2 '17 at 6:45
  • $\begingroup$ Because I have just one column I did not considered to {Number, Number} but it is very useful $\endgroup$ – Inzo Babaria Aug 2 '17 at 6:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.