0
$\begingroup$

I have around 240 files with lots of data, I have been using Import[] to import these data for further processing. But as my simulations are growing bigger, I am having lots of difficulty importing them, my computer is freezing and lagging (my computer has 6GB RAM). I was wondering if there is a better (less memory intensive) option to import these files?

I have been using the following loop command to import this data as strings and then converting them to numbers

Rawdata = Table[
     s = ToString[j];
     Import["C:\\Users\\Downloads\\Run 6 new z=45 sigma =0.845\\pinocchio." <>
          StringTake[s,2] <> "." <> StringDrop[s,2] <> ".example.catalog.out",
          "Data"],
     {j, 200000, 420000, 1000}];

Is there an analogous way to import the files without using so much memory?

Improve this question
$\endgroup$
19
  • $\begingroup$ Could you please format your code using linebreaks? It is very hard to read at the moment. Did you search the site for already existing solutions yet? $\endgroup$
    – Yves Klett
    Commented May 8, 2015 at 4:59
  • 1
    $\begingroup$ Look at OpenRead and ReadList $\endgroup$
    – Basheer Algohi
    Commented May 8, 2015 at 5:45
  • 1
    $\begingroup$ Plus, there's a missing quote in the code you provided. $\endgroup$
    – Sektor
    Commented May 8, 2015 at 6:41
  • 1
    $\begingroup$ You can use ReadList to read line-by-line or by blocks (use its third parameter). Erasing memory: just unset (=.) the variable you want to clear, or use ClearAll or Remove. $\endgroup$
    – Sjoerd C. de Vries
    Commented May 8, 2015 at 12:48
  • 1
    $\begingroup$ concerning memory: do you know about $HistoryLength=0? if you don't do that Mathematica will remember all output you ever generated in a session. concerning import: if changing the simulation, I'd consider another format for your files, text is just not a very good encoding for numeric data. Other than that, you might find this question and answers to it helpful. If sticking with text, you can read lines with ReadList and type String instead of BindaryReadList as in the answer... $\endgroup$
    – Albert Retey
    Commented May 8, 2015 at 23:34

1 Answer 1

Reset to default
2
$\begingroup$

If your data is or can be coerced into a rectangular array of machine Integer, Real, or Complex numbers (all the same type), then you can use ToPackedArray to reduce the amount of memory required, as well as make operations on the data faster. One might have hoped that Import would do this automatically, but it doesn't.

You can use ByteCount to see how much memory a variable is consuming.

Those operations, as well as PackedArrayQ and FromPackedArray require Needs["Developer`"] to load them.

Simple example:

Needs["Developer`"]
dat=ExportString[RandomReal[1,{1000,1000}],"Table"];
ImportString[dat,"Table"]//ByteCount

24208200

ImportString[dat,"Table"]//ToPackedArray//ByteCount

8000152
Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Does someone know how to put a backquote inside backquotes? I can't get just one backquote to show up for Needs["Developer`"] when inside backquotes. $\endgroup$
    – Mark Adler
    Commented May 8, 2015 at 17:00
  • $\begingroup$ use <code> .. </code> instead of back ticks.. $\endgroup$
    – george2079
    Commented May 8, 2015 at 20:12
  • $\begingroup$ @george2079 Thanks! $\endgroup$
    – Mark Adler
    Commented May 8, 2015 at 22:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.