4
$\begingroup$

Err... When I try to Import a .dat file by

dat = Import["D:\\TIEGCM\\solar_input\\omni_m_all_years.dat", 
"Table"];

or

dat = Import["D:\\TIEGCM\\solar_input\\omni_m_all_years.dat"];

mathematica run out of memory and crashed the kernel,I admitted that the dat file is a bit large ~38Mb, however, it shouldn't have been a problem considering my 2G RAM. And when I checked the progresses by Process Explorer, I found that the Private Bytes of one MathKernel.exe grows rapidly from 63Mb all the way to more than 700Mb when I evaluate the code dat = Import[...] listed above. How could this happen?Help!!!!!

the omni_m_all_years.dat file can be download under ftp://cdaweb.gsfc.nasa.gov/pub/data/omni/low_res_omni/ (in fact you can Import the dat file directly from website if your Mma has good connection with internet) and I use Mathematica9.0 under Windows7

$\endgroup$
6
  • 5
    $\begingroup$ Try using ReadList, instead. $\endgroup$
    – rcollyer
    Apr 7, 2015 at 15:10
  • 4
    $\begingroup$ unfortunatly I'd consider this a "known problem". Many of the import functions do work but use an unreasonable amount of memory when the files get "large" (where something like 38MB by todays standards is far from large IMHO). My impression is that the default Import functionality is too general and tries to be too smart (that is auto-detect almost everything) and never has been decently optimized. The usual workaround, especially for text formats, is to use one of the lower level functions to write something specialiced as others have suggested and shown... $\endgroup$ Apr 7, 2015 at 15:57
  • $\begingroup$ @Albert Retey I think Harry should upgrade is PC considering Wolframs' minimum recommended 2GB+ for Mathematica, support.wolfram.com/kb/12432 $\endgroup$
    – penguin77
    Apr 7, 2015 at 16:42
  • $\begingroup$ @penguin77: meeting WRIs recommendations is a valid point, but I think OP has 2GB which at least almost meets those. My point is that any program which needs more than about 150MB to import a 38MB text file whose content will be much less than that in memory once imported has room for optimization. Data files of a few 100MB are very common for any kind of measurements these days, and these can't be loaded even when meeting WRIs recommendations. Unfortunately many potential users will consider Mathematica not fit for their purposes when they can't even import their data... $\endgroup$ Apr 7, 2015 at 18:19
  • $\begingroup$ @Albert Retey Thx for your thoughts, I have updated my answer to indicate memory usage. $\endgroup$
    – penguin77
    Apr 7, 2015 at 19:21

1 Answer 1

3
$\begingroup$

M9 under Windows 7, Intel Core i7 970. 24 GB RAM, Import without any problem.

Alternatively you can read the file as a stream.

str = OpenRead["z:\\temp\\omni_m_all_years.dat"]  
ReadList[str, Record]  // StringSplit
Close[str]

You may add //ToExpression to get strings converted to numbers

Example for 3 records (lines in text file):

 ReadList[str, Record, 3]  // StringSplit //ToExpression

{{1963, 2, 21, -3.2, 26.1, 999.9, 999.9, 999.9, 999.9, 9999., 999.9, 
  999.9, 999.9, 1.*10^7}, {1963, 2, 22, -3.2, 26.2, 999.9, 999.9, 
  999.9, 999.9, 9999., 999.9, 999.9, 999.9, 1.*10^7}, {1963, 2, 
  23, -3.2, 26.2, 999.9, 999.9, 999.9, 999.9, 9999., 999.9, 999.9, 
  999.9, 1.*10^7}}

Hint: a) You may close all other applications running including browser and b) increase virtual memory cross-check available resources using Piriform Speccy (https://www.piriform.com/speccy).

EDIT: Memory requirements using Import :

mmu = MemoryInUse[];
Import["z:\\temp\\omni_m_all_years.dat", "List"] // Timing
(MemoryInUse[] - mmu )/2^10 // N

(*58075.6*)

Memory requirements processing stream with ReadList:

m = MemoryInUse[];
str = OpenRead["z:\\temp\\omni_m_all_years.dat"];
ReadList[str, Record] 
(MemoryInUse[] - m )/2^10 // N

(*59178.1*)

Memory requirements processing stream with ReadList and splitting records into substrings and convert to expressions :

m = MemoryInUse[];
str = OpenRead["z:\\temp\\omni_m_all_years.dat"];
ReadList[str, Record]  // StringSplit // ToExpression
(MemoryInUse[] - m )/2^10 // N

(*163 403*)
$\endgroup$
8
  • $\begingroup$ I think that should be 1.24 GB not 1.24MB, which is much less than a fresh kernel needs on my machine? Anyway I think it is might not be very convincing that one needs a 24 GB machine to load a 38MB file with Mathematica in 38 sec. ... $\endgroup$ Apr 7, 2015 at 16:02
  • $\begingroup$ @Albert Retey , I use 24 GB for film-making post-production, no need for Mathematica. However it reveals that the large file import issue is not related to Mathematica itself. $\endgroup$
    – penguin77
    Apr 7, 2015 at 16:11
  • $\begingroup$ @Albert Retey, yes GB not MB, thx, I have corrected $\endgroup$
    – penguin77
    Apr 7, 2015 at 16:14
  • $\begingroup$ I understand that you'd not need 24 GB to import that specific file :-). But why do you propose that the import issue is not related to Mathematica itself? I'd think Mathematica using more than 1GB to import a 38MB text file is an issue that is very much related to Mathematica... $\endgroup$ Apr 7, 2015 at 18:27
  • 1
    $\begingroup$ @Albert Retey considering Wolframs' recommended minimum 2GB+ for Mathematica, support.wolfram.com/kb/12432 Harry is with 2GB at the edge of minimum recommended requirement. Your right that Mathematica memory consumption is an issue, however difficult to qualify it as a bug . I agree, I should rephrase my previous comment in this sens. $\endgroup$
    – penguin77
    Apr 7, 2015 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.