Err... When I try to Import a .dat file by

dat = Import["D:\\TIEGCM\\solar_input\\omni_m_all_years.dat", 


dat = Import["D:\\TIEGCM\\solar_input\\omni_m_all_years.dat"];

mathematica run out of memory and crashed the kernel,I admitted that the dat file is a bit large ~38Mb, however, it shouldn't have been a problem considering my 2G RAM. And when I checked the progresses by Process Explorer, I found that the Private Bytes of one MathKernel.exe grows rapidly from 63Mb all the way to more than 700Mb when I evaluate the code dat = Import[...] listed above. How could this happen?Help!!!!!

the omni_m_all_years.dat file can be download under ftp://cdaweb.gsfc.nasa.gov/pub/data/omni/low_res_omni/ (in fact you can Import the dat file directly from website if your Mma has good connection with internet) and I use Mathematica9.0 under Windows7

  • 5
    $\begingroup$ Try using ReadList, instead. $\endgroup$
    – rcollyer
    Apr 7 '15 at 15:10
  • 4
    $\begingroup$ unfortunatly I'd consider this a "known problem". Many of the import functions do work but use an unreasonable amount of memory when the files get "large" (where something like 38MB by todays standards is far from large IMHO). My impression is that the default Import functionality is too general and tries to be too smart (that is auto-detect almost everything) and never has been decently optimized. The usual workaround, especially for text formats, is to use one of the lower level functions to write something specialiced as others have suggested and shown... $\endgroup$ Apr 7 '15 at 15:57
  • $\begingroup$ @Albert Retey I think Harry should upgrade is PC considering Wolframs' minimum recommended 2GB+ for Mathematica, support.wolfram.com/kb/12432 $\endgroup$
    – penguin77
    Apr 7 '15 at 16:42
  • $\begingroup$ @penguin77: meeting WRIs recommendations is a valid point, but I think OP has 2GB which at least almost meets those. My point is that any program which needs more than about 150MB to import a 38MB text file whose content will be much less than that in memory once imported has room for optimization. Data files of a few 100MB are very common for any kind of measurements these days, and these can't be loaded even when meeting WRIs recommendations. Unfortunately many potential users will consider Mathematica not fit for their purposes when they can't even import their data... $\endgroup$ Apr 7 '15 at 18:19
  • $\begingroup$ @Albert Retey Thx for your thoughts, I have updated my answer to indicate memory usage. $\endgroup$
    – penguin77
    Apr 7 '15 at 19:21

M9 under Windows 7, Intel Core i7 970. 24 GB RAM, Import without any problem.

Alternatively you can read the file as a stream.

str = OpenRead["z:\\temp\\omni_m_all_years.dat"]  
ReadList[str, Record]  // StringSplit

You may add //ToExpression to get strings converted to numbers

Example for 3 records (lines in text file):

 ReadList[str, Record, 3]  // StringSplit //ToExpression

{{1963, 2, 21, -3.2, 26.1, 999.9, 999.9, 999.9, 999.9, 9999., 999.9, 
  999.9, 999.9, 1.*10^7}, {1963, 2, 22, -3.2, 26.2, 999.9, 999.9, 
  999.9, 999.9, 9999., 999.9, 999.9, 999.9, 1.*10^7}, {1963, 2, 
  23, -3.2, 26.2, 999.9, 999.9, 999.9, 999.9, 9999., 999.9, 999.9, 
  999.9, 1.*10^7}}

Hint: a) You may close all other applications running including browser and b) increase virtual memory cross-check available resources using Piriform Speccy (https://www.piriform.com/speccy).

EDIT: Memory requirements using Import :

mmu = MemoryInUse[];
Import["z:\\temp\\omni_m_all_years.dat", "List"] // Timing
(MemoryInUse[] - mmu )/2^10 // N


Memory requirements processing stream with ReadList:

m = MemoryInUse[];
str = OpenRead["z:\\temp\\omni_m_all_years.dat"];
ReadList[str, Record] 
(MemoryInUse[] - m )/2^10 // N


Memory requirements processing stream with ReadList and splitting records into substrings and convert to expressions :

m = MemoryInUse[];
str = OpenRead["z:\\temp\\omni_m_all_years.dat"];
ReadList[str, Record]  // StringSplit // ToExpression
(MemoryInUse[] - m )/2^10 // N

(*163 403*)
  • $\begingroup$ I think that should be 1.24 GB not 1.24MB, which is much less than a fresh kernel needs on my machine? Anyway I think it is might not be very convincing that one needs a 24 GB machine to load a 38MB file with Mathematica in 38 sec. ... $\endgroup$ Apr 7 '15 at 16:02
  • $\begingroup$ @Albert Retey , I use 24 GB for film-making post-production, no need for Mathematica. However it reveals that the large file import issue is not related to Mathematica itself. $\endgroup$
    – penguin77
    Apr 7 '15 at 16:11
  • $\begingroup$ @Albert Retey, yes GB not MB, thx, I have corrected $\endgroup$
    – penguin77
    Apr 7 '15 at 16:14
  • $\begingroup$ I understand that you'd not need 24 GB to import that specific file :-). But why do you propose that the import issue is not related to Mathematica itself? I'd think Mathematica using more than 1GB to import a 38MB text file is an issue that is very much related to Mathematica... $\endgroup$ Apr 7 '15 at 18:27
  • 1
    $\begingroup$ @Albert Retey considering Wolframs' recommended minimum 2GB+ for Mathematica, support.wolfram.com/kb/12432 Harry is with 2GB at the edge of minimum recommended requirement. Your right that Mathematica memory consumption is an issue, however difficult to qualify it as a bug . I agree, I should rephrase my previous comment in this sens. $\endgroup$
    – penguin77
    Apr 7 '15 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.