mathematica use too much memory to Import large dat file

Question

Err... When I try to Import a .dat file by

dat = Import["D:\\TIEGCM\\solar_input\\omni_m_all_years.dat", 
"Table"];

or

dat = Import["D:\\TIEGCM\\solar_input\\omni_m_all_years.dat"];

mathematica run out of memory and crashed the kernel,I admitted that the dat file is a bit large ~38Mb, however, it shouldn't have been a problem considering my 2G RAM. And when I checked the progresses by Process Explorer, I found that the Private Bytes of one MathKernel.exe grows rapidly from 63Mb all the way to more than 700Mb when I evaluate the code dat = Import[...] listed above. How could this happen?Help!!!!!

the omni_m_all_years.dat file can be download under ftp://cdaweb.gsfc.nasa.gov/pub/data/omni/low_res_omni/ (in fact you can Import the dat file directly from website if your Mma has good connection with internet) and I use Mathematica9.0 under Windows7

Answer 1

M9 under Windows 7, Intel Core i7 970. 24 GB RAM, Import without any problem.

Alternatively you can read the file as a stream.

str = OpenRead["z:\\temp\\omni_m_all_years.dat"]  
ReadList[str, Record]  // StringSplit
Close[str]

You may add //ToExpression to get strings converted to numbers

Example for 3 records (lines in text file):

 ReadList[str, Record, 3]  // StringSplit //ToExpression

{{1963, 2, 21, -3.2, 26.1, 999.9, 999.9, 999.9, 999.9, 9999., 999.9, 
  999.9, 999.9, 1.*10^7}, {1963, 2, 22, -3.2, 26.2, 999.9, 999.9, 
  999.9, 999.9, 9999., 999.9, 999.9, 999.9, 1.*10^7}, {1963, 2, 
  23, -3.2, 26.2, 999.9, 999.9, 999.9, 999.9, 9999., 999.9, 999.9, 
  999.9, 1.*10^7}}

Hint: a) You may close all other applications running including browser and b) increase virtual memory cross-check available resources using Piriform Speccy (https://www.piriform.com/speccy).

EDIT: Memory requirements using Import :

mmu = MemoryInUse[];
Import["z:\\temp\\omni_m_all_years.dat", "List"] // Timing
(MemoryInUse[] - mmu )/2^10 // N

(*58075.6*)

Memory requirements processing stream with ReadList:

m = MemoryInUse[];
str = OpenRead["z:\\temp\\omni_m_all_years.dat"];
ReadList[str, Record] 
(MemoryInUse[] - m )/2^10 // N

(*59178.1*)

Memory requirements processing stream with ReadList and splitting records into substrings and convert to expressions :

m = MemoryInUse[];
str = OpenRead["z:\\temp\\omni_m_all_years.dat"];
ReadList[str, Record]  // StringSplit // ToExpression
(MemoryInUse[] - m )/2^10 // N

(*163 403*)