Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Ram limitations make import impractical for large CSV files. Is it possible to Stream CSV correctly?

str = OpenRead["EU.csv"]
Read[str, "CSV"] #=> Read::readn: "Invalid real number found when reading from \!\(\"EU.csv\"\)."
Read@str #=> Read::readt: "Invalid input found when reading \!\(\"lTid,cDealable,CurrencyPair,RateDateTime,RateBid,RateAsk\\\\n\"\) from \!\(\"EU.csv\"\)"

Here's the CSV file. Though I think it's properly formatted.

share|improve this question
You sure it's RAM limitations? Or it's not the same file as the one you post. I have no problem at all importing your file, as expected given it's less than a MB in size… – F'x Apr 3 '12 at 22:29
Running on very light hardware :P. But that's a week of forex data that should be 4.9MB expanded. A lot of the CSVs are 4-10x that, so 20-50MB expanded. – Mr. Demetrius Michael Apr 3 '12 at 22:34
@Mr.DemetriusMichael please add the CSV file to some permanent place like your question, it is missing. -1 until CSV back. – hhh Nov 11 '14 at 3:50
up vote 10 down vote accepted

Here is a function which may help:

readRows[stream_, n_] :=
   With[{str = ReadList[stream, "String", n]},
      ImportString[StringJoin[Riffle[str, "\n"]], "Table"] /; str =!= {}];
readRows[__] := $Failed;

I tested on your file and it works all right (it may make sense to read rows in batches, this is much faster):

str = OpenRead["C:\\Temp\\EUR_USD_Week1.csv"];
While[readRows[str, 1000] =!= $Failed, n++];

(*  82   *)

By the way, speaking of practicality of Import - I agree, but do read this answer - it is based on the same idea as above code, and makes importing whole files quite practical IMO. In particular, using the readTable function from there, I get your file read in its entirety under 3 seconds on my machine:

In[64]:= readTable["C:\\Temp\\EUR_USD_Week1.csv",2000]//Short//AbsoluteTiming
Out[64]= {2.4775391,{{lTid,cDealable,CurrencyPair,RateDateTime,RateBid,RateAsk},
  <<81172>>,{1385715072,D\[Ellipsis] SD,2011-01-07,\[Ellipsis] }}}

with a very decent memory usage:


(* 71652808 *)

meaning 50 Mb of net usage (when you subtract the startup memory usage) - for this particular example. You can tweak the second parameter to trade run-time for memory efficiency. See the linked post for more details.

share|improve this answer
beautiful, thanks. Not sure why Mathematica doesn't improve it's memory usage. – Mr. Demetrius Michael Apr 5 '12 at 21:06
@Mr.DemetriusMichael I think it will, in time. The current Import-Export framework is fairly new, quite natural that it has rough corners. Thanks for the accept. – Leonid Shifrin Apr 5 '12 at 21:21

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.