# Stream CSV files

Ram limitations make import impractical for large CSV files. Is it possible to Stream CSV correctly?

str = OpenRead["EU.csv"]
Read[str, "CSV"] #=> Read::readn: "Invalid real number found when reading from \!$$\"EU.csv\"$$."
Read@str #=> Read::readt: "Invalid input found when reading \!$$\"lTid,cDealable,CurrencyPair,RateDateTime,RateBid,RateAsk\\\\n\"$$ from \!$$\"EU.csv\"$$"
Close@str


Here's the CSV file. Though I think it's properly formatted.

-
You sure it's RAM limitations? Or it's not the same file as the one you post. I have no problem at all importing your file, as expected given it's less than a MB in size… –  F'x Apr 3 '12 at 22:29
Running on very light hardware :P. But that's a week of forex data that should be 4.9MB expanded. A lot of the CSVs are 4-10x that, so 20-50MB expanded. –  Mr. Demetrius Michael Apr 3 '12 at 22:34
@Mr.DemetriusMichael please add the CSV file to some permanent place like your question, it is missing. -1 until CSV back. –  hhh Nov 11 '14 at 3:50

Here is a function which may help:

Clear[readRows];
ImportString[StringJoin[Riffle[str, "\n"]], "Table"] /; str =!= {}];
readRows[__] := $Failed;  I tested on your file and it works all right (it may make sense to read rows in batches, this is much faster): n=0; str = OpenRead["C:\\Temp\\EUR_USD_Week1.csv"]; While[readRows[str, 1000] =!=$Failed, n++];
Close[str];
n

(*  82   *)


By the way, speaking of practicality of Import - I agree, but do read this answer - it is based on the same idea as above code, and makes importing whole files quite practical IMO. In particular, using the readTable function from there, I get your file read in its entirety under 3 seconds on my machine:

In[64]:= readTable["C:\\Temp\\EUR_USD_Week1.csv",2000]//Short//AbsoluteTiming
<<81172>>,{1385715072,D\[Ellipsis] SD,2011-01-07,\[Ellipsis] }}}


with a very decent memory usage:

MaxMemoryUsed[]

(* 71652808 *)


meaning 50 Mb of net usage (when you subtract the startup memory usage) - for this particular example. You can tweak the second parameter to trade run-time for memory efficiency. See the linked post for more details.

-
beautiful, thanks. Not sure why Mathematica doesn't improve it's memory usage. –  Mr. Demetrius Michael Apr 5 '12 at 21:06
@Mr.DemetriusMichael I think it will, in time. The current Import-Export framework is fairly new, quite natural that it has rough corners. Thanks for the accept. –  Leonid Shifrin Apr 5 '12 at 21:21