# Stream CSV files

Ram limitations make import impractical for large CSV files. Is it possible to Stream CSV correctly?

Read[str, "CSV"] #=> Read::readn: "Invalid real number found when reading from \!$$\"EU.csv\"$$."
Read@str #=> Read::readt: "Invalid input found when reading \!$$\"lTid,cDealable,CurrencyPair,RateDateTime,RateBid,RateAsk\\\\n\"$$ from \!$$\"EU.csv\"$$"
Close@str

Here's the CSV file. Though I think it's properly formatted.

-
You sure it's RAM limitations? Or it's not the same file as the one you post. I have no problem at all importing your file, as expected given it's less than a MB in size… –  F'x Apr 3 '12 at 22:29
Running on very light hardware :P. But that's a week of forex data that should be 4.9MB expanded. A lot of the CSVs are 4-10x that, so 20-50MB expanded. –  Mr. Demetrius Michael Apr 3 '12 at 22:34
@Mr.DemetriusMichael please add the CSV file to some permanent place like your question, it is missing. -1 until CSV back. –  hhh Nov 11 '14 at 3:50

Here is a function which may help:

With[{str = ReadList[stream, "String", n]},
ImportString[StringJoin[Riffle[str, "\n"]], "Table"] /; str =!= {}];
readRows[__] := $Failed; I tested on your file and it works all right (it may make sense to read rows in batches, this is much faster): n=0; str = OpenRead["C:\\Temp\\EUR_USD_Week1.csv"]; While[readRows[str, 1000] =!=$Failed, n++];
Close[str];
n

(*  82   *)

By the way, speaking of practicality of Import - I agree, but do read this answer - it is based on the same idea as above code, and makes importing whole files quite practical IMO. In particular, using the readTable function from there, I get your file read in its entirety under 3 seconds on my machine: