1
$\begingroup$

I'm trying to figure out how many rows/records my csv file has, which I have to stream because it's quite large.

Can someone tell me if my approach works and how I can get it to loop through the whole data. It works for the n rows I put in the code but since I don't know what number of the last row is I can't get it to work. Basically, I don't know how to finish the Do command properly where I put the ?????.

stream = OpenRead["C:\\Users\\Name\\Documents\\file.csv"];
list = {};
Do[
  record = Read[stream, Record];
  AppendTo[list, ToExpression @ First @ StringSplit[record, ","]], 
  {?????}];
entryCount = Length[list]
Improve this question
$\endgroup$
2
  • $\begingroup$ Read will return EndOfFile if you attempt to read past the end of the stream. $\endgroup$
    – Rohit Namjoshi
    Feb 26, 2019 at 1:04
  • $\begingroup$ This does not implement CSV (need to check delimiters because a new line might be in the data, or it might indicate the end of a row).. Take a look at this solution for now (mathematica.stackexchange.com/a/183652/38159), but I can tell you next version will make this much easier and use much less memory, so don't stick with these Read approaches past 11.3 $\endgroup$
    – GenericAccountName
    Feb 26, 2019 at 3:28

2 Answers 2

Reset to default
1
$\begingroup$

I have two main recommendations:

  1. Forget Do and use While to stop when Read returns EndOfFile.
  2. Write defensive code when dealing with the file system.

With those things in mind, the following code seems a reasonable solution to your problem.

readCSV[path_] :=
  Module[{strm, recd, data = {}},
    CheckAbort[
      If[(strm = OpenRead[path]) === $Failed, Return[$Failed]];
      While[(recd = Read[strm, Record]) =!= EndOfFile,
        AppendTo[data, ToExpression["{" <> recd <> "}"]]];
      Close[strm];
      data,
      Close[strm]]]

Test

path = FileNameJoin[{$HomeDirectory, "Desktop", "data.csv"}];
Export[path, Table[{i, i^2, i^3}, {i, 5}]];
readCSV[path]

{{1, 1, 1}, {2, 4, 8}, {3, 9, 27}, {4, 16, 64}, {5, 25, 125}}`
Improve this answer
$\endgroup$
2
  • $\begingroup$ OP says "which I have to stream because it's quite large". How does this solution help with that? The entire file is read. $\endgroup$
    – Rohit Namjoshi
    Feb 26, 2019 at 1:08
  • $\begingroup$ @RohitNamjoshi. It gives the OP a better code structure to work with for reading records in blocks. I didn't write the blocking part because the OP didn't give any information on how the handle the blocks. $\endgroup$
    – m_goldberg
    Feb 26, 2019 at 1:20
0
$\begingroup$

What's wrong with 

list = ReadList[stream, Records]
Improve this answer
$\endgroup$
1
  • $\begingroup$ ReadList::readf: Records is not a valid format specification. $\endgroup$
    – Rohit Namjoshi
    Feb 26, 2019 at 0:57

Not the answer you're looking for? Browse other questions tagged or ask your own question.