3
$\begingroup$

I want to know how many records a file (large data) contains... with streaming...like this:

str = OpenRead[
  "https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2015-08.csv"
]

Count[Read[str, {Word, Number}]]

but nothing happens...

$\endgroup$
  • $\begingroup$ Maybe use Length instead of Count? Or Count[bla, _]? Still, Read[str, {Word, Number}] is not what you think it is... $\endgroup$ – Henrik Schumacher Mar 1 '18 at 10:19
  • $\begingroup$ @HenrikSchumachert thank you... Length still doesn't work though $\endgroup$ – Cécile Meier Mar 1 '18 at 10:27
2
$\begingroup$

Something like

str = OpenRead["https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2015-08.csv"]
records = ReadList[str, {Record}];
Close[str];
Length[records]

would do it but that would imply that the whole file gets downloaded...

Alternatively, you can use

str = OpenRead["https://s3.amazonaws.com/nyc-tlc/trip+data/yellow_tripdata_2015-08.csv"]
c = 0;
Dynamic[c]
record = "";
While[record =!= EndOfFile,
  c++;
  record = Read[str, {Record}];
  ];
Close[str]

This also loads the whole file, but it keeps only one record in memory. Better use that only for files on your local drive.

A recursive variant could look like this (first an example data set)

Export["a.csv", StringSplit[ExampleData[{"Text", "FaustI"}]]];

countRecords[url_String] := Block[{f, $IterationLimit = \[Infinity]},

  f[str_, EndOfFile, c_] := c;
  f[str_, record_, c_] := f[str, Read[str, {Record}], c + 1];

  With[{str = OpenRead[url]},
   Internal`WithLocalSettings[
    Null
    ,
    f[str, Read[str, {Record}], 0]
    ,
    Close[str];
    ]
   ]
  ]

I use Internal`WithLocalSettings in order to have the stream closed even if the user aborts the execution.

Here a usage example:

countRecords["a.csv"]

30620

| improve this answer | |
$\endgroup$
  • $\begingroup$ Is there a functional version of this, eg avoiding the internal increment c++, that looks a lot like the 1970s. $\endgroup$ – alancalvitti Mar 1 '18 at 17:07
  • $\begingroup$ @alancalvitti Feel free to wrap a function around it yourself. $\endgroup$ – Henrik Schumacher Mar 1 '18 at 18:15
  • $\begingroup$ nope, I mean a functional alternative to the procedural method, with no iterators, only Map, Apply, recursion, etc. $\endgroup$ – alancalvitti Mar 1 '18 at 18:19
  • $\begingroup$ @alancalvitti Recursion will likely hit the recursion limit and there is no list to Map on; actually, one does not want to have a list in memory here since the data in the file is huge. The point of handling streams lies within the fact that one wants to operate only on the current part of the stream. $\endgroup$ – Henrik Schumacher Mar 1 '18 at 18:25
  • 1
    $\begingroup$ @alancalvitti It's just ridiculous to try to avoid any plus operations in the code. Recall: We have to count here something. $\endgroup$ – Henrik Schumacher Mar 3 '18 at 9:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.