We are working on a project that reads in a stream, extracts information from it, and then needs to stop the calculation when it reaches EndOfFile
.
So for example, this works to grab the first ten records (or ten from wherever the stream's current position is):
Do[
tempRecord = Read[wFile, Record, RecordSeparators -> {"WARC/1.0" }];
tempRecordAssociation = recordToAssociation[tempRecord];
Print[tempRecordAssociation["WARC-Target-URI"]], 10]
But if it bumps up against the EndOfFile
, we get the error:
StringReplace::strse: "String or list of strings expected at position..
Our ultimate goal is to get something like this working, where we can read the entire stream and it stops when it hits EndOfFile.
While[Reap[True,
tempRecord =
Read[wFile, Record, RecordSeparators -> {"WARC/1.0"}];
tempRecordAssociation = recordToAssociation[tempRecord];
Sow[tempRecordAssociation["WARC-Target-URI"]];
If[StreamPosition[wFile] == EndOfFile, Break[]]]];
But, as a relative newcomer in working with streams, we can't figure out how to get the Read to stop on EndOfFile and store the results up until then. We tried SameQ
to no avail.
So in short, our question is: how can we get read to end when it hits EndOfFile?
Appendix
In case the sample data and functions are useful, I will provide them below. I think this may be more of a conceptual question, however. Sample data file can be found here as a Dropbox public download. I will probably remove this data file later, but it is this file unzipped with a txt extension added manually.
File is loaded:
sampleWARCFile =
"/Users/ianmilligan1/dropbox/git/warcbase-resources/Sample-Data/\
ARCHIVEIT-227-QUARTERLY-XUGECV-20091218231727-00039-crawling06.us.\
archive.org-8091.warc.txt";
wFile = OpenRead[sampleWARCFile]
And the function is:
recordToAssociation[rec_] :=
Module[{recmod, fields, vals},
recmod = StringReplace[rec, "\r\n\r\n" -> "\r\n\r\nCONTENT: "];
fields =
StringTrim[StringExtract[StringSplit[recmod, "\r"], ":" -> 1]];
vals = Map[
StringTrim[
StringReplace[#,
Shortest[StartOfString ~~ Except[":"] ..] ~~ ":" -> ""]] &,
StringSplit[recmod, "\r"]];
Return[Association[
DeleteCases[MapThread[Rule, {fields, vals}],
Rule["", ""] | Rule["CONTENT", ""]]]]]
tmpRecord===EndOfFile
and control the logic appropriately. aside if you are reading an entire file consider usingImport
instead $\endgroup$