3
$\begingroup$

We are working on a project that reads in a stream, extracts information from it, and then needs to stop the calculation when it reaches EndOfFile.

So for example, this works to grab the first ten records (or ten from wherever the stream's current position is):

Do[
 tempRecord = Read[wFile, Record, RecordSeparators -> {"WARC/1.0" }];
 tempRecordAssociation = recordToAssociation[tempRecord];
 Print[tempRecordAssociation["WARC-Target-URI"]], 10]

But if it bumps up against the EndOfFile, we get the error:

StringReplace::strse: "String or list of strings expected at position..

Our ultimate goal is to get something like this working, where we can read the entire stream and it stops when it hits EndOfFile.

While[Reap[True, 
   tempRecord = 
    Read[wFile, Record, RecordSeparators -> {"WARC/1.0"}];
   tempRecordAssociation = recordToAssociation[tempRecord];
   Sow[tempRecordAssociation["WARC-Target-URI"]];
   If[StreamPosition[wFile] == EndOfFile, Break[]]]];

But, as a relative newcomer in working with streams, we can't figure out how to get the Read to stop on EndOfFile and store the results up until then. We tried SameQ to no avail.

So in short, our question is: how can we get read to end when it hits EndOfFile?

Appendix

In case the sample data and functions are useful, I will provide them below. I think this may be more of a conceptual question, however. Sample data file can be found here as a Dropbox public download. I will probably remove this data file later, but it is this file unzipped with a txt extension added manually.

File is loaded:

sampleWARCFile = 
  "/Users/ianmilligan1/dropbox/git/warcbase-resources/Sample-Data/\
ARCHIVEIT-227-QUARTERLY-XUGECV-20091218231727-00039-crawling06.us.\
archive.org-8091.warc.txt";
wFile = OpenRead[sampleWARCFile]

And the function is:

recordToAssociation[rec_] :=
 Module[{recmod, fields, vals},
  recmod = StringReplace[rec, "\r\n\r\n" -> "\r\n\r\nCONTENT: "];
  fields = 
   StringTrim[StringExtract[StringSplit[recmod, "\r"], ":" -> 1]];
  vals = Map[
    StringTrim[
      StringReplace[#, 
       Shortest[StartOfString ~~ Except[":"] ..] ~~ ":" -> ""]] &, 
    StringSplit[recmod, "\r"]];
  Return[Association[
    DeleteCases[MapThread[Rule, {fields, vals}], 
     Rule["", ""] | Rule["CONTENT", ""]]]]]
$\endgroup$
  • 2
    $\begingroup$ just test if tmpRecord===EndOfFile and control the logic appropriately. aside if you are reading an entire file consider using Import instead $\endgroup$ – george2079 Aug 27 '16 at 13:48
  • $\begingroup$ Thanks @george2079 – some of our files are too large to load into memory, so we're hoping to run them like so. Hopefully that's not a horrid idea.. :) $\endgroup$ – programming_historian Aug 28 '16 at 16:42
1
$\begingroup$

I believe your While condition and the positioning of Reap is a bit off. Try the following

Reap[
 While[(tempRecord = Read[wFile, Record, RecordSeparators -> {"WARC/1.0"}]) =!= EndOfFile,
  tempRecordAssociation = recordToAssociation[tempRecord];
  Sow[tempRecordAssociation["WARC-Target-URI"]];
  ]
 ]

Hope this helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.