2
$\begingroup$

We are working on a project that reads in a stream, extracts information from it, and then needs to stop the calculation when it reaches EndOfFile.

So for example, this works to grab the first ten records (or ten from wherever the stream's current position is):

Do[
 tempRecord = Read[wFile, Record, RecordSeparators -> {"WARC/1.0" }];
 tempRecordAssociation = recordToAssociation[tempRecord];
 Print[tempRecordAssociation["WARC-Target-URI"]], 10]

But if it bumps up against the EndOfFile, we get the error:

StringReplace::strse: "String or list of strings expected at position..

Our ultimate goal is to get something like this working, where we can read the entire stream and it stops when it hits EndOfFile.

While[Reap[True, 
   tempRecord = 
    Read[wFile, Record, RecordSeparators -> {"WARC/1.0"}];
   tempRecordAssociation = recordToAssociation[tempRecord];
   Sow[tempRecordAssociation["WARC-Target-URI"]];
   If[StreamPosition[wFile] == EndOfFile, Break[]]]];

But, as a relative newcomer in working with streams, we can't figure out how to get the Read to stop on EndOfFile and store the results up until then. We tried SameQ to no avail.

So in short, our question is: how can we get read to end when it hits EndOfFile?

Appendix

In case the sample data and functions are useful, I will provide them below. I think this may be more of a conceptual question, however. Sample data file can be found here as a Dropbox public download. I will probably remove this data file later, but it is this file unzipped with a txt extension added manually.

File is loaded:

sampleWARCFile = 
  "/Users/ianmilligan1/dropbox/git/warcbase-resources/Sample-Data/\
ARCHIVEIT-227-QUARTERLY-XUGECV-20091218231727-00039-crawling06.us.\
archive.org-8091.warc.txt";
wFile = OpenRead[sampleWARCFile]

And the function is:

recordToAssociation[rec_] :=
 Module[{recmod, fields, vals},
  recmod = StringReplace[rec, "\r\n\r\n" -> "\r\n\r\nCONTENT: "];
  fields = 
   StringTrim[StringExtract[StringSplit[recmod, "\r"], ":" -> 1]];
  vals = Map[
    StringTrim[
      StringReplace[#, 
       Shortest[StartOfString ~~ Except[":"] ..] ~~ ":" -> ""]] &, 
    StringSplit[recmod, "\r"]];
  Return[Association[
    DeleteCases[MapThread[Rule, {fields, vals}], 
     Rule["", ""] | Rule["CONTENT", ""]]]]]
Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ just test if tmpRecord===EndOfFile and control the logic appropriately. aside if you are reading an entire file consider using Import instead $\endgroup$
    – george2079
    Aug 27, 2016 at 13:48
  • $\begingroup$ Thanks @george2079 – some of our files are too large to load into memory, so we're hoping to run them like so. Hopefully that's not a horrid idea.. :) $\endgroup$
    – canadian_scholar
    Aug 28, 2016 at 16:42

1 Answer 1

Reset to default
1
$\begingroup$

I believe your While condition and the positioning of Reap is a bit off. Try the following

Reap[
 While[(tempRecord = Read[wFile, Record, RecordSeparators -> {"WARC/1.0"}]) =!= EndOfFile,
  tempRecordAssociation = recordToAssociation[tempRecord];
  Sow[tempRecordAssociation["WARC-Target-URI"]];
  ]
 ]

Hope this helps.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.