3
$\begingroup$

I've stumbled on a strange behavior while trying to use custom terminators for ReadString.

When I use a simple String as the terminator, I get the following behavior

stream = StringToStream[" abc  def g\n  hi"]
Do[{
   StreamPosition[stream],
   ReadString[stream, StringExpression[" "] // FullForm,
   StreamPosition[stream]
   } // Print
 , {6}]

(*
{0, "abc", 4}
{4, "def", 9}
{9, "g\n", 12}
{12, "hi", 16}
{16, EndOfFile, 16}
{16, EndOfFile, 16}
*)

Everything OK here. It looks as the default is to discard empy "reads". If I use, instead, a simple Alternative things behave very differently.

stream = StringToStream[" abc  def g\n  hi"]
Do[{
    StreamPosition[stream],
    ReadString[stream, StringExpression[" " | EndOfString]] // FullForm,
    StreamPosition[stream]
    }
  , {9}]

(*
{0, "", 16}
{16, "abc", 16}
{16, "", 16}
{16, "def", 16}
{16, "g\n", 16}
{16, "", 16}
{16, "hi", 16}
{16, EndOfFile, 16}
{16, EndOfFile, 16}
*)

Not only empty records are now presented, but the StreamPosition mechanism becomes useless.

I suppose the string is being read in it's entirety on the second example, but why? There is no ambiguity in the separator, and no need for look-aheads if the parser matches a single " ".

And is there a reason for empty records to be returned by default only on the second example?

$\endgroup$
  • $\begingroup$ Thanks for asking this. I was pondering the very same problem, the related question is mathematica.stackexchange.com/q/108814/26956 . I'd like to add, that this problem originates from a desire to have not only the functionality Read[strm, Word, 1] but also Read[strm, WhitespaceSequence, 1]. ReadString[strm, Except[WhitespaceCharacter] seemed like a good solution to me, however MMA does not like this kind of string pattern in ReadString. $\endgroup$ – LLlAMnYP Mar 5 '16 at 18:24
  • $\begingroup$ Note that ReadString[strm, StringExpression[Except[WhitespaceCharacter]] works, if you're willing to accept the idiosyncrasies I've pointed above! $\endgroup$ – Aisamu Mar 5 '16 at 19:10
  • $\begingroup$ If only ReadString could also return the terminator... I think, part of the problem is constantly setting the stream position. $\endgroup$ – LLlAMnYP Mar 5 '16 at 19:39
  • $\begingroup$ Btw, your first block has a syntax error (unmatched brackets) and after a kernel restart on my machine does not return proper StreamPositions, that you show in the example output, but rather suffers from the same problem as the second example. $\endgroup$ – LLlAMnYP Mar 6 '16 at 0:55
3
$\begingroup$

This is for the moment an incomplete answer, but I gathered lots of insight by using

<<GeneralUtilities`
PrintDefinitions[ReadString]

ReadString internally calls a functions that reads the stream in in chunks of 10 000 characters. It also updates the (down)value of ProcessLink`Private`$StreamCache[stream] on the fly. It contains the remainder of the chunk, that was not returned by ReadString.

The magical value of 10 000 is contained in a variable called ProcessLink`Private`binaryReadBlock. By changing this number

ProcessLink`Private`binaryReadBlock = 2

we get an interesting result:

stream = StringToStream["abc  def g\n  hi"];
StreamPosition[stream]
ReadString[stream, x : WhitespaceCharacter]
StreamPosition[stream]
(* 0 *)
(* "abc" *)
(* 4 *)

And then (this is more relevant to the original problem)

ReadString[stream, x : Except[WhitespaceCharacter]]
StreamPosition[stream]
(* " " *)
(* 6 *)

But now run this again:

StreamPosition[stream]
ReadString[stream, x : WhitespaceCharacter]
StreamPosition[stream]
(* 6 *)
(* "ef" *)
(* 10 *)

One would expect 9, but that's not the case. We are now reading in blocks of 2 characters at a time. The first block had "ef" and contained no Whitespace so another block was read (" g"). It was added to the cache, the leading space was dropped, but the StreamPosition got shifted to 10. ReadString pulls data from the cache and using SetStreamPosition is apparently highly unreliable. Unfortunately, it does not play nicely with patterns of the type x:WordBoundary (which is relevant to the original QA), but I'm sure, that with careful study of the internals, an implementation can be made, that will work correctly. This will eliminate the need to SetStreamPosition.

$\endgroup$
  • $\begingroup$ This means that the issue I noticed with results from the previous test bleeding over to later tests must be caused by ProcessLink`Private`$StreamCache[stream] not getting cleared on Close[stream] (which would not be a problem except that the "unique ID" for closed streams is reused for later streams). That sounds like a reportable bug. $\endgroup$ – Mario Carneiro Mar 6 '16 at 21:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.