`ReadString` and `StreamPosition` behaving strangely with custom terminators

Question

I've stumbled on a strange behavior while trying to use custom terminators for ReadString.

When I use a simple String as the terminator, I get the following behavior

stream = StringToStream[" abc  def g\n  hi"]
Do[{
   StreamPosition[stream],
   ReadString[stream, StringExpression[" "] // FullForm,
   StreamPosition[stream]
   } // Print
 , {6}]

(*
{0, "abc", 4}
{4, "def", 9}
{9, "g\n", 12}
{12, "hi", 16}
{16, EndOfFile, 16}
{16, EndOfFile, 16}
*)

Everything OK here. It looks as the default is to discard empy "reads". If I use, instead, a simple Alternative things behave very differently.

stream = StringToStream[" abc  def g\n  hi"]
Do[{
    StreamPosition[stream],
    ReadString[stream, StringExpression[" " | EndOfString]] // FullForm,
    StreamPosition[stream]
    }
  , {9}]

(*
{0, "", 16}
{16, "abc", 16}
{16, "", 16}
{16, "def", 16}
{16, "g\n", 16}
{16, "", 16}
{16, "hi", 16}
{16, EndOfFile, 16}
{16, EndOfFile, 16}
*)

Not only empty records are now presented, but the StreamPosition mechanism becomes useless.

I suppose the string is being read in it's entirety on the second example, but why? There is no ambiguity in the separator, and no need for look-aheads if the parser matches a single " ".

And is there a reason for empty records to be returned by default only on the second example?

Answer 1

This is for the moment an incomplete answer, but I gathered lots of insight by using

<<GeneralUtilities`
PrintDefinitions[ReadString]

ReadString internally calls a functions that reads the stream in in chunks of 10 000 characters. It also updates the (down)value of ProcessLink`Private`$StreamCache[stream] on the fly. It contains the remainder of the chunk, that was not returned by ReadString.

The magical value of 10 000 is contained in a variable called ProcessLink`Private`binaryReadBlock. By changing this number

ProcessLink`Private`binaryReadBlock = 2

we get an interesting result:

stream = StringToStream["abc  def g\n  hi"];
StreamPosition[stream]
ReadString[stream, x : WhitespaceCharacter]
StreamPosition[stream]
(* 0 *)
(* "abc" *)
(* 4 *)

And then (this is more relevant to the original problem)

ReadString[stream, x : Except[WhitespaceCharacter]]
StreamPosition[stream]
(* " " *)
(* 6 *)

But now run this again:

StreamPosition[stream]
ReadString[stream, x : WhitespaceCharacter]
StreamPosition[stream]
(* 6 *)
(* "ef" *)
(* 10 *)

One would expect 9, but that's not the case. We are now reading in blocks of 2 characters at a time. The first block had "ef" and contained no Whitespace so another block was read (" g"). It was added to the cache, the leading space was dropped, but the StreamPosition got shifted to 10. ReadString pulls data from the cache and using SetStreamPosition is apparently highly unreliable. Unfortunately, it does not play nicely with patterns of the type x:WordBoundary (which is relevant to the original QA), but I'm sure, that with careful study of the internals, an implementation can be made, that will work correctly. This will eliminate the need to SetStreamPosition.