# ReadString and StreamPosition behaving strangely with custom terminators

I've stumbled on a strange behavior while trying to use custom terminators for ReadString.

When I use a simple String as the terminator, I get the following behavior

stream = StringToStream[" abc  def g\n  hi"]
Do[{
StreamPosition[stream],
StreamPosition[stream]
} // Print
, {6}]

(*
{0, "abc", 4}
{4, "def", 9}
{9, "g\n", 12}
{12, "hi", 16}
{16, EndOfFile, 16}
{16, EndOfFile, 16}
*)


Everything OK here. It looks as the default is to discard empy "reads". If I use, instead, a simple Alternative things behave very differently.

stream = StringToStream[" abc  def g\n  hi"]
Do[{
StreamPosition[stream],
ReadString[stream, StringExpression[" " | EndOfString]] // FullForm,
StreamPosition[stream]
}
, {9}]

(*
{0, "", 16}
{16, "abc", 16}
{16, "", 16}
{16, "def", 16}
{16, "g\n", 16}
{16, "", 16}
{16, "hi", 16}
{16, EndOfFile, 16}
{16, EndOfFile, 16}
*)


Not only empty records are now presented, but the StreamPosition mechanism becomes useless.

I suppose the string is being read in it's entirety on the second example, but why? There is no ambiguity in the separator, and no need for look-aheads if the parser matches a single " ".

And is there a reason for empty records to be returned by default only on the second example?

• Thanks for asking this. I was pondering the very same problem, the related question is mathematica.stackexchange.com/q/108814/26956 . I'd like to add, that this problem originates from a desire to have not only the functionality Read[strm, Word, 1] but also Read[strm, WhitespaceSequence, 1]. ReadString[strm, Except[WhitespaceCharacter] seemed like a good solution to me, however MMA does not like this kind of string pattern in ReadString. Mar 5 '16 at 18:24
• Note that ReadString[strm, StringExpression[Except[WhitespaceCharacter]] works, if you're willing to accept the idiosyncrasies I've pointed above! Mar 5 '16 at 19:10
• If only ReadString could also return the terminator... I think, part of the problem is constantly setting the stream position. Mar 5 '16 at 19:39
• Btw, your first block has a syntax error (unmatched brackets) and after a kernel restart on my machine does not return proper StreamPositions, that you show in the example output, but rather suffers from the same problem as the second example. Mar 6 '16 at 0:55

<<GeneralUtilities

ReadString internally calls a functions that reads the stream in in chunks of 10 000 characters. It also updates the (down)value of ProcessLinkPrivate$StreamCache[stream] on the fly. It contains the remainder of the chunk, that was not returned by ReadString. The magical value of 10 000 is contained in a variable called ProcessLinkPrivatebinaryReadBlock. By changing this number ProcessLinkPrivatebinaryReadBlock = 2  we get an interesting result: stream = StringToStream["abc def g\n hi"]; StreamPosition[stream] ReadString[stream, x : WhitespaceCharacter] StreamPosition[stream] (* 0 *) (* "abc" *) (* 4 *)  And then (this is more relevant to the original problem) ReadString[stream, x : Except[WhitespaceCharacter]] StreamPosition[stream] (* " " *) (* 6 *)  But now run this again: StreamPosition[stream] ReadString[stream, x : WhitespaceCharacter] StreamPosition[stream] (* 6 *) (* "ef" *) (* 10 *)  One would expect 9, but that's not the case. We are now reading in blocks of 2 characters at a time. The first block had "ef" and contained no Whitespace so another block was read (" g"). It was added to the cache, the leading space was dropped, but the StreamPosition got shifted to 10. ReadString pulls data from the cache and using SetStreamPosition is apparently highly unreliable. Unfortunately, it does not play nicely with patterns of the type x:WordBoundary (which is relevant to the original QA), but I'm sure, that with careful study of the internals, an implementation can be made, that will work correctly. This will eliminate the need to SetStreamPosition. • This means that the issue I noticed with results from the previous test bleeding over to later tests must be caused by ProcessLinkPrivate$StreamCache[stream] not getting cleared on Close[stream]` (which would not be a problem except that the "unique ID" for closed streams is reused for later streams). That sounds like a reportable bug. Mar 6 '16 at 21:28