Internal function overload when calling ReadString on a fresh kernel

Question

Problem description:

I am capturing all FE messages to redirect them to my own window while blocking them from showing in the evaluation notebook using this method by rcollyer. It works great, but I have noticed a strange thing. If my code includes ReadString, then my custom error window is flooded with a lot of messages:

The function run has already been loaded with argument types {Integer,UTF8String,UTF8String,UTF8String,UTF8String}. The new function will be an overload for different types.

This happens only the first time with a fresh kernel (or after Quit). The second and all consecutive calls to ReadString with the same or other arguments work fine without any errors. There are no errors as well if I call ReadString with any arguments even one time on a fresh kernel prior to message capturing, which means it has something to do with the initial loading of built-in functions.

I have experimented with other functions (I/O such as Read, Import, and standard Map, Applyetc., and even WriteString), but none of them produces that error.

This is the simplified version of the code that generates the same behavior (using Print here for simplicity and retaining Messageto display all messages in the evaluation notebook for debugging -- not using it in the main code):

Internal`InheritedBlock[{Message, $InMsg = False}, Unprotect[Message];
 Message[msg_, vars___] /; (! $InMsg && msg =!= $Off[]) := 
  Block[{$InMsg = True, controlstring},
   controlstring = 
    First@ReleaseHold[
      msg /. Cases[HoldPattern@msg, 
        HoldPattern[MessageName[x_, _]] :> Messages@x]];
   Print[StringForm[controlstring, vars]];
   Message[msg, vars];
   ];
 Module[{}, ReadString["ew"]];
 ]

The message name for this error is LibraryFunction::overload, but the strange thing is that msg doesn't get that value, instead it is set to the control string already, the same we can find by calling Messages[LibraryFunction]. In this example code, I am still allowing for the messages to be displayed normally through Message, which shows all messages (including the message that a file cannot be opened) except for this one, so we see it only because of Print, but it is also why my custom window gets all those errors, as I am using the controlstring variable. Is there a hidden reason why Message doesn't print this error?

The questions are:

1) Why is ReadStringso special and what is that error?

2) Are there other functions like that? I can of course, call ReadStringwith a dummy parameter before I set up message capturing to make sure it is loaded, but what if there are other functions with a similar behavior? I don't want to see the internal messages that are normally not visible in the front end -- Message doesn't print it under normal circumstances.

3) What can be done to eliminate that error -- e.g. pre-loading some packages etc.? Other versions of the code that allows for message capturing (and also preventing it from displaying normally, so EvaluationDatawouldn't work here)?

Answer 1

(analysis current as of version 11.1)

1) Why is ReadString so special and what is that error?

In a fresh kernel, ReadString is auto-loaded from the ProcessLink` package. In that package, LibraryFunctionLoad is used to create 201 overloads for the external libProcessLink function run. Those overloads allow a potentially large variable number of command line arguments to be passed to an external process. On every call to LibraryFunctionLoad but the first, the warning message LibraryFunction::overload is issued. These are only warnings, and the code explicitly uses Quiet to suppress them.

All of this can be observed by inspecting the definition of the symbol runs in the file:

NotebookOpen @ FileNameJoin @
  { $InstallationDirectory, "SystemFiles", "Links", "ProcessLink", "ProcessLink.m"}

As rcollyer notes in the linked post, the code there will capture all messages, even those suppressed by Quiet or Off. The code in the present question ignores Off cases, but still captures quieted messages. It is such quieted messages that we are seeing here.

2) Are there other functions like that?

I am sure that there are many -- any functions that intentionally Quiet messages. At the very least, all of the other functions auto-loaded along with ReadString will exhibit the same behaviour:

(* in a fresh kernel *)
OwnValues[ReadString]

(* { HoldPattern[ReadString] :>
       Package`ActivateLoad[
         ReadString,

         (* these functions: *)
         {ProcessEnvironment, ProcessDirectory, WriteLine,
          ReadLine, ReadString, EndOfBuffer, $SystemShell, Processes,
          ProcessObject, ProcessStatus, ProcessInformation,
          ProcessConnection, KillProcess, RunProcess, StartProcess},

         "ProcessLink`",
         {Package`HiddenImport -> False, Path -> Automatic}
       ]
   } *)

3) What can be done to eliminate that error?

If our intention is to capture quieted messages, then the behaviour we see is correct. Otherwise, we can find a way to identify quieted messages.

We can use the undocumented function Internal`QuietStatus to check whether a message has been intentionally quieted:

Quiet[Internal`QuietStatus[], LibraryFunction::overload]

(* { Global -> Unquiet,
     Off -> {HoldForm[LibraryFunction::overload]},
     On -> {},
     Stack -> {{2443, {HoldForm[LibraryFunction::overload]}, {}}},
     MessageList -> {},
     Check -> None
   } *)

We could incorporate this information into a further check in our example code. For instance:

Internal`InheritedBlock[{Message, $InMsg = False}
, Unprotect[Message]
; Message[msg_, vars___] /; !$InMsg
                         && msg =!= $Off[]
                         && FreeQ[Off /. Internal`QuietStatus[], Unevaluated[msg]]
                         := 
    Block[{$InMsg = True, controlString}
    , controlString =  msg /. m:HoldPattern[MessageName[f_, _]] :> (m /. Messages[f])
    ; Print[StringForm[controlString, vars]]
    ; Message[msg, vars]
    ]
; ReadString
]

Naturally, the use of undocumented functionality comes with no support guarantees.

Bonus Question: Why is msg a string instead of a MessageName?

The value of msg at the time the function is invoked is in fact a MessageName, but as soon as it is evaluated the result is a string:

LibraryFunction::overload

(* "The function `1` has already been loaded with argument types `2`. \
    The new function will be an overload for different types." *)

This follows the convention for defining messages in high-level code. On the other hand, built-in kernel functions do not always have user-visible definitions and this includes message definitions:

Map::nonopt

(* Map::nonopt *)

Many messages, even ones for built-in kernel symbols, are not available until the message has been evaluated (to trigger auto-loading):

(* from a fresh kernel *)

Messages[Map]
(* {} *)

Map;
Messages[Map]
(* {} *)

Map::nonopt;
Messages[Map]
(* {{HoldPattern[Map::nonopt]:> "..."} *)

Fortunately, these corner cases can be addressed by a relatively simple expression:

msg /. m:HoldPattern[MessageName[f_, _]] :> (m /. Messages[f])

The last code block in the preceding question uses this form. That code also uses Unevaluated in the FreeQ expression since Internal`QuietStatus keeps messages in held form. Message has the HoldFirst attribute, so additional holding is unnecessary there.

Caution

Patching internal low-level symbols like Message can have very subtle and far-reaching effects. The issues discussed here are only the tip of the iceberg. For example, the treatment of $Off would need to be generalized to handle cases when it takes the form $Off["..."], or for when it is lazy-loaded.

Many internal system components fail in surprising ways if built-in functionality is disturbed in the slightest. So techniques like these should not be used on a routine basis, but rather in specialized circumstances (e.g. debugging).