How does Mathematica wrap code for front-end needs? Why does this wrapping affect the evaluation?

Question

Mathematica wraps any expression with some boxing code before showing the evaluation result in the front-end. Unfortunately, the wrapping algorithm and its position in the Mathematica end-to-end flow are not described in the official documentation. Surprisingly, this stuff can affect the evaluation of the expressions. Please, see an UpValues-based approach to problem of removing head X by head Y:

UpValues[Y] = {};
UpValues[X] =
   {HoldPattern[(head: __)[a___, X[b__], c___]] :> Hold[head[a, Y[b], c]]};

1 + X[432] + A[2] (* ==> Hold[1 + A[2] + Y[432] *)
X[X[545]] (* ==> Hold[Hold[Y[Y[545]]]] *)

Hold/HoldPattern were used in the transformation pattern to keep expression unevaluated but transformed by UpValues[X]. So it works. Now I want this transformation rule to work only if pattern head is not X:

UpValues[Y] = {};
UpValues[X] =
   {HoldPattern[(head: Except[X])[a___, X[b__], c___]] :> Hold[head[a, Y[b], c]]};

1 + X[432] + A[2] (* ==> Hold[1 + A[2] + Y[432] *)
X[X[545]]
(* ==> 
    Hold[
        If[False,
            Y[X[545]],
            With[{OutputSizeLimit`Dump`boxes$ =
                    Block[{$RecursionLimit = Typeset`$RecursionLimit},
                        MakeBoxes[X[X[545]], StandardForm]
                    ]},

                OutputSizeLimit`Dump`loadSizeCountRules[];
                If[TrueQ[BoxForm`SizeCount[OutputSizeLimit`Dump`boxes$, 1048576]],
                    OutputSizeLimit`Dump`boxes$,
                    OutputSizeLimit`Dump`encapsulateOutput[X[X[545]], $Line, $SessionID, 5]
                ]
           ],
           X[X[545]]
       ]
   ] *)

It seems, that pattern object Except[X] captures wrapping code that Mathematica uses to adopt any expression for the front-end needs. As for me it is a surprise because the case using general pattern object __ works well. So, my claim is that code wrapping used for front-end needs can affect the evaluation process in the Mathematica core.

Resulting,

1. Where can I read about Mathematica's end-to-end in details? I mean something like "The TeXbook" by D. Knuth for TeX.
2. Is the described behaviour a bug? Probably, it corresponds to features of implicit end-to-end flow.
3. Probably, the code above is not the only case. What technique can help me to avoid such behaviour?

Answer 1

This is not a full answer to all of your inquiries but it might help you along the way of understanding what's going on.

The front end wraps your code in a check to see how long it is, and if it's very long displays a shorter version. Try Range@1000000 for instance. which results in a box telling you that a very large output was generated. This check is formulated schematically as wrapping your code exp in If[ToLongQ[exp],ShortForm[exp],exp]. But if your code expression contains an UpValue that changes this code, it might not evaluate as expected.

You can catch the frontend code by defining your function to always move outside anything that's called with it for instance:

 SetAttributes[capture, HoldAll]
 head_[___, capture, r___] ^:= capture[head[x, r]]

Now calling just capture yields:

capture[If[False, triggered, 
 With[{OutputSizeLimit`Dump`boxes$ = 
       Block[{$RecursionLimit = Typeset`$RecursionLimit}, 
        MakeBoxes[capture, StandardForm]]}, 
     OutputSizeLimit`Dump`loadSizeCountRules[]; 
     If[TrueQ[BoxForm`SizeCount[OutputSizeLimit`Dump`boxes$, 1048576]], 
  OutputSizeLimit`Dump`boxes$, 
      OutputSizeLimit`Dump`encapsulateOutput[capture, $Line, $SessionID,
    5]]], capture]
 ]

Now for your code, the reason that your first definition doesn't cause the same output as the second is not due to the difference in matches between the definitions. It's because in your first example, the inner X is triggered first turning X[X[545]] into Hold[X[Y[545]], therefore when the second triggers it's not at the top level. You get the same spilling into the top behavior if you run only X[545]. This spilling behavior will happen to anything that appears at the top level and wraps out into the containing expression, as far as I can tell.

You can avoid spilling into the top as Rojo mentions by simply excluding If. But it does seem somewhat inelegant as it relies on exact knowledge about the structure of the front-end code, and could stop working if the front end changes it's wrapper.