What is the complete sequence of evaluations/transformations from submitting a cell to actual evaluation?

Question

This question is inspired by that one. Reading it, I noticed that while (I think) I know what happens when entering an expression directly to the kernel (i.e. after typing math in the terminal), the notebook adds extra complications which I don't fully understand.

To start with, I describe what I think happens when interacting directly with the Kernel (so you can correct me if I'm wrong):

• After the expression is input, it first is a string, which is passed to $PreRead (if set). That function is expected to return a string, which gets assigned to InString[$Line].
• Then this string is parsed (I guess with MakeExpression), and the resulting expression assigned to In[$Line]
• Then $Pre is applied to this expression, if defined.
• Finally, the expression obtained this way is evaluated

(After the expression is evaluated, other things happen to generate the output, but that's outside the scope of this question.)

Now, for Notebooks, things are obviously more complicated:

• To start with, as far as I understand, the notebook cells contain boxes, not strings. However I noticed that nevertheless InString is set. What is not clear to me is whether this string is part of the processing, or if the boxes-to-string conversion is only done in order to set InString.
• Next, there's the Notation package, which also is involved in the pre-evaluation process. However being a package, I guess it has to hook into some lower-level interface to do its work.
• There's also the fact that notebooks can define their own private contexts. Those contexts also have to somehow be communicated to the kernel. Since expressions from different notebooks can be sent to the kernel in arbitrary order, those contexts need to be associated with the expression, and setting them must thus be part of expression pre-evaluation.
• There might be other things which I'm not aware of.

Therefore my question is:

What is the exact sequence of events which happens when submitting a notebook cell, as far as customizable behaviour is concerned?

Answer 1

Preamble

I will interpret your question in the narrow context of customizibility. Most of the information can be found in the documentation, but is alas scattered over several places. Below I will attempt to assemble a single sequence, mostly using this source. Where my description deviates from it, I will put the LS sign indicating that what follows ismy interpretation. Also, I have to make a disclaimer that some parts of this I haven't tested thoroughly enough to be fully sure that I am 100 percent correct, so I invite anyone to edit this if some mistakes are found.

Evaluation sequence

With that out of the way, here is what I believe is a reasonably accurate description of what happens when you press SHIFT+ENTER to evaluate a cell in the FrontEnd session:

• Evaluate CellProlog, if defined for a given cell (LS: CellProlog is an option for the cell which defines a piece of code to evaluate, not a function of the input).

• Evaluate CellEvaluationFunction, if defined for a given cell

  • In a StandardForm or TraditionalForm cell, CellEvaluationFunction is applied to the BoxData expression representing the input to be evaluated.
  • In an InputForm cell, it is applied to the string corresponding to the input to be evaluated.

  LS: In all cases, this is a function which is applied to the input (string or boxes). At this point, one can divert the entire evaluation loop as one wants, including evaluations not involving Mathematica kernel at all (e.g. code in other languages, processed externally. See this excellent answer by WReach for the exposition of the possibilities this offers).

  All the steps below assume the default CellEvaluationFunction, which is Identity

• Read in input (LS: in a string or boxes form, presumably depends on the cell type).

• Apply $PreRead function, if defined, to the input string (LS: also for a box expression if an input is read as boxes. Note that if CellEvaluationFunction has been defined and still calls the kernel to evaluate the result, then $PreRead is applied to the result of CellEvaluationFunction execution).

• Print syntax warnings if necessary.

• Apply $SyntaxHandler function if there is a syntax error.

• LS: Call MakeExpression to create an expression from boxes

• LS: As a part of expression creation, call $NewSymbol on every new symbol to be created, then create that symbol. The choice of context where the symbol is created is based on the current (at the time of symbol's creation) values of $Context and $ContextPath. Obviously, cell and notebook contexts, when enabled, must be communicated to the kernel as a part of the cell evaluation process. That this is indeed the case, you can confirm by evaluating $Context variable in such a cell. How and at which stage this is done I don't know, perhaps someone will add this info.

• Assign InString[n].

• Apply $Pre function, if defined, to the input expression.

• Assign In[n].

• Evaluate expression which is a process with many steps as explained at here.

• Apply $Post function, if defined.

• Assign Out[n], stripping off any formatting wrappers.

• Apply $PrePrint function, if defined.

• Assign MessageList[n] and clear $MessageList.

• Print expression, if it is not Null.

• Increment $Line.

• Clear any pending aborts.

• Evaluate CellEpilog, if defined for a given cell.

Notation` package

The Notation` package hooks into MakeExpression. What this means is that the notation changes one can do with it are limited to what is possible to do on the level of fully parsed boxes, rather than strings. In other words, your input must represent a syntactically valid string of Mathematica code, for the purposes of parsing to boxes.

This means that there will be a number of things you can not do, such as change the precedence of operators, or introduce syntax which won't normally parse to boxes. Generally, this means that you can not assign custom string syntax, rather you can somewhat augment Mathematica's one, within what can be done on the box level.

Another limitation of Notation` is that, since it is based on MakeExpression, and the latter is not called by Get (which is called to read in packages), Notation` will not work for packages without some extra work-arounds. One such work-around I suggested here. While it is really a rather ugly hack, it illustrates the way Notation` works, to some extent. More information about the workings of Notation` can be obtained, of course, by reading the source code of Notation.m.

Further information

Some additional sources of information include

• Reference pages for the functions or options such as CellProlog,CellEpilog, CellEvaluationFunction, etc.
• A good brief (but dated) description is the 1992 WRI technical report "Mathematica internals" by David Withoff (available online as .pdf).
• Some interesting investigations on the order of some of the operations here were carried out by Alexey Popkov in this question.

Remarks

I am almost certain that I missed or misrepresented something, simply because I don't have a lot of personal experience with all this steps, or exhaustive tests performed to test this description. If you found a mistake, feel free to edit.

Answer 2

As far as the Notation package is concerned, I suspect, that it acts on the MakeExpression (and MakeBoxes for output) stages (not something lower-level).

An interesting Add-on to your question is: At which stages does MMA define/handel which of it's internal replacements? (trivial example: Spaces are treated as Times[]) Where can we look up, erase or change the definition of that? I suspect it is the MakeExpression stage, too.