Is it important that the front-end has a built-in parser?

Question

The Notebook interface has a parser that seems aware of all operator-precedences, allowing us to step through the expressions (rather: Box-) hierarchy with Ctrl+.

This is extremely convenient, saving a learner the trouble of calling FullForm on things to figure out precedences in many cases.

But it seems like this is a pure convenience. Afterall the Kernel can parse plain-text mathematica code just fine by itself (with ToExpression for example).

This makes me wonder: Is the result of the FrontEnd's parsing, i.e. the Boxes it creates ever used or is the code sent in plain-text form to the kernel?

Are there some operator-precedences not resolved or not correctly implemented in the FrontEnd?

Answer 1

Is it strictly necessary? In most cases, no, but doing so gives us many advantages. Most of those advantages might be worked around with various heuristics. But most importantly, it would have been very difficult to have a robust typesetting system.

First, the FrontEnd was always going to need some intermediate representation which is not simply textual. Because two-dimensional typesetting has no textual representation. And at the time we implemented our typesetting system, there was only one significant language which represented typesetting...TeX. But TeX represents the typesetting with semantic ambiguities which make it difficult to understand how to turn it into a Wolfram Language expression. Does sin(x) mean Sin[x], or does it mean Times[sin,x]? The front end has TraditionalForm, which uses heuristics to attempt to navigate these ambiguities, but I think most people would agree that it's a significant step backward from a programming language which can be represented with precise, non-heuristic semantics. So, StandardForm was born out of necessity...we want typesetting and precise semantics.

Let's now add an additional requirement to the typesetting system. It's a pretty small one. We would like the keystroke for "create fraction" to not simply allow for the creation of an empty fraction, but also allow us to type in a fraction inline (so that one can type, e.g., 1, Ctrl+/, 2). Consider it a nice usability tweak that maybe we could implement over the weekend.

If I type Ctrl+/ to input a fraction, how much should I pull from the left of the cursor to create that fraction? Obviously, if there's a number of symbol, we want that, and that's it. Well, maybe we might want some prefix operators. Prefix minus, maybe?

- a Ctrl+/ b
c - a Ctrl+/ b
c + - a Ctrl+/ b
c - - a Ctrl+/ b

If we want our prefix minus pulled in, then the second case looks pretty different than the others. In order to solve this problem, we need to be able to distinguish a prefix minus from an infix minus. But surely, we want the same thing to work for a prefix plus, as well. But maybe the whole prefix minus was just a bad convention, and we can ignore the problem.

But...we want \[PartialD], right? People want to use our system for calculus, so that seems a pretty important operator to automatically go into the fraction. Okay, so let's consider the possibility of maybe creating a set of prefix operators that we apply special behavior to. The weekend just got a little longer.

Oh, but postfix operators surely should work, too. You want all of f' to go into a fraction, right? Okay, so another rule...all postfix operators go into the fraction. Fine...we're in great shape now.

(-Sin[Cos[x]] Ctrl+/ 2 Ctrl+space + 1)

Oops. We just blew out our weekend. By the time you apply all these heuristics we've been building up, you've just gone and built yourself a parser...but a really bad one because it was created from heuristics rather than a proper sense of operators and precedence.

Okay, so let's assume that we've built a parser. What else can we do with it?

• Syntax coloring. That seems useful. Once you have a parser, it becomes really trivial to look for mismatched operators, or to determine whether a given swath of text represents a head or a body of a given expression. Doing local variable highlighting would be really hard without a parser living somewhere.

• The kernel parser can only represent complete expressions. The expression a+ is going to throw a syntax error, and there's simply no way around that. We could try to put it in a string and hide the fact that it's a string-ish thing, but that's going to be tricky. But the front end parser deals with incomplete expressions all the time. Every time you type something, you're constantly creating incomplete expressions, so it really is a requirement. Now we have a way to represent those expressions that we can attach new kernel parsing rules to.

• Structured selection. As you already conceded, so I won't elaborate on this one.

• You can do automated line-breaking which is guided by the parsed structure. Instead of merely figuring out rules based on characters which allow linebreaks, you can now add rules which depend upon things like expression depth.

• We can now create typesetting constructs with metadata so that we can provide precise semantics to the kernel while still maintaining visual fidelity to some typesetting standard. Want to say that J_n(z) maps to BesselJ? No problem. Just embed a bit of metadata into that typesetting construct (which might even include other visual distinctions so as to remove visual ambiguities), and it's all done.

Among other things.

Answer 2

Not really an answer and I may have to delete it, but...I had something of a different reaction to this business of the front end having its own parser, circa 2004. For motivation I recommend the original.

---

I knew a front end
that swallowed a kernel.
I don't know why
it swallowed a kernel.
They taste infernal.

I knew a front end
that swallowed a debugger.
(Most of us
would think that meshugger.)
It swallowed a debugger
to fix a kernel.
But I don't know why
it swallowed a kernel.
They taste infernal.

I knew a front end
that swallowed text highlighting.
Now you, or I,
would find that quite frightening.
It swallowed the highlighter
to highlight its debugger.
It swallowed the debugger
to scare bugs from the kernel.
But noone knows why
it swallowed that kernel.
They taste infernal.

I knew a front end
that swallowed interactive graphics
(Prematurely, they say,
cause the design was half-buttoxed.)
It swallowed the graphics
to illuminate highlighting.
It swallowed a highlighter
to show off debugging
It swallowed debugging
for a kernel kept chugging.
But I don't know why
it swallowed that kernel.
They taste infernal.

I knew a front end
that swallowed its code source.
It crashed, of course!

Answer 3

The result of the front-end's parsing is definitely used, but it does not need to be complete, it seems. This can be seen by constructing Cells/Boxes manually.

Try

RawBoxes@RowBox[{"1", "+", RowBox[{"2", "*", "3"}]}]
RawBoxes@RowBox[{RowBox[{"1", "+", "2"}], "*", "3"}]
RawBoxes@RowBox[{"1", "+", "2", "*", "3"}]

These all display as

1 + 2 * 3

but you can see that they behave differently in the front end by Ctrl+. '-ing around "2" or looking at the Cell expression (Ctrl+shift+e).

If you send them to the kernel for evaluation, the second one will give 9, while the others give 7 as usually. So I assume the kernel will complete the parsing but will accept any pre-parsed boxes as-is.

The frontend does not parse everything:

1~f~2~f~3

is understood as

RowBox[{"1", "~", "f", "~", "2", "~", "f", "~", "3"}]

so Ctrl+. does not tell what will be going on after parsing in this case.

The kernel turns this into f[f[1, 2], 3].

Answer 4

Maybe that is not exactly the point of your question but is relevant.

We need the FronEnd's parser, it is the only one that knows what is $CellContext` in given cell expression. It also cares about all those *`x$$ symbols which is the base of DynamicModule.

x = 5;
$CellContext`x
Button["-", Echo@$CellContext`x]

clicked button shows 5 while evaluated $CellContext`x gives its input. It is because in the Button $CellContext`x is inside ButtonBox ButtonFunction and FrontEnd parses it before deciding what to do with it.

In the former case no value was returned because a string part of input RowBox - "$CellContext`x" was sent to the Kernel for which it means nothing special.