# MakeExpression can't deal with FE boxes riffled with whitespaces

Bug introduced in 8 or earlier and persisting through 12.0

CASE:3910218

When you input

{ 1 , 2 } (*spaces intended, retype them if you want to copy this*)


The underlying structure (Ctrl+Shift+E) is:

boxes = RowBox[{"{", " ", RowBox[{"1", " ", ",", " ", "2"}], " ", "}"}]


You can evaluate input cell or do something like

ToExpression[boxes]


{1,2}

and it works nicely, however:

MakeExpression[boxes, StandardForm]


Syntax::sntxf : "{ " cannot be followed by "1 , 2 }".

ErrorBox[RowBox[{"{", " ", ErrorBox[RowBox[{"1", " ", ",", " ", "2"}]], " ", "}"}]]

## Questions

1. Is this a bug or have I missed something?

2. What would be the best workaround?

MakeExpression[boxes /. " " -> Nothing, StandardForm] looks fine

3. What does the FrontEnd do before sending boxes for evaluation, LinkSnooper shows that " " are not present when the content is sent:

EnterExpressionPacket[
MakeExpression[
BoxData[RowBox[{"{", RowBox[{"1", ",", "2"}], "}"}]]
, StandardForm
]
]

• Have you tried Trace[ToExpression[boxes], TraceInternal -> True]? It calls StripBoxes as the first step which removes the redundant spaces. Then the output is sent to MakeExpression which returns ErrorBox[ErrorBox[RowBox[{"1", ",", "2"}]]] as intermediate result. But I don't understand why this intermediate result is ignored and the expected output is returned nevertheless. – Alexey Popkov Jun 26 '17 at 16:11
• From the Docs for StripBoxes: "When boxes are sent by the front end to the kernel for evaluation, the front end strips the boxes using a more extensive process that may depend upon active stylesheets and front end option settings. However, when working with functions that directly serialize notebook constructs such as NotebookRead, no stripping will occur." – Alexey Popkov Jun 26 '17 at 16:17
• As a workaround MakeExpression[StripBoxes@boxes, StandardForm] seems to work (but note the citation in the previous comment: "... the front end strips the boxes using a more extensive process ..."). – Alexey Popkov Jun 26 '17 at 16:20
• @AlexeyPopkov Ah, I was aware of StripBoxes but didn;t have time to play with Trace, also I didn't know if ToExpression does the same thing as what is done at Shift+Return event. p.s. I think StripBoxes is worth an answer, – Kuba Jun 26 '17 at 16:34
• I've checked it with version 5.2 and the behavior is exactly the same despite the Docs claiming that StripBoxes is introduced in version 6.0 (actually StripBoxes was present already in version 3.0!). I'm pretty sure it isn't a bug, just incomplete documentation on the difference between ToExpression and MakeExpression. – Alexey Popkov Sep 21 '17 at 9:45

## 2 Answers

Tracing the evaluation with Trace[ToExpression[boxes], TraceInternal -> True] reveals that ToExpression calls StripBoxes as the first step which removes the redundant spaces. So the obvious workaround is

MakeExpression[StripBoxes@boxes, StandardForm]


which seems to work.

Note however that according to the Docs page for StripBoxes,

When boxes are sent by the front end to the kernel for evaluation, the front end strips the boxes using a more extensive process that may depend upon active stylesheets and front end option settings.

• StripBoxes is what I use now as " " is not the only problem it seems. Multi space " " is just another example. And all this is what I get from Boxes InputField. – Kuba Jun 27 '17 at 5:25
• I faced another issue, please check this MakeFormattedBoxes@Plot[Sin @ x, {x, 0, (3 + TemplateSlot["x5"]) Pi}, ImageSize -> 200 ] if you wrap this with RawBoxes and evaluate the outpput cell you get it will work fine. ToExpression @ StripBoxes @ MakeFormattedBoxes[...] does not work though. Any ideas? I'm hesitating if it is worth another question. – Kuba Jul 31 '17 at 21:00
• – Alexey Popkov Aug 1 '17 at 3:48
• There are no special boxes there, the couse seems to be unorthodox RowBox placement. I could do Flatten->StringJoin->MakeExpression but it feels like an overkill. – Kuba Aug 1 '17 at 7:36
• @Kuba As John Fultz stresses in the comments linked above, we should realize that the expression - boxes - expression route isn't reliable. They just attempt to make it as realible as possible, but it still have many limitations. The fact that boxes can be rendered by FE doesn't mean that ToExpression can convert them into a valid expression. – Alexey Popkov Aug 1 '17 at 16:18

This is not a bug. MakeExpression, as a low-level function, only deals with canonical boxes as sent by the FE. Sometimes you get lucky with non-canonical boxes, but there's no guarantee. ToExpression, which is a more user-friendly function, will apply StripBoxes first, which attempts to simulate the canonicalization performed by the FE, so it will work on more cases (except for the occasional bug in StripBoxes, in which case you might end up with wrong thing).

The documentation example referenced in the comments was added in 12.1, but this has been the intent forever. It is perhaps still underdocumented.

BTW, newer versions of the kernel will attempt to ask the FE, if it is available, to resolve template boxes--and make their best guess from Core.nb if it is not available. Hence, John's statement about only the FE knowing the how to resolve some boxes, while still true, is not as relevant as it used to be.