4
$\begingroup$

Bug introduced in 8 or earlier and persisting through 12.0

CASE:3910218

When you input

{ 1 , 2 } (*spaces intended, retype them if you want to copy this*)

The underlying structure (Ctrl+Shift+E) is:

boxes = RowBox[{"{", " ", RowBox[{"1", " ", ",", " ", "2"}], " ", "}"}]

You can evaluate input cell or do something like 

ToExpression[boxes]

{1,2}

and it works nicely, however:

MakeExpression[boxes, StandardForm]

Syntax::sntxf : "{ " cannot be followed by "1 , 2 }".

ErrorBox[RowBox[{"{", " ", ErrorBox[RowBox[{"1", " ", ",", " ", "2"}]], " ", "}"}]]

Questions

  1. Is this a bug or have I missed something?

  2. What would be the best workaround?

    MakeExpression[boxes /. " " -> Nothing, StandardForm] looks fine

  3. What does the FrontEnd do before sending boxes for evaluation, LinkSnooper shows that " " are not present when the content is sent:

    EnterExpressionPacket[
    MakeExpression[
       BoxData[RowBox[{"{", RowBox[{"1", ",", "2"}], "}"}]]
     , StandardForm
    ]
]
Improve this question
$\endgroup$
6
  • $\begingroup$ Have you tried Trace[ToExpression[boxes], TraceInternal -> True]? It calls StripBoxes as the first step which removes the redundant spaces. Then the output is sent to MakeExpression which returns ErrorBox[ErrorBox[RowBox[{"1", ",", "2"}]]] as intermediate result. But I don't understand why this intermediate result is ignored and the expected output is returned nevertheless. $\endgroup$
    – Alexey Popkov
    Jun 26, 2017 at 16:11
  • $\begingroup$ From the Docs for StripBoxes: "When boxes are sent by the front end to the kernel for evaluation, the front end strips the boxes using a more extensive process that may depend upon active stylesheets and front end option settings. However, when working with functions that directly serialize notebook constructs such as NotebookRead, no stripping will occur." $\endgroup$
    – Alexey Popkov
    Jun 26, 2017 at 16:17
  • $\begingroup$ As a workaround MakeExpression[StripBoxes@boxes, StandardForm] seems to work (but note the citation in the previous comment: "... the front end strips the boxes using a more extensive process ..."). $\endgroup$
    – Alexey Popkov
    Jun 26, 2017 at 16:20
  • $\begingroup$ @AlexeyPopkov Ah, I was aware of StripBoxes but didn;t have time to play with Trace, also I didn't know if ToExpression does the same thing as what is done at Shift+Return event. p.s. I think StripBoxes is worth an answer, $\endgroup$
    – Kuba ♦
    Jun 26, 2017 at 16:34
  • $\begingroup$ I've checked it with version 5.2 and the behavior is exactly the same despite the Docs claiming that StripBoxes is introduced in version 6.0 (actually StripBoxes was present already in version 3.0!). I'm pretty sure it isn't a bug, just incomplete documentation on the difference between ToExpression and MakeExpression. $\endgroup$
    – Alexey Popkov
    Sep 21, 2017 at 9:45

2 Answers 2

Reset to default
5
$\begingroup$

Tracing the evaluation with Trace[ToExpression[boxes], TraceInternal -> True] reveals that ToExpression calls StripBoxes as the first step which removes the redundant spaces. So the obvious workaround is 

MakeExpression[StripBoxes@boxes, StandardForm]

which seems to work.

Note however that according to the Docs page for StripBoxes,

When boxes are sent by the front end to the kernel for evaluation, the front end strips the boxes using a more extensive process that may depend upon active stylesheets and front end option settings.

Improve this answer
$\endgroup$
9
  • $\begingroup$ StripBoxes is what I use now as " " is not the only problem it seems. Multi space " " is just another example. And all this is what I get from Boxes InputField. $\endgroup$
    – Kuba ♦
    Jun 27, 2017 at 5:25
  • $\begingroup$ I faced another issue, please check this MakeFormattedBoxes@Plot[Sin @ x, {x, 0, (3 + TemplateSlot["x5"]) Pi}, ImageSize -> 200 ] if you wrap this with RawBoxes and evaluate the outpput cell you get it will work fine. ToExpression @ StripBoxes @ MakeFormattedBoxes[...] does not work though. Any ideas? I'm hesitating if it is worth another question. $\endgroup$
    – Kuba ♦
    Jul 31, 2017 at 21:00
  • 2
    $\begingroup$ @Kuba John Fultz: "... there are some boxes, most notably TemplateBox, which require FE secret sauce to properly convert into expressions. " $\endgroup$
    – Alexey Popkov
    Aug 1, 2017 at 3:48
  • $\begingroup$ There are no special boxes there, the couse seems to be unorthodox RowBox placement. I could do Flatten->StringJoin->MakeExpression but it feels like an overkill. $\endgroup$
    – Kuba ♦
    Aug 1, 2017 at 7:36
  • 1
    $\begingroup$ @Kuba As John Fultz stresses in the comments linked above, we should realize that the expression - boxes - expression route isn't reliable. They just attempt to make it as realible as possible, but it still have many limitations. The fact that boxes can be rendered by FE doesn't mean that ToExpression can convert them into a valid expression. $\endgroup$
    – Alexey Popkov
    Aug 1, 2017 at 16:18
4
$\begingroup$

This is not a bug. MakeExpression, as a low-level function, only deals with canonical boxes as sent by the FE. Sometimes you get lucky with non-canonical boxes, but there's no guarantee. ToExpression, which is a more user-friendly function, will apply StripBoxes first, which attempts to simulate the canonicalization performed by the FE, so it will work on more cases (except for the occasional bug in StripBoxes, in which case you might end up with wrong thing).

The documentation example referenced in the comments was added in 12.1, but this has been the intent forever. It is perhaps still underdocumented.

BTW, newer versions of the kernel will attempt to ask the FE, if it is available, to resolve template boxes--and make their best guess from Core.nb if it is not available. Hence, John's statement about only the FE knowing the how to resolve some boxes, while still true, is not as relevant as it used to be.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.