Convert between string and expression representation of boxes

Question

Boxes may be represented as expressions or as strings. How can I convert between the two forms? In particular, how can I convert from the string representation to the expression representation?

Example string representation:

"For all \!\(\*FormBox[\(x > 0\),TraditionalForm]\) there is an \!\(\*FormBox[\(y \[Element] \[DoubleStruckCapitalR]\),TraditionalForm]\) such that \!\(\*FormBox[\(\*SuperscriptBox[\(y\), \(2\)] = x\),TraditionalForm]\)."

Example expression representation:

RowBox[{"For", " ", "all", " ", 
    FormBox[
      RowBox[{"x", ">", "0"}],
      TraditionalForm], " ", "there", " ", "is", " ", "an", " ", 
    FormBox[
      RowBox[{"y", "\[Element]", "\[DoubleStruckCapitalR]"}],
      TraditionalForm], " ", "such", " ", "that", " ", 
    FormBox[
      RowBox[{
         SuperscriptBox["y", "2"], "=", "x"}],
      TraditionalForm]}]

Answer 1

It seems relatively straightforward to implement the translation to boxes from linear syntax:

toBoxForm = FrontEndExecute @* FrontEnd`ReparseBoxStructurePacket

For the reverse I wasn't able to find a really clean solution. The one I thought of initially is this:

toStringFormNaive = Function[
  boxes, 
  First @ FrontEndExecute @ FrontEnd`ExportPacket[
    BoxData @ boxes, 
    "InputText"
  ]
]

this however does not round-trip.

The best I was able to do is this rather hacky one:

boxesToString[boxes_, form_] := 
  ToExpression[
    boxes, 
    form, 
    Function[e, ToString[Unevaluated @ e, form], HoldAllComplete]
  ];

allCases[expr_, patt_] := 
  Cases[Unevaluated @ expr, patt, {0, Infinity}, Heads -> True]

toStringFormExperimental = 
  Function[
    boxes, 
    StringJoin @ allCases[
        ReplaceAll[
            boxes, 
            {
                expr: FormBox[_, form_] :>  boxesToString[expr, form], 
                expr: (Except[List | RowBox][___]) :> boxesToString[expr, StandardForm]
            }
        ],
        _String
    ]
  ]

which doesn't fully round-trip either, but, at least on your input, produces something much closer to the original string.

This code also has an undesirable side effect of possibly producing symbols in global or other namespaces as a result of calling ToExpression. I didn't make any attempt to fix this, although one could in principle make the parsing happen in some temporary context and then Remove that context.

So this is definitely a hack which is almost certain to break in some cases. I will gladly remove this answer or at least this part of it if / when something better appears, but I thought this might still be helpful in the mean time.

A few links: reparse function by Kuba, and this answer by John Fultz.