# Transform fancy usage messages in 1D string

When we look at the usage messages of built-in functions nowadays (not in the good old times, when they were a simple descriptions) we see that although they look pretty in the front end, it is really hard to get a simple, one-dimensional string representation. Just try this and see yourself

ToString[Integrate::usage]


This is completely useless when one seeks for a simple string form. When you look at the Wolfram Workbench you may have noted, that they do provide simple string usage messages even for built-in functions.

Question: Is it possible to create/extract/steal simple, non-formatted usage messages for all (or almost all) built-in functions automatically? With automatically I mean having one function which works for almost all usage messages.

To give an example, the Integrate help message of the Workbench looks like this:

Integrate[f,x] gives the indefinite integral Integral f dx. Integrate[f,{x,xmin,xmax}] gives the definite integral Integral _xmin ^xmax f dx. Integrate[f,{x,xmin,xmax},{y,ymin,ymax},...] gives the multiple integral Integral _xmin ^xmaxdxIntegral _ymin ^ymaxd y ... f.

-
Would it be possible to export it from the WB plugin? –  rm -rf Apr 5 '13 at 16:24
I presume you've already tried playing with the second parameter of ToString? –  Mr.Wizard Apr 5 '13 at 16:27
@rm-rf I have tried to find some information in the plugin folder of Eclipse and indeed you find some stuff but not for built-in functions. Maybe I should carefully grep the contents of those folders again. –  halirutan Apr 5 '13 at 17:05
@Mr.Wizard Yes, but without luck. I'm open for ideas. –  halirutan Apr 5 '13 at 17:06
Sorry, no ideas. :-/ –  Mr.Wizard Apr 5 '13 at 17:07

i have a solution that uses the UndocumentedTestFEParserPacket as described by John Fultz! in this question.

This funny named Tool parses a String Input into the real unchanged Mathematica BoxForm.

This is my code:

str0 = Sum::usage

str1=StringJoin[ToString[StringReplace[#, "\\\"" -> "\""]]& /@
(Riffle[MathLinkCallFrontEnd[
FrontEndUndocumentedTestFEParserPacket[str0, True]]〚1〛
//. RowBox[{seq___}] :> seq /. BoxData -> List, " "]
/. SubscriptBox[a_, b_] :> a<>"_"<>b
/. Except[List, _Symbol][args__] :> Sequence@@Riffle[{args}, " "])];

str2 = Fold[StringReplace, str1,
{((WhitespaceCharacter...)~~br:("["|"("|"=") ~~ (WhitespaceCharacter ...)) :> br,
((WhitespaceCharacter ...) ~~ br:("]"|"}"|","|".")) :> br,
(br:("{") ~~ (WhitespaceCharacter ...)) :> br,
". " ~~ Except[EndOfString] -> ". \n"}]


and this is how the Output looks like (first Output formatted fancy str0, second simple flat str2)

Code Explanation:

str0 is the formatted string with all the StyleBoxes and other formatting boxes.

str1:

UndocumentedTestFEParserPacket[str0, True] gives Boxes and strips off all StyleBoxes, thats because the second argument is True. First Replacement removes all RowBoxes. The outer BoxForm changed to a List of strings. Whitespaces are inserted between these strings the by Riffle. SubscriptBox gets a special treatment. The last line replaces every remaining FormatBox such as UnderoverscriptBox and it does that by adding Whitespaces between the arguments, and returning the arguments as a flat Sequence.

Edit:

ToString[StringReplace[#, "\\\"" -> "\""]]& /@


was added to include more cases such as StringReplace::usage like halirutan mentioned in his comment. This cases include string representations "" with Styles inside of a the usage-string, when "args" has to be given as strings. Its funny that StringReplace helped to fix its own usage-Message :))

str2:

In this block of code i only remove unwanted WhitespaceCharacter from the string str1 and i add linebreaks "/n" after the ".", because they got lost during the Parsing. There are 3 different cases where WhitespaceCharacter can be removed. 1 removing left-and right sided WithespaceCharacter from a character like "[". 2. and 3. removing WithespaceCharacter from left(2) or right(3) side.

Thanks for the answer, you got my upvote. Not to discourage you, but only to show that there are a lot of places that call for trouble, try: str0 = StringReplace::usage. For my usage-extractor I had to make sure that really all usages work. –  halirutan Mar 22 at 15:23
Some of them are not only hard cases, they are plain wrong. I don't remember which exactly, but most of the times it happened with usage strings that itself contained strings with formatting like the StringReplace example I gave. Since I had to convert all of them, I just stopped when errors happened and looked what went wrong. –  halirutan Mar 26 at 5:57
Btw, if you write a function that takes a symbol (or the symbol name as string) and returns the formatted usages, be very careful to never evaluate the symbol on its way through your function. Nowadays, we have simple variables that want to connect you to the internet. Try for instance f[s_] := MessageName[s, "usage"]; f[$CloudRootDirectory] – halirutan Mar 26 at 6:01 good hint but MessageName[s,"usage"] is equivalent to s::usage. s::usage also holds its argument s, since MessageName has Attribute HoldFirst. But something like Evaluate[Symbol["SymbolName"]]::usage would be dangerous. I have no clue how to prevent a Symbol from Evaluation, when i have to transform from the symbolName(String) to the Symbol, because this Transformation allready is somehow an Evaluation... maybe with ToExpression["SymbolName", StandardForm, Hold]? – sacratus Mar 27 at 4:49 I think I found an easy solution. Although my question was how to extract a simple 1d string, I show how to transform usages into nice and simple html. The rules for this can be adapted so that each box-structure is converted into whatever representation is wanted. The basic trick is the following: A usage message consists of simple text and of special 2d string which are embraced in "\!$$\*" and "$$". Now the way is to extract the contents of such a special string and to transform it into a Mathematica box expression. In this nested boxes we can replace reliable. This is the main difference to what @einbandi proposed who did this box replacement in the string which will always fail at some point. Now we have to think about a set of rules to replace the box expressions. Since I wanted this for my IDEA plugin which can handle HTML, I will create a mixture of HTML and MathML. The boxes use pure HTML boxRules = { StyleBox[f_, "TI"] :> {"<em>", f, "</em>"}, StyleBox[f_, ___] :> {f}, RowBox[l_] :> {l}, SubscriptBox[a_, b_] :> {a, "<sub>", b, "</sub>"}, SuperscriptBox[a_, b_] :> {a, "<sup>", b, "</sup>"}, RadicalBox[x_, n_] :> {x, "<sup>1/", n, "</sup>"}, FractionBox[a_, b_] :> {"(", a, ")/(", b, ")"}, SqrtBox[a_] :> {"&radic;(", a, ")"}, CheckboxBox[a_, ___] :> {"<u>", a, "</u>"}, OverscriptBox[a_, b_] :> {"Overscript[", a, b, "]"}, OpenerBox[a__] :> {"Opener[", a, "]"}, RadioButtonBox[a__] :> {"RadioButton[", a, "]"}, UnderscriptBox[a_, b_] :> {"Underscript[", a, b, "]"}, UnderoverscriptBox[a_, b_, c_] :> {"Underoverscript[", a, b, c, "]"}, SubsuperscriptBox[a_, b_, c_] :> {a, "<sub><small>", b, "</small></sub><sup><small>", c, "</small></sup>"} };  With this rules we can replace inside a box expression until nothing changes anymore. convertBoxExpressionToHTML[boxexpr_] := StringJoin[ ToString /@ Flatten[ReleaseHold[MakeExpression[boxexpr] //. boxRules]]]  This is basically everything you need to create a html-page of the usages of all known functions. Since I put some more stuff in it like • creating of links to the official online documentation • display of attributes • display of options I'll put the whole code at the end of this post and please note that it is not cleaned. The whole page looks then like this. Although there are some minor things (like nested 2d strings which are used about 5 times) I think for my plugin I can live with this: extractUsage[str_] := With[{usg = Function[expr, expr::usage, HoldAll] @@ MakeExpression[str]}, If[Head[usg] === String, usg, ""]] createLinkName[s_] := If[StringMatchQ[ToString@FullForm[s], "\"\\[" ~~ __ ~~ "]\""], {StringReplace[ToString@FullForm[s], {"\"" :> "", "\\" -> "\\\\"}], StringReplace[ ToString@FullForm[s], {"\"" :> "", "\\[" ~~ c__ ~~ "]" :> "character/" ~~ c}]}, {s, s}] createOptionString[s_] := With[{opts = Function[expr, Options[expr], HoldAll] @@ MakeExpression[s]}, If[opts === {}, "</p><b>Symbol has no options.</b>", "</p><b>Options: </b>" <> StringJoin@Riffle[ToString[First[#]] & /@ opts, ", "] ] ] createHtmlUsage[s_String] := Module[{ usg = extractUsage[s], attr = Attributes[s], link, linkname}, {linkname, link} = createLinkName[s]; "<h3><a href=\"http://reference.wolfram.com/mathematica/ref/" <> link <> ".html\">" <> linkname <> "</a></h3>" <> If[usg =!= "", "<ul><li>" <> StringReplace[ StringReplace[ usg, {Shortest["\!$$\*" ~~ content__ ~~ "$$"] :> convertBoxExpressionToHTML[content], "\n" :> "<li>"} ], {"\[Null]" :> "", a_?(StringMatchQ[ToString@FullForm[#], "\"\\[" ~~ __ ~~ "]\""] &) :> StringReplace[ ToString[a, MathMLForm], {WhitespaceCharacter :> ""}]} ] <> "</ul>", ""] <> "<b>Attributes:</b> " <> StringJoin[ToString /@ Riffle[attr, ", "]] <> createOptionString[s] <> "\n" ] names = Names["System*"]; Export["tmp/usageMessages.html", StringJoin[createHtmlUsage /@ names], "Text"]  - But does IDEA render MathML ... ? – Szabolcs Apr 15 '13 at 16:02 @Szabolcs Yep. When you press Ctrl+q (or Ctrl+j on Mac) and the documentation pops up then what you see are snips of HTML. I tried it and you can mix it with MathML. Looks awesome. I probably post a screenshot of that too. – halirutan Apr 15 '13 at 17:31 Wow, that's a surprise! I really wonder what rendering engine it uses. I thought only Gecko had MathML support, and WebKit had a simple one which was turned on only in Safari (??) Does it work on all platforms? – Szabolcs Apr 15 '13 at 20:52 @Szabolcs Currently, I only tested it on OSX since I was writing it yesterday during a DVD. But the mix of html and MathML works in both, my Chrome and IDEA. I haven't really investigated in this issue, I just tried it because it was my only hope.. – halirutan Apr 15 '13 at 20:55 If I try this page and select MathML from the dropdown, Chrome doesn't render the MathML at all, Safari renders it poorly and Firefox renders it well. But it's possible that the examples you have don't even use MathML because they only need subscripts? – Szabolcs Apr 16 '13 at 3:10 Edit: Since my old approach with TeXForm turned out to be quite a bad idea, here is a new one that uses InputForm. It is much more stable and already correctly covers many, many symbols. Let's start with the code: usageString[s_Symbol] := Module[{string, rules = { "\\\"" ~~ a___ ~~ "\\\"" /; StringFreeQ[a, "\\"] :> a, "\"" ~~ a__ ~~ "\"" :> a, "StyleBox[" ~~ a___ ~~ ", " ~~ ("TI]" | "TR]") /; StringFreeQ[a, "]"] :> a, "\\!$\\*" ~~ a___ ~~ "$" /; StringFreeQ[a, "\\("] :> a, "SubscriptBox[" ~~ a__ ~~ ", " ~~ b__ ~~ "]" /; StringFreeQ[a <> b, "Box" | "]"] :> a <> "_" <> b, "SuperscriptBox[" ~~ a__ ~~ ", " ~~ b__ ~~ "]" /; StringFreeQ[a <> b, "Box" | "]"] :> a <> "^" <> b, "SubsuperscriptBox[" ~~ a___ ~~ ", " ~~ b___ ~~ ", " ~~ c___ ~~ "]" /; StringFreeQ[a <> b <> c, "Box" | "]"] :> a <> "_(" <> b <> ")^(" <> c <> ")"} }, string = Fold[StringReplace, ToString[MessageName[s, "usage"], InputForm], rules]; string = FixedPoint[StringReplace[#, "RowBox[{" ~~ a__ ~~ "}]" /; StringFreeQ[a, "RowBox" | "}]"] :> a] &, string]; StringReplace[StringJoin@StringSplit[string, ", "], "\\n" -> "\n"] ]  Before covering the main problem that usageString still has, let's have a look at what it can do (sorry, but I have to use images to convey this): You can see that it transforms many of the RowBox, SupersciptBox etc. constructs found in those fancy usage messages to standard strings. It still lacks some tranformation rules, however, for things like UnderoverscriptBox or StyleBox with options: I think that by adding some more replacement rules to cover the remaining boxing constructs and options, this could be a nice way to get simple string representations of the fancy 2D strings. - it does not work properly on mathematica 9 macos X. I get$RecursionLimit::reclim: Recursion depth of 4096 exceeded for usageString[Integrate::usage] –  chris Apr 6 '13 at 7:42
@chris This is not the fault of einbandi! Somehow TeXForm is broken. On Linux too. Just try TeXForm[Integrate::usage]. –  halirutan Apr 6 '13 at 8:35
@einbandi +1 for your effort, although I want to have it for all approx. 5000 symbols. If I have to look at the TeX-code for every message manually, it is maybe fast to just use ToString and rewrite incorrect text. –  halirutan Apr 6 '13 at 8:43
@halirutan Please have a look at the updated code. (Vll kann man damit ja mehr anfangen :)`) –  einbandi Apr 6 '13 at 23:29