Transform fancy usage messages in 1D string

Question

When we look at the usage messages of built-in functions nowadays (not in the good old times, when they were a simple descriptions) we see that although they look pretty in the front end, it is really hard to get a simple, one-dimensional string representation. Just try this and see yourself

ToString[Integrate::usage]

This is completely useless when one seeks for a simple string form. When you look at the Wolfram Workbench you may have noted, that they do provide simple string usage messages even for built-in functions.

Question: Is it possible to create/extract/steal simple, non-formatted usage messages for all (or almost all) built-in functions automatically? With automatically I mean having one function which works for almost all usage messages.

To give an example, the Integrate help message of the Workbench looks like this:

Integrate[f,x] gives the indefinite integral Integral f dx. Integrate[f,{x,xmin,xmax}] gives the definite integral Integral _xmin ^xmax f dx. Integrate[f,{x,xmin,xmax},{y,ymin,ymax},...] gives the multiple integral Integral _xmin ^xmaxdxIntegral _ymin ^ymaxd y ... f.

Attributes: {Protected,ReadProtected} Options: {Assumptions,GenerateConditions,PrincipalValue}

Answer 1

Edit:

Since my old approach with TeXForm turned out to be quite a bad idea, here is a new one that uses InputForm. It is much more stable and already correctly covers many, many symbols. Let's start with the code:

usageString[s_Symbol] := Module[{string,
   rules = {
     "\\\"" ~~ a___ ~~ "\\\"" /; StringFreeQ[a, "\\"] :> a,
     "\"" ~~ a__ ~~ "\"" :> a,
     "StyleBox[" ~~ a___ ~~ ", " ~~ ("TI]" | "TR]") /; 
       StringFreeQ[a, "]"] :> a,
     "\\!\\(\\*" ~~ a___ ~~ "\\)" /; StringFreeQ[a, "\\("] :> a, 
     "SubscriptBox[" ~~ a__ ~~ ", " ~~ b__ ~~ "]" /; 
       StringFreeQ[a <> b, "Box" | "]"] :> a <> "_" <> b,
     "SuperscriptBox[" ~~ a__ ~~ ", " ~~ b__ ~~ "]" /; 
       StringFreeQ[a <> b, "Box" | "]"] :> a <> "^" <> b,
     "SubsuperscriptBox[" ~~ a___ ~~ ", " ~~ b___ ~~ ", " ~~ c___ ~~ 
        "]" /; StringFreeQ[a <> b <> c, "Box" | "]"] :> 
      a <> "_(" <> b <> ")^(" <> c <> ")"}
   },
  string = 
   Fold[StringReplace, ToString[MessageName[s, "usage"], InputForm], 
    rules];
  string = FixedPoint[StringReplace[#,
      "RowBox[{" ~~ a__ ~~ "}]" /; StringFreeQ[a, "RowBox" | "}]"] :> 
       a] &,
    string];
  StringReplace[StringJoin@StringSplit[string, ", "], "\\n" -> "\n"]
  ]

Before covering the main problem that usageString still has, let's have a look at what it can do (sorry, but I have to use images to convey this):

You can see that it transforms many of the RowBox, SupersciptBox etc. constructs found in those fancy usage messages to standard strings. It still lacks some tranformation rules, however, for things like UnderoverscriptBox or StyleBox with options:

I think that by adding some more replacement rules to cover the remaining boxing constructs and options, this could be a nice way to get simple string representations of the fancy 2D strings.

Answer 2

I think I found an easy solution. Although my question was how to extract a simple 1d string, I show how to transform usages into nice and simple html. The rules for this can be adapted so that each box-structure is converted into whatever representation is wanted.

The basic trick is the following: A usage message consists of simple text and of special 2d string which are embraced in "\!\(\*" and "\)". Now the way is to extract the contents of such a special string and to transform it into a Mathematica box expression. In this nested boxes we can replace reliable. This is the main difference to what @einbandi proposed who did this box replacement in the string which will always fail at some point.

Now we have to think about a set of rules to replace the box expressions. Since I wanted this for my IDEA plugin which can handle HTML, I will create a mixture of HTML and MathML. The boxes use pure HTML

boxRules = {
   StyleBox[f_, "TI"] :> {"<em>", f, "</em>"},
   StyleBox[f_, ___] :> {f},
   RowBox[l_] :> {l},
   SubscriptBox[a_, b_] :> {a, "<sub>", b, "</sub>"},
   SuperscriptBox[a_, b_] :> {a, "<sup>", b, "</sup>"},
   RadicalBox[x_, n_] :> {x, "<sup>1/", n, "</sup>"},
   FractionBox[a_, b_] :> {"(", a, ")/(", b, ")"},
   SqrtBox[a_] :> {"&radic;(", a, ")"},
   CheckboxBox[a_, ___] :> {"<u>", a, "</u>"},
   OverscriptBox[a_, b_] :> {"Overscript[", a, b, "]"},
   OpenerBox[a__] :> {"Opener[", a, "]"},
   RadioButtonBox[a__] :> {"RadioButton[", a, "]"},
   UnderscriptBox[a_, b_] :> {"Underscript[", a, b, "]"},
   UnderoverscriptBox[a_, b_, c_] :> {"Underoverscript[", a, b, c, 
     "]"},
   SubsuperscriptBox[a_, b_, c_] :> {a, "<sub><small>", b, 
     "</small></sub><sup><small>", c, "</small></sup>"}
   };

With this rules we can replace inside a box expression until nothing changes anymore.

convertBoxExpressionToHTML[boxexpr_] := 
 StringJoin[
  ToString /@ 
   Flatten[ReleaseHold[MakeExpression[boxexpr] //. boxRules]]]

This is basically everything you need to create a html-page of the usages of all known functions. Since I put some more stuff in it like

• creating of links to the official online documentation
• display of attributes
• display of options

I'll put the whole code at the end of this post and please note that it is not cleaned. The whole page looks then like this. Although there are some minor things (like nested 2d strings which are used about 5 times) I think for my plugin I can live with this:

extractUsage[str_] := 
 With[{usg = 
    Function[expr, expr::usage, HoldAll] @@ MakeExpression[str]},
  If[Head[usg] === String, usg, ""]]

createLinkName[s_] := 
 If[StringMatchQ[ToString@FullForm[s], "\"\\[" ~~ __ ~~ "]\""],
  {StringReplace[ToString@FullForm[s], {"\"" :> "", "\\" -> "\\\\"}],
   StringReplace[
    ToString@FullForm[s], {"\"" :> "", 
     "\\[" ~~ c__ ~~ "]" :> "character/" ~~ c}]},
  {s, s}]

createOptionString[s_] := 
 With[{opts = 
    Function[expr, Options[expr], HoldAll] @@ MakeExpression[s]},
  If[opts === {},
   "</p><b>Symbol has no options.</b>",
   "</p><b>Options: </b>" <> 
    StringJoin@Riffle[ToString[First[#]] & /@ opts, ", "]
   ]
  ]

createHtmlUsage[s_String] := Module[{
   usg = extractUsage[s],
   attr = Attributes[s],
   link, linkname},
  {linkname, link} = createLinkName[s];

  "<h3><a href=\"http://reference.wolfram.com/mathematica/ref/" <> 
   link <> ".html\">" <> linkname <> "</a></h3>" <> If[usg =!= "",
    "<ul><li>" <>
     StringReplace[
      StringReplace[
       usg, {Shortest["\!\(\*" ~~ content__ ~~ "\)"] :> 
         convertBoxExpressionToHTML[content],
        "\n" :> "<li>"}
       ], {"\[Null]" :> "", 
       a_?(StringMatchQ[ToString@FullForm[#], 
            "\"\\[" ~~ __ ~~ "]\""] &) :> 
        StringReplace[
         ToString[a, MathMLForm], {WhitespaceCharacter :> ""}]}
      ] <> "</ul>", ""] <> "<b>Attributes:</b> " <> 
   StringJoin[ToString /@ Riffle[attr, ", "]] <> 
   createOptionString[s] <> "\n"
  ]

names = Names["System`*"];
Export["tmp/usageMessages.html", StringJoin[createHtmlUsage /@ names], "Text"]

Answer 3

i have a solution that uses the UndocumentedTestFEParserPacket as described by John Fultz! in this question.

This funny named Tool parses a String Input into the real unchanged Mathematica BoxForm.

This is my code:

str0 = Sum::usage

str1=StringJoin[ToString[StringReplace[#, "\\\"" -> "\""]]& /@
(Riffle[MathLink`CallFrontEnd[
FrontEnd`UndocumentedTestFEParserPacket[str0, True]]〚1〛
//. RowBox[{seq___}] :> seq /. BoxData -> List, " "]
/. SubscriptBox[a_, b_] :> a<>"_"<>b
/. Except[List, _Symbol][args__] :> Sequence@@Riffle[{args}, " "])];

str2 = Fold[StringReplace, str1, 
{((WhitespaceCharacter...)~~br:("["|"("|"=") ~~ (WhitespaceCharacter ...)) :> br,
((WhitespaceCharacter ...) ~~ br:("]"|"}"|","|".")) :> br, 
(br:("{") ~~ (WhitespaceCharacter ...)) :> br, 
". " ~~ Except[EndOfString] -> ". \n"}]

and this is how the Output looks like (first Output formatted fancy str0, second simple flat str2)

Code Explanation:

str0 is the formatted string with all the StyleBoxes and other formatting boxes.

str1:

UndocumentedTestFEParserPacket[str0, True] gives Boxes and strips off all StyleBoxes, thats because the second argument is True. First Replacement removes all RowBoxes. The outer BoxForm changed to a List of strings. Whitespaces are inserted between these strings the by Riffle. SubscriptBox gets a special treatment. The last line replaces every remaining FormatBox such as UnderoverscriptBox and it does that by adding Whitespaces between the arguments, and returning the arguments as a flat Sequence.

Edit:

ToString[StringReplace[#, "\\\"" -> "\""]]& /@

was added to include more cases such as StringReplace::usage like halirutan mentioned in his comment. This cases include string representations "" with Styles inside of a the usage-string, when "args" has to be given as strings. Its funny that StringReplace helped to fix its own usage-Message :))

str2:

In this block of code i only remove unwanted WhitespaceCharacter from the string str1 and i add linebreaks "/n" after the ".", because they got lost during the Parsing. There are 3 different cases where WhitespaceCharacter can be removed. 1 removing left-and right sided WithespaceCharacter from a character like "[". 2. and 3. removing WithespaceCharacter from left(2) or right(3) side.

Context of this Answer

I found this question important because i am currently working on rewriting the ApplicationMaker by jmlopez for automatic creation of Mathematica Documentation. If you want to build ReferencePages, you might use the usage information to fill in some contents. The formatted usage-messages can be used to make a definitionbox of the symbol, but for the summary that appears, when you search for a symbol you need a simple flat String, otherwise all the StyleBoxes will show up. This is also discussed here: formatting usage messages

Answer 4

In order to obtain the non-formatted usage string for any symbol, e.g. Integrate, you can just use WolframLanguageData["Integrate", "PlaintextUsage"].

A function which does this for any symbol may be defined as

plainUsage[sym_] := WolframLanguageData[ToString[sym], "PlaintextUsage"]

For example:

Of course, the string may still contain a few special characters, such as the integral sign which you can convert to \[Integral] using FullForm.