How can I extract docs data from a WRI Workbench template?

Question

I would like to move my docs out of Workbench in something simpler, but it's painful to imagine moving over all of the pages that I already have. DocuTools has lots of custom style names and has become complicated and bloated over time, so I can't just copy paste into something new.

Is there a way I can get all the data from my Workbench docs templates programmatically and move it to something else?

Answer 1

This effort was born of trying to make Chris K's life easier. To move his stuff from Workbench, which is out-of-date, complicated, and clunky to my bare bones but cross-version, simple to use system we it had to be easy to move his docs.

To that end we'll basically do two things: 1) segment the notebook 2) transform the chunks

Notebook Segmentation

These templates look like this:

Obviously they look a little nicer with the actual stylesheet they're supposed to have, but in general they're basically just a bunch of cells indicating either metadata or sections. We want to split this so every section becomes a key in an Association and so that we have all the metadata.

I did this by defining some cell styles that would define where a new section starts and mapping them to tags. This is what that function looks like:

getStyleType[sname_] :=
 Catch[
  If[StringStartsQ[sname, "Related"],
   Throw[StringTrim[sname, "Section"]]
   ];
  KeyValueMap[
   If[StringEndsQ[sname, #], Throw[#2]] &,
   <|
    "Title" | "ObjectName" -> "Body",
    "FunctionsSection" -> "Body",
    "ExampleSection" -> "Examples",
    "ExamplesSection" -> "Examples",
    "MoreAboutSection" -> "RelatedGuides",
    "TutorialsSection" -> "RelatedTutorials",
    "SeeAlsoSection" -> "RelatedFunctions"
    |>
   ];
  None
  ]

Then I Sow every cell with the current tag, updating it if the style is right:

nbData =
 GroupBy[
  Reap[
    Do[
     Replace[
      getStyleType@cell[[2]], 
      s_String :> Set[ctag, s]
      ];
     Sow[cell, ctag],
     {cell, cellData}
     ],
    _,
    Rule
    ][[2]],
  First -> Last,
  processCellSections
  ]

And with that I have an Association telling me what all my sections are.

I also want to pull all my metadata. To do that I leverage the fact that each metadata entry has a CellLabel and its content is a String. Here's how I do that segmentation:

{Pick[cellData, #], Pick[cellLabels, #]}&@Map[StringQ, cellLabels]

where cellData is basically just NotebookRead@Cells[] and cellLabels is CurrentValues[Cells[], CellLabel]

Chunk Transformation

Metadata

This is a rather more involved process. To start off with, we need to transform the metadata we just pulled. I do that like so:

extractMetadata[cellData_, cellLabels_] :=
 With[
  {
   basics =
    AssociationThread[
     cellLabels /. {
       "Entity Type" -> "Type",
       "Paclet Name" -> "Paclet"
       },
     cellData[[All, 1]]
     ]
   },
  Normal@
    Merge[
     {
      DeleteCases[
       basics,
       "XXXX" | Except[_String]
       ],
      "Label" -> URLParse[basics["URI"], "Path"][[-1]]
      },
     First
     ] // BTools`Paclets`DocGen`DocMetadata
  ]

That BTools`Paclets`DocGen`DocMetadata is a function I wrote for a documentation system a long time ago to fill out all the necessary metadata for a documentation notebook. You can see how it's defined here.

Style Conversions

Next we have to convert over the content itself. For the most part this can just be achieved through some style conversions. That could look like:

styleConvert // Clear;
fallbackStyleConversions =
  Dispatch@{
    "TI" :> Sequence @@ {"InlineFormula", FontSlant -> Italic},
    "ModInfo" -> "Text",
    "Notes" -> "Item",
    "SeeAlso" -> "Text",
    "ObjectName" -> "Section",
    "GuideTitle" -> "Section",
    "MoreAbout" -> "Text",
    "Tutorials" -> "Text",
    "GuideAbstract" -> "Text",
    "InlineGuideFunction" :> Sequence @@ {"InlineFormula", "Input"},
    "InlineFormula" :> Sequence @@ {"InlineFormula", "Input"},
    "DisplayMath" :> 
     Sequence @@ {"DisplayFormulaNumbered", TextAlignment -> Center},
    _String?(StringStartsQ["Related"]) -> "Text"
    };
styleConvert[s_String] :=
  Sequence @@ Flatten@List@Catch[
      Scan[
       If[StringEndsQ[s, #], Throw@#] &,
       {
        "Text",
        "Link"
        }
       ];
      KeyValueMap[
       If[StringEndsQ[s, #], Throw[#2]] &,
       Join[
        <|
         "Listing" -> "Text",
         "TableMod" :> {"Input", "DetailsGrid"}
         |>,
        AssociationThread[
         NestList[StringReplace["S" -> "Subs"], "Section", 5],
         NestList[StringReplace["S" -> "Subs"], "Subsection", 5]
         ]
        ]
       ];
      Replace[s, fallbackStyleConversions]
      ];
styleConvert[e___] := e

Data Cleaning

Some times the actual BoxData the built-in docs use is unnecessarily complicated. To make this simpler we'll sometimes want to post-process this like:

convertUsageData // Clear;

convertUsageData[TextData[l_List]] :=
  Replace[
   Join @@@ Partition[
     SplitBy[
      Flatten@
       Reap[
         Replace[l,
          {
           Cell[a_, "InlineFormula", o___] :>
            Sow@Cell[a, "Code", "UsageInput", o],
           e : Except[_Cell] :> Sow[e],
           _ -> Nothing
           },
          1
          ]
         ][[2]],
      Head[#] === Cell &
      ],
     2
     ],
   {
    {c_Cell, s___} :>
     ReplaceAll[
      {
       c, 
       Cell[TextData[{s}], "Text", "UsageText"]
       },
      {
       r_String :> StringDelete[r, "\[LineSeparator]" | "\n"],
       StyleBox[a_, style_String, b___] :>
        RuleCondition[StyleBox[a, styleConvert[style], b], True]
       }
      ]
    },
   1
   ];

cleanUpInterpBoxes[InterpretationBox[a_, b_]] :=
  a;
cleanUpInterpBoxes[
   InterpretationBox[GridBox[{{a_, ___}, ___}, ___], b_]] :=
  a;

cellDataConvert // Clear
cellDataConvert[Cell[data_, "Usage", o___]] :=
  Sequence @@ {
    Cell[
     CellGroupData[Flatten@{
        Cell["", "UsageSection"],
        convertUsageData[data],
        Cell["", "UsageSectionFooter"]
        }]
     ],
    Cell["Details", "Subsection"]
    };
cellDataConvert[Cell[data_, "ExampleDelimiter", o___]] :=

 cellDataConvert@Cell["", "GuideDelimiter"]
cellDataConvert[
   Cell[data_, _String?(StringEndsQ["Delimiter"]), o___]] :=

  Cell["", "PageBreak", 
   PageBreakWithin -> False,
   PageBreakBelow -> False, 
   PageBreakAbove -> False
   ];
cellDataConvert[Cell[data_, k : "DisplayMath", o___]] :=
 Cell[
  FirstCase[data, f_FormBox :> BoxData[f], data, Infinity], 
  styleConvert[k],
  o
  ]
cellDataConvert[
   Cell[data_, s_String?(StringStartsQ["Example"]), o___]] :=
  Cell[
   data /. i_InterpretationBox :> cleanUpInterpBoxes[i],
   styleConvert[s],
   o
   ];
cellDataConvert[Cell["XXXX", ___]] :=
  Nothing;
cellDataConvert[Cell[data_, style_, o___]] :=
  Cell[data /. {
     c_Cell :> cellDataConvert[c],
     StyleBox[a_, s_String, b___] :>
      RuleCondition[StyleBox[a, styleConvert[s], b], True]
     },
    styleConvert[style], 
   o
   ];
cellDataConvert~SetAttributes~Listable;

Stitching it all together

I don't want to copy all of this code out here, since I don't think that'd be a good use of the space. Instead I'll just mention that all of the code can be found here.

We can use it through SimpleDocs too if we want. Here's how you pull the data from a function page (the same works for guides and tutorials too):

SimpleDocs@"WorkbenchData"["~/Desktop/FindEcoEvoEq.nb"] // Keys

{"Metadata", "Preamble", "Body", "RelatedTutorials", \
"RelatedDemonstrations", "RelatedLinks", "RelatedFunctions", \
"RelatedGuides", "Examples"}

And here's how you then reconvert that into a notebook that SimpleDocs can process:

SimpleDocs@"FromWorkbench"[
   "~/Desktop/FindEcoEvoEq.nb",
   "Directory" -> "~/Desktop/EcoEvo"
   ];
"~/Desktop/EcoEvo/project/docs/content/ref/FindEcoEvoEq.nb" // 
  ExpandFileName // SystemOpen