5
$\begingroup$

I would like to move my docs out of Workbench in something simpler, but it's painful to imagine moving over all of the pages that I already have. DocuTools has lots of custom style names and has become complicated and bloated over time, so I can't just copy paste into something new.

Is there a way I can get all the data from my Workbench docs templates programmatically and move it to something else?

$\endgroup$
6
$\begingroup$

This effort was born of trying to make Chris K's life easier. To move his stuff from Workbench, which is out-of-date, complicated, and clunky to my bare bones but cross-version, simple to use system we it had to be easy to move his docs.

To that end we'll basically do two things: 1) segment the notebook 2) transform the chunks

Notebook Segmentation

These templates look like this:

enter image description here

Obviously they look a little nicer with the actual stylesheet they're supposed to have, but in general they're basically just a bunch of cells indicating either metadata or sections. We want to split this so every section becomes a key in an Association and so that we have all the metadata.

I did this by defining some cell styles that would define where a new section starts and mapping them to tags. This is what that function looks like:

getStyleType[sname_] :=
 Catch[
  If[StringStartsQ[sname, "Related"],
   Throw[StringTrim[sname, "Section"]]
   ];
  KeyValueMap[
   If[StringEndsQ[sname, #], Throw[#2]] &,
   <|
    "Title" | "ObjectName" -> "Body",
    "FunctionsSection" -> "Body",
    "ExampleSection" -> "Examples",
    "ExamplesSection" -> "Examples",
    "MoreAboutSection" -> "RelatedGuides",
    "TutorialsSection" -> "RelatedTutorials",
    "SeeAlsoSection" -> "RelatedFunctions"
    |>
   ];
  None
  ]

Then I Sow every cell with the current tag, updating it if the style is right:

nbData =
 GroupBy[
  Reap[
    Do[
     Replace[
      getStyleType@cell[[2]], 
      s_String :> Set[ctag, s]
      ];
     Sow[cell, ctag],
     {cell, cellData}
     ],
    _,
    Rule
    ][[2]],
  First -> Last,
  processCellSections
  ]

And with that I have an Association telling me what all my sections are.

I also want to pull all my metadata. To do that I leverage the fact that each metadata entry has a CellLabel and its content is a String. Here's how I do that segmentation:

{Pick[cellData, #], Pick[cellLabels, #]}&@Map[StringQ, cellLabels]

where cellData is basically just NotebookRead@Cells[] and cellLabels is CurrentValues[Cells[], CellLabel]

Chunk Transformation

Metadata

This is a rather more involved process. To start off with, we need to transform the metadata we just pulled. I do that like so:

extractMetadata[cellData_, cellLabels_] :=
 With[
  {
   basics =
    AssociationThread[
     cellLabels /. {
       "Entity Type" -> "Type",
       "Paclet Name" -> "Paclet"
       },
     cellData[[All, 1]]
     ]
   },
  Normal@
    Merge[
     {
      DeleteCases[
       basics,
       "XXXX" | Except[_String]
       ],
      "Label" -> URLParse[basics["URI"], "Path"][[-1]]
      },
     First
     ] // BTools`Paclets`DocGen`DocMetadata
  ]

That BTools`Paclets`DocGen`DocMetadata is a function I wrote for a documentation system a long time ago to fill out all the necessary metadata for a documentation notebook. You can see how it's defined here.

Style Conversions

Next we have to convert over the content itself. For the most part this can just be achieved through some style conversions. That could look like:

styleConvert // Clear;
fallbackStyleConversions =
  Dispatch@{
    "TI" :> Sequence @@ {"InlineFormula", FontSlant -> Italic},
    "ModInfo" -> "Text",
    "Notes" -> "Item",
    "SeeAlso" -> "Text",
    "ObjectName" -> "Section",
    "GuideTitle" -> "Section",
    "MoreAbout" -> "Text",
    "Tutorials" -> "Text",
    "GuideAbstract" -> "Text",
    "InlineGuideFunction" :> Sequence @@ {"InlineFormula", "Input"},
    "InlineFormula" :> Sequence @@ {"InlineFormula", "Input"},
    "DisplayMath" :> 
     Sequence @@ {"DisplayFormulaNumbered", TextAlignment -> Center},
    _String?(StringStartsQ["Related"]) -> "Text"
    };
styleConvert[s_String] :=
  Sequence @@ Flatten@List@Catch[
      Scan[
       If[StringEndsQ[s, #], Throw@#] &,
       {
        "Text",
        "Link"
        }
       ];
      KeyValueMap[
       If[StringEndsQ[s, #], Throw[#2]] &,
       Join[
        <|
         "Listing" -> "Text",
         "TableMod" :> {"Input", "DetailsGrid"}
         |>,
        AssociationThread[
         NestList[StringReplace["S" -> "Subs"], "Section", 5],
         NestList[StringReplace["S" -> "Subs"], "Subsection", 5]
         ]
        ]
       ];
      Replace[s, fallbackStyleConversions]
      ];
styleConvert[e___] := e

Data Cleaning

Some times the actual BoxData the built-in docs use is unnecessarily complicated. To make this simpler we'll sometimes want to post-process this like:

convertUsageData // Clear;

convertUsageData[TextData[l_List]] :=
  Replace[
   Join @@@ Partition[
     SplitBy[
      Flatten@
       Reap[
         Replace[l,
          {
           Cell[a_, "InlineFormula", o___] :>
            Sow@Cell[a, "Code", "UsageInput", o],
           e : Except[_Cell] :> Sow[e],
           _ -> Nothing
           },
          1
          ]
         ][[2]],
      Head[#] === Cell &
      ],
     2
     ],
   {
    {c_Cell, s___} :>
     ReplaceAll[
      {
       c, 
       Cell[TextData[{s}], "Text", "UsageText"]
       },
      {
       r_String :> StringDelete[r, "\[LineSeparator]" | "\n"],
       StyleBox[a_, style_String, b___] :>
        RuleCondition[StyleBox[a, styleConvert[style], b], True]
       }
      ]
    },
   1
   ];

cleanUpInterpBoxes[InterpretationBox[a_, b_]] :=
  a;
cleanUpInterpBoxes[
   InterpretationBox[GridBox[{{a_, ___}, ___}, ___], b_]] :=
  a;

cellDataConvert // Clear
cellDataConvert[Cell[data_, "Usage", o___]] :=
  Sequence @@ {
    Cell[
     CellGroupData[Flatten@{
        Cell["", "UsageSection"],
        convertUsageData[data],
        Cell["", "UsageSectionFooter"]
        }]
     ],
    Cell["Details", "Subsection"]
    };
cellDataConvert[Cell[data_, "ExampleDelimiter", o___]] :=

 cellDataConvert@Cell["", "GuideDelimiter"]
cellDataConvert[
   Cell[data_, _String?(StringEndsQ["Delimiter"]), o___]] :=

  Cell["", "PageBreak", 
   PageBreakWithin -> False,
   PageBreakBelow -> False, 
   PageBreakAbove -> False
   ];
cellDataConvert[Cell[data_, k : "DisplayMath", o___]] :=
 Cell[
  FirstCase[data, f_FormBox :> BoxData[f], data, Infinity], 
  styleConvert[k],
  o
  ]
cellDataConvert[
   Cell[data_, s_String?(StringStartsQ["Example"]), o___]] :=
  Cell[
   data /. i_InterpretationBox :> cleanUpInterpBoxes[i],
   styleConvert[s],
   o
   ];
cellDataConvert[Cell["XXXX", ___]] :=
  Nothing;
cellDataConvert[Cell[data_, style_, o___]] :=
  Cell[data /. {
     c_Cell :> cellDataConvert[c],
     StyleBox[a_, s_String, b___] :>
      RuleCondition[StyleBox[a, styleConvert[s], b], True]
     },
    styleConvert[style], 
   o
   ];
cellDataConvert~SetAttributes~Listable;

Stitching it all together

I don't want to copy all of this code out here, since I don't think that'd be a good use of the space. Instead I'll just mention that all of the code can be found here.

We can use it through SimpleDocs too if we want. Here's how you pull the data from a function page (the same works for guides and tutorials too):

SimpleDocs@"WorkbenchData"["~/Desktop/FindEcoEvoEq.nb"] // Keys

{"Metadata", "Preamble", "Body", "RelatedTutorials", \
"RelatedDemonstrations", "RelatedLinks", "RelatedFunctions", \
"RelatedGuides", "Examples"}

And here's how you then reconvert that into a notebook that SimpleDocs can process:

SimpleDocs@"FromWorkbench"[
   "~/Desktop/FindEcoEvoEq.nb",
   "Directory" -> "~/Desktop/EcoEvo"
   ];
"~/Desktop/EcoEvo/project/docs/content/ref/FindEcoEvoEq.nb" // 
  ExpandFileName // SystemOpen

enter image description here

$\endgroup$
  • $\begingroup$ Thanks, I'm just having time to give this a try. When I run SimpleDocs@"FromWorkbench", it seems to work but beeps. Why the Beep?... says The shared stylesheet "Wolfram/FunctionPageStyles.nb" could not be found. The default stylesheet will be used instead. Any ideas on how to fix that? $\endgroup$ – Chris K Jun 17 '19 at 9:19
  • $\begingroup$ @ChrisK that's because I need to open the Workbench notebook to use some FE programming tricks on it and when I do so it complains about the stylesheet. I could add a little hack that forces it to open with the Default styles... $\endgroup$ – b3m2a1 Jun 18 '19 at 5:45
  • $\begingroup$ Don't worry about it. I get the same beep & message with Workbench, so I think my problem isn't related to your package. $\endgroup$ – Chris K Jun 18 '19 at 7:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.