#tl;dr
The Workbench packages are out-of-date for 11.1 and the whole system is not configured to easily work from Mathematica using a structure differing from the standard `/language/ref` URL structure or things like that.
It's a massive pain to make it all work, but in the end we can get half-way decent, largely-functional documentation pages like [this](https://www.wolframcloud.com/objects/b3m2a1.paclets/reference/BTools/ref/GenerateHTMLDocumentation.html)
---
###Shameless Self-Promotion:
I added this system + auto-generation into my [DocGen package](https://github.com/b3m2a1/mathematica-BTools/blob/master/Packages/DocGen.m) letting me build out docs like [this PacletManager documentation](https://www.wolframcloud.com/objects/b3m2a1.paclets/reference/PacletManager/guide/PacletManager.html) using just the following:
(* Build out the docs from the contexts*)
GenerateDocumentation[
{
"PacletTools`",
"PacletManager`",
"DataPaclets`"
},
"Install" -> True
]
(* Generate HTML and web deploy from paclets *)
GenerateHTMLDocumentation[
PacletFind[{"*Paclet*", "0"}],
CloudDeploy -> True
]
---
#Full Answer
To automate the entire process takes many tedious steps, but if you don't mind doing things by hand and follow Frank Martin's guide [here](https://mathematica.stackexchange.com/questions/128597/workbench-2-and-3-documentation-html-build-fails-output-file-test-failed) and don't mind doing things by hand you only really need the first two steps.
### *Copy the packages from Eclipse*
Install Workbench on Eclipse, as described [here](http://support.wolfram.com/kb/27221).
Then find the directory where the documentation packages are stored:
$eclipsePath = "path/to/Eclipse";
FileNames["DocumentationBuild", $eclipsePath, \[Infinity]]
Copy `"DocumentationTools"`, `"DocumentationBuild"` and `"Transmogrify"` to `FileNameJoin@{$UserBaseDirectory, "Applications"}`, then delete the `MenuSetup` that's in `"DocumentationTools"` (just for safety)
### *11.1 to 11.0 Down Conversion*
The heart of my initial problem was that the build tools I copied over from Workbench were out of date, so I got around that by down-converting my notebooks to 11.0 style before passing them through `ExportWebPage`.
Here's the code I used for that:
webExportNotebookPrep[nb_] :=
DeleteCases[
Visible -> False
]@
ReplaceRepeated[
nb, {
Cell[TextData[{c_, ___}], "ObjectNameGrid", ___] :>
c,
Cell[_, "NotesThumbnails" | "ObjectNameTranslation", ___] :>
Sequence @@ {},
Cell[TextData[{Cell[BoxData[_DynamicBox], ___], e__}],
t : "NotesSection" | "PrimaryExamplesSection" |
"ExampleSection" | "ExampleSubsection", r___] :>
Cell[TextData[e], t, r],
DynamicBox[e : If[$VersionNumber < _, __], ___] :>
e,
Cell[TextData[s_String], r___] :> Cell[s, r],
t : TemplateBox[{_, _String}, ___] :>
refLinkRewind[t],
TagBox[a : ActionMenuBox[_FrameBox, ___],
MouseAppearanceTag["LinkHand"]] :>
a,
Cell[
TextData[{t_, " ",
Cell[
BoxData[
GraphicsBox[{{
{
_GrayLevel, _Thickness,
LineBox[{{-1.8, 0.5}, {0, 0}, {1.8, 0.5}}]
}
}},
___
]
],
___
]
}],
___
] :>
InterpretationBox[
Cell[
TextData[{
t,
StyleBox[" \[FilledDownTriangle]", "AnchorBarArrow",
StripOnInput -> False]
}]],
TextCell[
Row[{"See Also",
Style[" \[FilledDownTriangle]", "AnchorBarArrow"]}]
]
]
}
]
With those corrections we can get out raw HTML, but there are still many issue to correct, which are largely documented by Frank Martin, which I dealt with programmatically.
---
### *File Post-Processing*
Some resources we want to link do built for a specific directory layout, corrupted or inadequate.
We'll write a general purpose function for handling this post-processing.
webExportPostProcessSingleReplacements // Clear;
webExportPostProcessMultiReplacements // Clear;
(* MultiReplacements *)
webExportPostProcessMultiReplacements[strip_, res_, url_] :=
With[{
trimmedRes =
StringTrim[res, "/" ~~ EndOfString],
paddedRes =
StringTrim[res, "/" ~~ EndOfString] <> "/"
},
Flatten@{(*
(* Insert prototype into loaded JS characters *)
"<script src='javascript/common.js' type='text/javascript'></script>\n"<>
"<script src='javascript/jquery.js' \
type='text/javascript'></script>\n"->
"<script src='javascript/jquery.js' type='text/javascript'></script>\n"<>
"<script src='javascript/common.js' \
type='text/javascript'></script>\n",*)
(* Relink loaded JS and CSS *)
h : ("src" | "href") ~~ "=" ~~ (q : "'" | "\"") ~~
t : "javascript" | "css" ~~ "/" :>
h <> "=" <> q <> URLBuild@{res, t} <> "/",
h : ("src" | "href") ~~ "=" ~~ (q : "'" | "\"") ~~ "/" ~~
t : "javascript" | "css" ~~ "/" :>
h <> "=" <> q <> URLBuild@{res, t} <> "/",
"/webMathematica/Resources/Documentation/English/" ->
paddedRes,
(* Relink loaded JS and CSS *)
l : ("<link rel=\"stylesheet\" href=\"" ~~
(paddedRes | "") ~~
"css/reference.css\" />") :>
l <> "\n<link rel=\"stylesheet\" href=\"" <>
URLBuild@{trimmedRes, "css", "clipboard.css"} <> "\" />",
l : ("<script src=\"" ~~ (paddedRes | "") ~~
"javascript/reference.js\"></script>") :>
"<script src=\"" <>
URLBuild@{trimmedRes, "javascript",
"jquery", "core", "1.7.2", "jquery.min.js"} <> "\" /></script>\n" <> l
}
];
(* SingleReplacements *)
webExportPostProcessSingleReplacements[strip_, res_, url_] :=
With[{
trimmedRes =
StringTrim[res, "/" ~~ EndOfString],
paddedRes =
StringTrim[res, "/" ~~ EndOfString] <> "/"
},
Flatten@{
Replace[strip, Except[_List | _Rule] -> {}],
(* Protect true links *)
"http://reference.wolfram.com/language/" ~~
t : "ref" | "guide" | "tutorial" :>
"http://reference.wolfram.com/language/" <> t,
paddedRes ~~
t : "ref" | "guide" | "tutorial" :>
"http://reference.wolfram.com/language/" <> t,
"/language/" ~~
t : "ref" | "guide" | "tutorial" :>
URLBuild@{
"http://reference.wolfram.com/language",
t
},
(* Patch intra site links *)
"/language/" ~~ param : Except[WhitespaceCharacter] ... ~~
t : "ref" | "guide" | "tutorial" ~~
doc : Except[WhitespaceCharacter | "\""] ... :>
URLBuild@
DeleteCases[{url, StringTrim[param, "/"], t, StringTrim[doc, "/"]}, ""],
"http://reference.wolfram.com/language" ->
url,
(* Patch resource directory *)
"/common/" ->
paddedRes,
(* Patch jquery version *)
"jquery/core/1.6.1" ->
"jquery/core/1.7.2",
(* Patch mathematicaImages to work a resource base *)
"\"../" ~~ i : "images/mathematicaImages" :>
"\"" <> URLBuild@{trimmedRes, i},
"\"" ~~ i : "images/mathematicaImages" :>
"\"" <> URLBuild@{trimmedRes, i},
"\"/mathematicaImages" ->
"\"" <> URLBuild@{trimmedRes, "images", "mathematicaImages"},
(* Patch includes to work a resource base *)
"\"/includes" ->
"\"" <> URLBuild@{trimmedRes, "includes"},
(* Downconvert 2013 resources to the standard resoruce base *)
"/2013/" ->
paddedRes,
(* Delete duplicates *)
(l :
"<script src=\"" <>
URLBuild@{trimmedRes, "javascript", "jquery.min.js"} <>
"\" /></script>\n") .. :>
l,
(l : "<link rel=\"stylesheet\" href=\"" <>
URLBuild@{trimmedRes, "css", "clipboard.css"} <> "\" />\n") .. :>
l
}
];
(* Extension specific replacements *)
webExportPostProcessSingleReplacements["css", strip_, res_, url_] :=
{
(* Add clipboard styles into reference.css *)
t : "/*##############################\n\tCOLORS" :>
"/* clipboard styles */\n@import url('clipboard.css');\n" <> t
};
webExportPostProcessSingleReplacements["js", strip_, res_, url_] :=
{
(* Clean up dangerous characters while protecting whitespace *)
w : WhitespaceCharacter :> w,
_?(Not@*PrintableASCIIQ) -> ""
};
webExportPostProcessMultiReplacements[_, _, _, _] :=
{
};
webExportPostProcessSingleReplacements[_, _, _, _] :=
{
};
(* Main *)
webExportPostProcess // Clear
Options[webExportPostProcess] = {
"ResourceBase" -> Automatic,
"URLBase" -> Automatic,
"StripExtensions" -> Automatic
};
webExportPostProcess[file_String?FileExistsQ, ops : OptionsPattern[]] :=
With[{
s = ReadString[file],
ext = Replace[OptionValue["StripExtensions"],
Automatic -> {".en." -> "."}],
res =
Replace[OptionValue["ResourceBase"],
Automatic :>
URLBuild@
ConstantArray["..",
With[{split = FileNameSplit@DirectoryName@file},
SelectFirst[Range[Length@split],
DirectoryQ@FileNameJoin@
Append[
Drop[split, -#],
"javascript"
] &,
1
]
]
]
],
url =
Replace[OptionValue["URLBase"],
Automatic :>
URLBuild@
ConstantArray["..",
With[{split = FileNameSplit@DirectoryName@file},
SelectFirst[Range[Length@split],
DirectoryQ@FileNameJoin@
Append[
Drop[split, -#],
"javascript"
] &,
1
]
]
]
]
},
If[StringQ[s],
DeleteFile[file];
Export[
StringReplace[file,
Replace[OptionValue["StripExtensions"], Except[_List | _Rule] -> {}]
],
FixedPoint[#, s, 6] &@
StringReplace@Join[
webExportPostProcessMultiReplacements[
ext,
res,
url
],
webExportPostProcessMultiReplacements[
FileExtension[file],
ext,
res,
url
]
] //
StringReplace@Join[
webExportPostProcessSingleReplacements[
ext,
res,
url
],
webExportPostProcessSingleReplacements[
FileExtension[file],
ext,
res,
url
]
]
,
"Text"
],
file
]
];
webExportPostProcess[___] := $Failed
---
### *Resource copying*
Then we'll take jkuczm's resource list to copy from [this answer](https://mathematica.stackexchange.com/questions/131104/how-can-i-build-documentation-using-docbuild-xml-outside-of-the-workbench/131569#131569) and copy each file to the documentation directory with requisite post-processing:
$webExportAssets :=
Map[
<|
"Drop" -> FileNameDepth[#[[1]]] - FileNameDepth@#[[2]],
"Files" -> FileNames[#[[3]], #[[1]], \[Infinity]],
"Insert" -> If[Length@# > 3, #[[4]], Nothing]
|> &,
{
{
FileNameJoin@{
DocumentationBuild`Common`$DocumentationBuildDirectory,
"Internal", "web", "html", "javascript"
},
"javascript",
{
"*.js"
}
},
{
FileNameJoin@{
DocumentationBuild`Common`$DocumentationBuildDirectory,
"Internal", "web", "html", "css"
},
"css",
{
"*.css"
}
},
{
FileNameJoin@{
DocumentationBuild`Common`$DocumentationBuildDirectory,
"Internal", "web", "html", "2014", "standard"
},
"",
{
"*.js", "*.json", "*.css", "*.png", "*.gif", "*.jpg",
"*.svg", "*.eot", "*.otf", "*.ttf", "*.woff", "*.html"
}
},
{
FileNameJoin@{
DocumentationBuild`Common`$DocumentationBuildDirectory,
"Internal", "web", "html", "2014", "standard"
},
"",
{
"*.ttf", "*.woff"
},
"fonts"
},
{
FileNameJoin@{
DocumentationBuild`Common`$DocumentationBuildDirectory,
"Internal", "web", "html", "2014",
"minimal",
"javascript"
},
"javascript",
"faster-page-load.js"
},
{
FileNameJoin@{
DocumentationBuild`Common`$DocumentationBuildDirectory,
"Internal", "web", "html", "images",
"mathematicaImages"
},
FileNameJoin@{"images", "mathematicaImages"},
"bullet.gif"
}
}
];
webExportAssetsCopy[dir : (_String | _File)?DirectoryQ] :=
Block[{d, f},
Monitor[
Table[
With[{depth = a["Drop"], insert = a["Insert"]},
Table[
d =
FileNameJoin@
Flatten@{dir,
DirectoryName@FileNameDrop[f, depth],
insert
};
If[! DirectoryQ@d,
CreateDirectory[d,
CreateIntermediateDirectories -> True
]
];
If[StringMatchQ[FileExtension[#], "css" | "js" | "html"],
webExportPostProcess[#],
#
] &@
CopyFile[f,
FileNameJoin@
Flatten@{
d,
FileNameTake@f
},
OverwriteTarget -> True
],
{f, a["Files"]}
]
],
{a, $webExportAssets}
],
Internal`LoadingPanel[
"Copying `` to ``"~TemplateApply~
{
f,
FileNameJoin@
Flatten@{
d,
FileNameTake@f
}
}
]
]
];
---
### *HTML Generation*
With all of that in place we can now generate pages that will look right:
webExportNotebook[file_, nb_, ops___?OptionQ] :=
With[{params = Flatten@{ops}},
Needs["DocumentationBuild`"];
Block[{
DocumentationBuild`Common`PubsEvaluateWithFE =
Identity,
DocumentationBuild`Common`Private`PubsStartFrontEnd =
Identity,
Print =
Identity,
NotebookOpen,
NotebookPut,
expFile =
If[DirectoryQ@file,
FileNameJoin@Flatten@{
file,
URLParse[
Lookup[params, "URI",
URLBuild@{
"",
"DocPage-" <> ToString[RandomInteger[10000]]
}
],
"Path"]
} <> ".html",
file
]
},
NotebookOpen[e___] :=
MathLink`CallFrontEnd[
FrontEnd`NotebookOpenReturnObject[e, Visible -> False]
];
NotebookPut[e_] :=
MathLink`CallFrontEnd[
FrontEnd`NotebookPutReturnObject[Append[e, Visible -> False]]
];
Quiet[
DocumentationBuild`Export`ExportWebPage[
ExpandFileName[expFile],
webExportNotebookPrep@nb,
Normal@params,
Sequence @@
FilterRules[
Flatten@{
ops,
"CompleteHTMLQ" -> True
},
FilterRules[
Options[DocumentationBuild`Export`ExportWebPage],
Except["HistoryData" | "Language"]
]
]
]
]
]
]
Where we'll feed the result of gathering the notebook metadata parameters in as `ops`:
$webExportParameters =
{
"title" -> "Title",
"titlemodifier" -> "TitleModifier",
"history" -> "HistoryData",
"type" -> "EntityType",
"paclet" -> "PacletName",
"context" -> "Context",
"uri" -> "URI",
"keywords" -> "Keywords",
"tutorialcollectionlinks" -> "TutorialCollectionLinks",
"summary" -> "Summary",
"index" -> "IndexQ"
};
webExportGatherParameters[nb_, ops___?OptionQ] :=
Normal@
KeyMap[
Capitalize@*
Replace[$webExportParameters],
Association@
FilterRules[
Flatten@{
ops,
Fold[Lookup,
Options[nb, TaggingRules], {TaggingRules, "Metadata"}]
},
Alternatives @@
Flatten[Through@{Keys, Values}[$webExportParameters]]
]
];
Note that the export process also blocks / changes a lot of things that Workbench uses that we don't need, e.g opens the notebook in hidden form, prevents a new front-end from being configured, and has the `Transmogrify` process work in quieted mode.
---
### *Composite function*
We can then build all this into a single function:
Options[GenerateHTMLDocumentation] =
Join[
Options[GenerateSymbolPages],
Options[webExportPostProcess],
Options[DocumentationBuild`Export`ExportWebPage],
Options[webExportAssetsDeploy], {
CloudConnect -> Automatic,
CloudDeploy -> False,
"DeployAssets" -> False,
"CopyAssets" -> Automatic
}];
GenerateHTMLDocumentation[dir_String?DirectoryQ,
nb : {__Notebook} | None,
ops : OptionsPattern[]
] :=
With[{
deploy = OptionValue[CloudDeploy],
highlight =
CurrentValue[$FrontEndSession,
{AutoStyleOptions, "HighlightUndefinedSymbols"}
]
},
If[
TrueQ@OptionValue["CopyAssets"] ||
And[
OptionValue["CopyAssets"] === Automatic ||
TrueQ@OptionValue["DeployAssets"],
AnyTrue[{"css", "fonts", "images", "includes", "javascript"},
Not@DirectoryQ[FileNameJoin@{dir, #}] &
]
],
webExportAssetsCopy[dir]
];
CheckAbort[
Function[
CurrentValue[$FrontEndSession,
{AutoStyleOptions, "HighlightUndefinedSymbols"}
] =
highlight;
If[deploy,
Replace[
Replace[OptionValue[CloudConnect],
Automatic -> Key["PacletsAccount"]
], {
s_String :>
If[$WolframID =!= s,
CloudConnect[s]
],
{s__String} :>
CloudConnect[s],
k_Key :>
KeyChainConnect[k]
}]
];
If[deploy && OptionValue["DeployAssets"],
Replace[
Replace[OptionValue["ResourceBase"], {
Automatic -> "reference",
CloudObject[c_, ___] :> c
}],
s_String :>
webExportAssetsDeploy[dir, s,
FilterRules[
{
ops
},
Options@webExportAssetsDeploy
]
]
]
];
If[deploy,
Replace[
Replace[OptionValue["URLBase"], {
Automatic -> "reference",
CloudObject[c_, ___] :> c
}],
s_String :>
If[# =!= None,
webExportCloudDeploy[dir, #, s,
FilterRules[
{
ops
},
Options@webExportCloudDeploy
]
]
]
],
#
]
]@
If[nb === None,
None,
If[Length@nb > 1,
Monitor[ReleaseHold[#],
Internal`LoadingPanel[
"Generating HTML page `` of ``"
~TemplateApply~
{i, Length@nb}
]
],
ReleaseHold@#
] &@
Hold@
Table[
With[{
params = webExportGatherParameters[nb[[i]], ops]
},
webExportPostProcess[
webExportNotebook[dir, nb[[i]], params],
FilterRules[
Flatten@{
If[deploy,
"URLBase" ->
Replace[OptionValue["URLBase"], {
Automatic :>
First@CloudObject["reference"],
CloudObject[u_, ___] :>
u,
s_String :>
First@CloudObject[s]
}],
Nothing
],
If[deploy,
"ResourceBase" ->
Replace[OptionValue["ResourceBase"], {
Automatic :>
First@CloudObject["reference"],
CloudObject[u_, ___] :>
u,
s_String :>
First@CloudObject[s]
}],
Nothing
],
params
},
Options[webExportPostProcess]
]
]
],
{i, Length@nb}
]
],
CurrentValue[$FrontEndSession,
{AutoStyleOptions, "HighlightUndefinedSymbols"}
] =
highlight;
$Aborted
]
];
Then we merge it with the automatic documentation generation process I wrote up [here](https://mathematica.stackexchange.com/questions/146670/automating-documentation-generation/146671#146671):
GenerateHTMLDocumentation[
dir : (_String | _File)?
DirectoryQ | _FileName?(DirectoryQ@*ToFileName),
pattern : _String?(Not@*FileExistsQ) | _Symbol,
ops___?OptionQ
] :=
With[{n = Names[pattern]},
If[Length@n > 1,
$Failed,
With[{
nb =
First[{NotebookGet[#], NotebookClose[#]}] &@
First@Flatten@List@
GenerateSymbolPages[
Evaluate@First@n,
Evaluate@FilterRules[{ops}, Options@GenerateSymbolPages],
Visible -> False
]
},
GenerateHTMLDocumentation[
Replace[dir, f_FileName :> ToFileName[f]],
nb,
ops
]
]
]
];
And then let it generate the docs without specifying a directory:
$webDocsHTMLBuildDirectory =
FileNameJoin@{$TemporaryDirectory, "web_docs"};
GenerateHTMLDocumentation[
Optional[Automatic, Automatic],
s___
] :=
With[{dir =
Quiet[
DeleteDirectory[$webDocsHTMLBuildDirectory,
DeleteContents -> True];
CreateDirectory[$webDocsHTMLBuildDirectory]
]},
GenerateHTMLDocumentation[
dir,
s
]
]
---
### *Viewing / Deploy the Docs*
Some things in the docs require HTTP, so to view the docs we make we'll want a function to start a server in a directory. It doesn't look like ``HTTPHandling`StartWebServer`` does that so we'll need something else. I chose to use python's [SimpleHTTPServer](https://docs.python.org/2/library/simplehttpserver.html)
I put some sugar on it in my [PyTools package](https://github.com/b3m2a1/mathematica-BTools/blob/master/Packages/PyTools.m). Look for `PySimpleServerStart` and `PySimpleServerOpen`.
But long-term having local docs is rather useless so I chose to put mine in the cloud, using these two functions to copy the resources and files up:
Options[webExportAssetsDeploy] =
DeleteDuplicatesBy[First]@
Join[
{
Permissions -> "Public"
},
Options[CloudObject]
];
webExportAssetsDeploy[
dir_String?DirectoryQ,
uri_String,
ops : OptionsPattern[]
] :=
With[{
files =
Select[
FileNames[
"css" | "fonts" | "images" | "includes" |
"javascript" ~~
$PathnameSeparator <> "*",
dir,
\[Infinity]],
Not@*DirectoryQ
]
},
Block[{f},
Monitor[
Table[
CopyFile[
f,
CloudObject[
URLBuild@
Flatten@{uri,
FileNameSplit@FileNameDrop[f, FileNameDepth@dir]},
Sequence @@ FilterRules[Flatten@{
ops,
Options[webExportAssetsDeploy]
},
Options[CloudObject]
]
]
],
{f, files}
],
Internal`LoadingPanel[
"Copying `` to ``"~TemplateApply~
{
f,
URLBuild@Flatten@{uri,
FileNameSplit@FileNameDrop[f, FileNameDepth@dir]}
}
]
]
]
]
Options[webExportCloudDeploy] =
DeleteDuplicatesBy[First]@
Join[
{
Permissions -> "Public"
},
Options[CloudObject]
];
webExportCloudDeploy[
dir_String?DirectoryQ,
pages : {__String?FileExistsQ},
uri_String,
ops : OptionsPattern[]
] :=
With[{
files =
Join[
pages,
Select[
FileNames[
"Files" <> $PathnameSeparator ~~
Alternatives @@
Map[FileBaseName, pages] ~~
$PathnameSeparator <> "*",
DirectoryName@First@pages,
\[Infinity]],
Not@*DirectoryQ
]
]
},
Block[{f},
Monitor[
Table[
Take[#, 1] &@
CopyFile[
f,
CloudObject[
URLBuild@
Flatten@{uri,
FileNameSplit@FileNameDrop[f, FileNameDepth@dir]},
Sequence @@ FilterRules[
Flatten@{
ops,
Options[webExportCloudDeploy]
},
Options[CloudObject]
]
]
],
{f, files}
],
Internal`LoadingPanel[
"Copying `` to ``"~TemplateApply~
{
f,
URLBuild@Flatten@{uri,
FileNameSplit@FileNameDrop[f, FileNameDepth@dir]}
}
]
]
]
]
(Note that they're already included in `GenerateHTMLDocumentation`)
---
### *End Product*
After all of that we have docs that are (mostly) functional and in the cloud. Here's an example of what this is good for:
In[60]:= GenerateHTMLDocumentation[
GenerateHTMLDocumentation,
CloudDeploy -> True,
"CopyAssets" -> False
]
Out[60]= CloudObject["https://www.wolframcloud.com/objects/b3m2a1.docs/reference/BTools/ref/GenerateHTMLDocumentation.html"]
If we then go to https://www.wolframcloud.com/objects/b3m2a1.paclets/docs/BTools/ref/GenerateHTMLDocumentation.html we get a mostly functional 11.0 style documentation page:
[![docs][1]][1]
[1]: https://i.sstatic.net/PkDYP.png