How to create search indices for custom documentation pages?

I have a package that uses custom documentation pages that are not generated with Wolfram Workbench.

How can I create search indices for these pages?

Requirements:

I expect an answer which shows the simplest possible package that fits the requirements: a single barebones documentation page, and straightforward instructions for creating the indices.

The obligatory "what have you tried?"

The following is a small example package that is searchable in M11.1 but not in M11.2.

Create the TestApp directory in $UserBaseDirectory/Applications, and add the following PacletInfo.m file: Paclet[ Name -> "TestApp", Version -> "0.0.1", MathematicaVersion -> "10.0+", Extensions -> {{"Documentation", Language -> "English", MainPage -> "TestApp"}} ]  Then create the Documentation/English subdirectory within TestApp and add the following notebook: Notebook[{ Cell[CellGroupData[{ Cell["Lorem Ipsum", \ "Section",ExpressionUUID->"cbc045dc-447e-4711-8aac-cb341b739ef9"], Cell["\<\ Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur non \ volutpat lectus. Orci varius natoque penatibus et magnis dis parturient \ montes, nascetur ridiculus mus. Class aptent taciti sociosqu ad litora \ torquent per conubia nostra, per inceptos himenaeos. Lorem ipsum dolor sit \ amet, consectetur adipiscing elit. Aliquam eget efficitur quam. Sed a aliquam \ libero, non porta erat. Mauris facilisis velit felis, hendrerit consequat \ massa posuere ac. Sed luctus suscipit dictum. Aliquam egestas orci vitae \ turpis ultricies sollicitudin. Etiam vitae nisi lobortis, scelerisque nulla \ ut, euismod lorem. Sed non nulla ex. In vel eros venenatis, mollis tortor ac, \ convallis nibh. Aenean vestibulum dui dolor, sed ultricies lacus dignissim \ nec. Aenean iaculis sapien et tincidunt porttitor. Nulla pulvinar est dapibus \ posuere convallis.\ \>", "Text",ExpressionUUID->"a609adb0-6225-4cdd-9894-e399706c82ac"] }, Open ]] }, TaggingRules -> { "Metadata" -> { "keywords" -> {"lipsum", "testapp", "foobarbaz"}, "index" -> True, "label" -> "TestApp page", "language" -> "en", "paclet" -> "TestApp", "summary" -> "This is a doc page test.", "title" -> "TestApp Title", "windowtitle" -> "TestApp", "type" -> "Guide", "uri" -> "TestApp/TestApp" } }, Saveable->False, DockedCells -> FEPrivateFrontEndResource["FEExpressions", "HelpViewerToolbar"] TrackCellChangeTimes->False, StyleDefinitions->"Default.nb" ]  Some "Metadata" must be present in the TaggingRules of the notebook, but it is not clear to me how much is required (other than "index" -> True). Without it, indexing will fail. Now evaluate in M11.1 or earlier: SetDirectory@FileNameJoin[{$UserBaseDirectory, "Applications", "TestApp", "Documentation", "English"}]
Needs["DocumentationSearch"]
indexDir = CreateDirectory["Index"]
ind = NewDocumentationNotebookIndexer[indexDir]
CloseDocumentationNotebookIndexer[ind]


Finally, restart Mathematica (including the front end), open the documentation centre, and search for "lipsum" (a keyword) or "consequat" (a word in the body of the notebook). These queries will all return the TestApp page.

This works in M11.1 or earlier, but it does not work in M11.2. How can I fix this test package to make sure that it will work in M11.2, and that M11.2 can automatically convert the indices?

• In fact I have this working for up to 11.1, but 11.2 isn't able to update the indexes for some reasons. I kept this question short and simple because I am hoping to have a straightforward and simple QA that shows how to do this properly, and that people can refer to without needing to read several other QAs (many of which may provide outdated or irrelevant). – Szabolcs Sep 17 '17 at 10:24
• Also asked on Wolfram Community. – Szabolcs Sep 17 '17 at 13:33
• I gather that you are seeing the 11.2 popup, but it is not creating a new index. Have you checked what directory it is trying to update? Does your user have write permissions to that directory? Is there any evidence of any writes in the directory while the popup is open? – Itai Seggev Sep 17 '17 at 21:53
• @Itai Yes, I see the popup (after every restart of M and trying to search). "Have you checked what directory it is trying to update?" How can I check that? There's no TextSearchIndex directory appearing anywhere. I have full write permissions. The popup goes away in a split second, so I'm not sure how to check if anything happens during that time. I assume that the way this package (or its index) is structured is somehow not compatible with the new text search. But I couldn't figure out what exactly is wrong. Can you not reproduce the problem using the minimal example above? – Szabolcs Sep 17 '17 at 22:43
• @Itai I have looked there already, but only the documentation directories are logged there (i.e. {"$UserBaseDirectory/TestApp/Documentation/English"}, each time there is an attempt to convert indices. Since the conversion fails and is re-tried on next startup, I have multiple entries there. Today I did manage to get this working. The solution is in the answer below. I am not 100% confident about this, so I would welcome a Wolfram developer's blessing. – Szabolcs Sep 18 '17 at 10:28 1 Answer Below I show an example of a minimal package with custom documentation pages that are searchable both in 10.0–11.1 and 11.2. This is not a definitive solution because it was arrived at by trial and error, and it may cease working in a future version again. What needs to be changed in the above example? • Pages should not be in the root documentation directory (i.e. Documentation/English). They should reside in subdirectories, such as Guides, Tutorials, ReferencePages, etc. See the documentation of built-in packages for reference. • The metadata in the tagging rules must also have the "context" entry. Just to be safe, below I included all metadata entries that the built-in documentation pages have. A working example Create the TestApp directory within $UserBaseDirectory. Place PacletInfo.m there with the following contents:

Paclet[
Name -> "TestApp",
Version -> "0.0.1",
MathematicaVersion -> "10.0+",
Extensions -> {{"Documentation", Language -> "English", MainPage -> "Tutorials/TestApp"}}
]


Place TestApp.nb in Documentation/English/Tutorials within the TestApp directory. Other locations are possible too instead of Tutorials, though I have only tried standard ones, such as Guides. If you use a different location, be sure to adjust the metadata below, as well as MainPage within the "Documentation" extension of PacletInfo.m.

The contents of TestApp.nb:

Notebook[{

Cell[CellGroupData[{
Cell["Lorem Ipsum", \
"Section",ExpressionUUID->"cbc045dc-447e-4711-8aac-cb341b739ef9"],

Cell["\<\
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur non \
volutpat lectus. Orci varius natoque penatibus et magnis dis parturient \
montes, nascetur ridiculus mus. Class aptent taciti sociosqu ad litora \
torquent per conubia nostra, per inceptos himenaeos. Lorem ipsum dolor sit \
amet, consectetur adipiscing elit. Aliquam eget efficitur quam. Sed a aliquam \
libero, non porta erat. Mauris facilisis velit felis, hendrerit consequat \
massa posuere ac. Sed luctus suscipit dictum. Aliquam egestas orci vitae \
turpis ultricies sollicitudin. Etiam vitae nisi lobortis, scelerisque nulla \
ut, euismod lorem. Sed non nulla ex. In vel eros venenatis, mollis tortor ac, \
convallis nibh. Aenean vestibulum dui dolor, sed ultricies lacus dignissim \
nec. Aenean iaculis sapien et tincidunt porttitor. Nulla pulvinar est dapibus \
posuere convallis.\
}, Open  ]]
},
TaggingRules -> {
"built" -> {2017, 9, 18, 10, 0, 0},
"history" -> {"XX", "", "", ""},
"context" -> "TestApp",
"keywords" -> {"lipsum", "testapp", "foobarbaz"},
"specialkeywords" -> {},
"index" -> True,
"label" -> "TestApp Documentation",
"language" -> "en",
"paclet" -> "TestApp",
"summary" -> "This is a doc page test.",
"synonyms" -> {},
"tabletags" -> {},
"title" -> "TestApp Title",
"titlemodifier" -> "",
"windowtitle" -> "TestApp",
"type" -> "Tutorial",
"uri" -> "TestApp/tutorial/TestApp"
}
},
Saveable->False,
WindowTitle -> "TestApp",
DockedCells -> FEPrivateFrontEndResource["FEExpressions", "HelpViewerToolbar"]
TrackCellChangeTimes->False,
StyleDefinitions->"Default.nb"
]


Now in 11.1 or earlier, evaluate

SetDirectory@FileNameJoin[{$UserBaseDirectory, "Applications", "TestApp", "Documentation", "English"}] Needs["DocumentationSearch"] indexDir = CreateDirectory["Index"] ind = NewDocumentationNotebookIndexer[indexDir] AddDocumentationDirectory[ind, "Tutorials"] CloseDocumentationNotebookIndexer[ind] spellIndexDir = CreateDirectory["SpellIndex"] CreateSpellIndex[indexDir, spellIndexDir]  Now start up 11.2 and try searching for "lipsum" or "vestibulum". Both return the TestApp page. Final notes The following may work in 11.2 for creating new-style indexes directly (and thus avoiding the need to re-index): Needs["DocumentationSearch"] CreateDocumentationIndex[FileNameJoin[{$UserBaseDirectory, "Applications", "TestApp", "Documentation", "English"}]]


Update and warning: If both 11.2 and 11.3 are installed, then creating new-style indices in 11.3 will cause documentation search not to work anymore in 11.2. Indices created with 11.2 will not cause problems in 11.3 though.

This QA discusses only how to get search working. There is more to creating a usable documentation page, such as using an appropriate stylesheet, controlling the editability of cells, etc. These are not discussed here.