18
$\begingroup$

I have a package that uses custom documentation pages that are not generated with Wolfram Workbench.

How can I create search indices for these pages?

Requirements:

I expect an answer which shows the simplest possible package that fits the requirements: a single barebones documentation page, and straightforward instructions for creating the indices.

The obligatory "what have you tried?"

The following is a small example package that is searchable in M11.1 but not in M11.2.

Create the TestApp directory in $UserBaseDirectory/Applications, and add the following PacletInfo.m file:

Paclet[
    Name -> "TestApp",
    Version -> "0.0.1",
    MathematicaVersion -> "10.0+",
    Extensions -> {{"Documentation", Language -> "English", MainPage -> "TestApp"}}
]

Then create the Documentation/English subdirectory within TestApp and add the following notebook:

Notebook[{

Cell[CellGroupData[{
Cell["Lorem Ipsum", \
"Section",ExpressionUUID->"cbc045dc-447e-4711-8aac-cb341b739ef9"],

Cell["\<\
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur non \
volutpat lectus. Orci varius natoque penatibus et magnis dis parturient \
montes, nascetur ridiculus mus. Class aptent taciti sociosqu ad litora \
torquent per conubia nostra, per inceptos himenaeos. Lorem ipsum dolor sit \
amet, consectetur adipiscing elit. Aliquam eget efficitur quam. Sed a aliquam \
libero, non porta erat. Mauris facilisis velit felis, hendrerit consequat \
massa posuere ac. Sed luctus suscipit dictum. Aliquam egestas orci vitae \
turpis ultricies sollicitudin. Etiam vitae nisi lobortis, scelerisque nulla \
ut, euismod lorem. Sed non nulla ex. In vel eros venenatis, mollis tortor ac, \
convallis nibh. Aenean vestibulum dui dolor, sed ultricies lacus dignissim \
nec. Aenean iaculis sapien et tincidunt porttitor. Nulla pulvinar est dapibus \
posuere convallis.\
\>", "Text",ExpressionUUID->"a609adb0-6225-4cdd-9894-e399706c82ac"]
}, Open  ]]
},
TaggingRules -> {
    "Metadata" -> {
        "keywords" -> {"lipsum", "testapp", "foobarbaz"},
        "index" -> True,
        "label" -> "TestApp page",
        "language" -> "en",
        "paclet" -> "TestApp",
        "summary" -> "This is a doc page test.",
        "title" -> "TestApp Title",
        "windowtitle" -> "TestApp",
        "type" -> "Guide",
        "uri" -> "TestApp/TestApp"
    }
},
Saveable->False,
DockedCells -> FEPrivate`FrontEndResource["FEExpressions", "HelpViewerToolbar"]
TrackCellChangeTimes->False,
StyleDefinitions->"Default.nb"
]

Some "Metadata" must be present in the TaggingRules of the notebook, but it is not clear to me how much is required (other than "index" -> True). Without it, indexing will fail.

Now evaluate in M11.1 or earlier:

SetDirectory@FileNameJoin[{$UserBaseDirectory, "Applications", "TestApp", "Documentation", "English"}]
Needs["DocumentationSearch`"]
indexDir = CreateDirectory["Index"]
ind = NewDocumentationNotebookIndexer[indexDir]
AddDocumentationNotebook[ind, "TestApp.nb"];
CloseDocumentationNotebookIndexer[ind]

Finally, restart Mathematica (including the front end), open the documentation centre, and search for "lipsum" (a keyword) or "consequat" (a word in the body of the notebook). These queries will all return the TestApp page.

This works in M11.1 or earlier, but it does not work in M11.2. How can I fix this test package to make sure that it will work in M11.2, and that M11.2 can automatically convert the indices?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ In fact I have this working for up to 11.1, but 11.2 isn't able to update the indexes for some reasons. I kept this question short and simple because I am hoping to have a straightforward and simple QA that shows how to do this properly, and that people can refer to without needing to read several other QAs (many of which may provide outdated or irrelevant). $\endgroup$
    – Szabolcs
    Sep 17, 2017 at 10:24
  • $\begingroup$ Also asked on Wolfram Community. $\endgroup$
    – Szabolcs
    Sep 17, 2017 at 13:33
  • $\begingroup$ I gather that you are seeing the 11.2 popup, but it is not creating a new index. Have you checked what directory it is trying to update? Does your user have write permissions to that directory? Is there any evidence of any writes in the directory while the popup is open? $\endgroup$
    – Itai Seggev
    Sep 17, 2017 at 21:53
  • $\begingroup$ @Itai Yes, I see the popup (after every restart of M and trying to search). "Have you checked what directory it is trying to update?" How can I check that? There's no TextSearchIndex directory appearing anywhere. I have full write permissions. The popup goes away in a split second, so I'm not sure how to check if anything happens during that time. I assume that the way this package (or its index) is structured is somehow not compatible with the new text search. But I couldn't figure out what exactly is wrong. Can you not reproduce the problem using the minimal example above? $\endgroup$
    – Szabolcs
    Sep 17, 2017 at 22:43
  • 1
    $\begingroup$ @Itai I have looked there already, but only the documentation directories are logged there (i.e. {"$UserBaseDirectory/TestApp/Documentation/English"}, each time there is an attempt to convert indices. Since the conversion fails and is re-tried on next startup, I have multiple entries there. Today I did manage to get this working. The solution is in the answer below. I am not 100% confident about this, so I would welcome a Wolfram developer's blessing. $\endgroup$
    – Szabolcs
    Sep 18, 2017 at 10:28

2 Answers 2

Reset to default
9
$\begingroup$

Below I show an example of a minimal package with custom documentation pages that are searchable both in 10.0–11.1 and 11.2.

This is not a definitive solution because it was arrived at by trial and error, and it may cease working in a future version again.

What needs to be changed in the above example?

  • Pages should not be in the root documentation directory (i.e. Documentation/English). They should reside in subdirectories, such as Guides, Tutorials, ReferencePages, etc. See the documentation of built-in packages for reference.

  • The metadata in the tagging rules must also have the "context" entry. Just to be safe, below I included all metadata entries that the built-in documentation pages have.

A working example

Create the TestApp directory within $UserBaseDirectory. Place PacletInfo.m there with the following contents:

Paclet[
    Name -> "TestApp",
    Version -> "0.0.1",
    MathematicaVersion -> "10.0+",
    Extensions -> {{"Documentation", Language -> "English", MainPage -> "Tutorials/TestApp"}}
]

Place TestApp.nb in Documentation/English/Tutorials within the TestApp directory. Other locations are possible too instead of Tutorials, though I have only tried standard ones, such as Guides. If you use a different location, be sure to adjust the metadata below, as well as MainPage within the "Documentation" extension of PacletInfo.m.

The contents of TestApp.nb:

Notebook[{

Cell[CellGroupData[{
Cell["Lorem Ipsum", \
"Section",ExpressionUUID->"cbc045dc-447e-4711-8aac-cb341b739ef9"],

Cell["\<\
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur non \
volutpat lectus. Orci varius natoque penatibus et magnis dis parturient \
montes, nascetur ridiculus mus. Class aptent taciti sociosqu ad litora \
torquent per conubia nostra, per inceptos himenaeos. Lorem ipsum dolor sit \
amet, consectetur adipiscing elit. Aliquam eget efficitur quam. Sed a aliquam \
libero, non porta erat. Mauris facilisis velit felis, hendrerit consequat \
massa posuere ac. Sed luctus suscipit dictum. Aliquam egestas orci vitae \
turpis ultricies sollicitudin. Etiam vitae nisi lobortis, scelerisque nulla \
ut, euismod lorem. Sed non nulla ex. In vel eros venenatis, mollis tortor ac, \
convallis nibh. Aenean vestibulum dui dolor, sed ultricies lacus dignissim \
nec. Aenean iaculis sapien et tincidunt porttitor. Nulla pulvinar est dapibus \
posuere convallis.\
\>", "Text",ExpressionUUID->"a609adb0-6225-4cdd-9894-e399706c82ac"]
}, Open  ]]
},
TaggingRules -> {
    "Metadata" -> {
        "built" -> {2017, 9, 18, 10, 0, 0},
        "history" -> {"XX", "", "", ""},
        "context" -> "TestApp`",
        "keywords" -> {"lipsum", "testapp", "foobarbaz"},
        "specialkeywords" -> {}, 
        "tutorialcollectionlinks" -> {},
        "index" -> True,
        "label" -> "TestApp Documentation",
        "language" -> "en",
        "paclet" -> "TestApp",
        "summary" -> "This is a doc page test.",
        "synonyms" -> {},
        "tabletags" -> {},
        "title" -> "TestApp Title",
        "titlemodifier" -> "",
        "windowtitle" -> "TestApp",
        "type" -> "Tutorial",
        "uri" -> "TestApp/tutorial/TestApp"
    }
},
Saveable->False,
WindowTitle -> "TestApp",
DockedCells -> FEPrivate`FrontEndResource["FEExpressions", "HelpViewerToolbar"]
TrackCellChangeTimes->False,
StyleDefinitions->"Default.nb"
]

Now in 11.1 or earlier, evaluate

SetDirectory@FileNameJoin[{$UserBaseDirectory, "Applications", "TestApp", "Documentation", "English"}]
Needs["DocumentationSearch`"]
indexDir = CreateDirectory["Index"]
ind = NewDocumentationNotebookIndexer[indexDir]
AddDocumentationDirectory[ind, "Tutorials"]
CloseDocumentationNotebookIndexer[ind]

spellIndexDir = CreateDirectory["SpellIndex"]
CreateSpellIndex[indexDir, spellIndexDir]

Now start up 11.2 and try searching for "lipsum" or "vestibulum". Both return the TestApp page.

Documentation search

Final notes

The following may work in 11.2 for creating new-style indexes directly (and thus avoiding the need to re-index):

Needs["DocumentationSearch`"]
CreateDocumentationIndex[FileNameJoin[{$UserBaseDirectory, "Applications", "TestApp", "Documentation", "English"}]]

Update and warning: If both 11.2 and 11.3 are installed, then creating new-style indices in 11.3 will cause documentation search not to work anymore in 11.2. Indices created with 11.2 will not cause problems in 11.3 though.

This QA discusses only how to get search working. There is more to creating a usable documentation page, such as using an appropriate stylesheet, controlling the editability of cells, etc. These are not discussed here.

Improve this answer
$\endgroup$
0
$\begingroup$

I tried out Szabolcs' suggestion in his/her "final notes".

I tested this code

Needs["DocumentationSearch`"]
CreateDocumentationIndex[FileNameJoin[{$UserBaseDirectory, "Applications", "TestApp", "Documentation", "English"}]]

in both 11.2 and 12.1. In both cases a search index was built and saved in the folder

...\Applications\TestApp\Documentation\English\TextSearchIndex

However, this index at this location is working for 11.2 but not for 12.1.

I found out that coping the content of the folder (built with 12.1) of TextSearchIndex to the location

...\Applications\TestApp\Documentation\English\SearchIndex\2

makes the index accessable also from the the documentation center of 12.1

Does this makes sense to someone of you?

Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ Don't do that (don't copy it to directories meant for different versions). There is a risk that you will completely or partially break documentation search for the users of your package. I am speaking from experience. I have seen the documentation become completely unsearchable (!!!), not just for the package, but all of it. 12.1 (as well as the few preceding versions) should be able to create the indices automatically, as needed $\endgroup$
    – Szabolcs
    Sep 3, 2020 at 14:21
  • $\begingroup$ I don't recall what exactly I did to break things. I might have moved indices generated with new versions to locations used by old versions. You are proposing the reverse. Maybe that won't cause problems. Do be extremely careful, and if you do not have the possibility to test this package in multiple versions, then I suggest not to risk breaking things. $\endgroup$
    – Szabolcs
    Sep 3, 2020 at 14:24
  • $\begingroup$ No, actually I called CreateDocumentationIndex from 12.1. But apparently the index is built at the wrong location. That location works for 11.2 (even if build in 12.1), but I needed to copy it to the other location to make it working also in 12.1. $\endgroup$
    – dnet
    Sep 3, 2020 at 14:46
  • $\begingroup$ If I want to use the index in both versions, I need the index to be at both locations. At least I do not know how else I can achieve this. $\endgroup$
    – dnet
    Sep 3, 2020 at 14:47
  • $\begingroup$ I edited my original answer to make clear that I copied around the version of the index which I build in 12.1. $\endgroup$
    – dnet
    Sep 3, 2020 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.