The Documentation Center was completely changed in version 6.0 to a search-based system from a table of contents based system. However, the old version 5 style Help Browser was still functional. It did not contain the Mathematica documentation, but it could be used to read the documentation of certain packages.
Exploring Abstract Algebra uses this very old help browser system.
Support for this old help browser was removed in version 11.0, as you can see e.g. here. Thus the documentation of this old package is no longer readable. The package needs a compatibility update.
The good news is that you can still read the documentation notebooks as they are. What is missing is the table of contents, and partitioning the notebooks into small sections.
Update:
It seems that the documentation structure is contained in the file BrowserCategories.m. I tried to write a very rudimentary program that will create a table of contents from it. There are very likely multiple issues with it, so you will want to improve it. But I think it will be a useful starting point.
base = FileNameJoin[{$UserBaseDirectory, "Applications", "AbstractAlgebra", "Documentation", "English"}]
Block[{BrowserCategory, Item, Delimiter},
BrowserCategory[name_, _, contents_] := OpenerView[{name, Column[contents]}, True];
Options[Item] = {CopyTag -> None};
Item[name_, nb_, opt : OptionsPattern[]] :=
If[OptionValue[CopyTag] === None,
Hyperlink[name, FileNameJoin[{base, nb}]],
Hyperlink[name, {FileNameJoin[{base, nb}], OptionValue[CopyTag]}]
];
Delimiter := Graphics[{Line[{{0, 0}, {1, 0}}]}, AspectRatio -> 1/20];
Get@FileNameJoin[{base, "BrowserCategories.m"}]
]
