Sub-packages and auto-completion

Question

Cross-posted on Wolfram Community

---

I have a Mathematica application that consists of two sub-packages, corresponding to the contexts foo` (foo.m) and foo`b` (b.m). Some of the definitions in foo.m rely on functions in b.m. Therefore b.m must be loaded before foo.m in some way (there are multiple ways to achieve this).

If I do this, auto-completion will be available for foo` symbols, but not for foo`b` symbols. Why? How can I enable auto-completion for both sub-packages? Actually, completion for foo`b` symbols works when requested explicitly with Command-K, but the completion window doesn't pop up automatically.

If the loading order is reversed, i.e. load foo` first, then foo`b`, then everything is fine. But I cannot do that because of the dependency structure.

---

Here's the package source, for convenience: https://www.dropbox.com/s/z6eoxumeqd42411/foo.zip?dl=0

The directory structure looks like this:

And here's the file contents:

(* init.m *)

Get["foo`b`"] (* Must be loaded first in some way because foo.m uses it. *)
Get["foo`foo`"]

(* foo.m *)

(* Normally, BeginPackage would refer to foo`b` in its second argument.
   Since in this basic example foo` doesn't actually use foo`b`, I removed
   this to demonstrate that the problem isn't caused by it. *)
BeginPackage["foo`" (* , {"foo`b`"} *)]

TheName::usage == "TheName[]";

Begin["`Private`"]

End[]

EndPackage[]

(* b.m *)

BeginPackage["foo`b`"]

TheOtherName::usage == "TheOtherName[]";

Begin["`Private`"]

End[]

EndPackage[]

If I load foo` (which in turn loads foo`b` as well, through init.m), TheOtherName is not offered for completion.

I need a solution where one sub-package can be dependent on the other, but auto-completion works for both.

---

If I change init.m to

Get["foo`foo`"]
Get["foo`b`"]

(i.e. I exchange the order of the two Get commands), then auto-completion works fine.

I do not want to do this because in my real use case, foo` depends on foo`b`, therefore foo.m would need to load b.m in some way. A second Get["foo`b`"] after this would cause double-loading of b.m, which is ugly and slow. These are large packages and the slowdown is actually noticeable on a Raspberry Pi.

If I use

Get["foo`b`"]
Get["foo`foo`"]

or

Get["foo`foo`"]
Needs["foo`b`"]

to prevent double-loading, then auto-completion is broken again.

I am hoping to find out what precisely in Get["foo`b`"] re-enables auto-completion and re-create that thing in isolation without actually re-evaluating all definitions from b.m.

Answer 1

So I can provide some clarification and a workaround

---

Clarification

First off, this only triggers on the EndPackage:

For example, this still allows ggg to autocomplete:

BeginPackage["foo`a`"];
ggg::usage = "asdasd";
EndPackage[];

(*Break this into different cells or the FE acts like EndPackage was called!*)
Abort[]
    
BeginPackage["foo`", {"foo`a`"}];

But the moment you call EndPackage[] ggg stops autocompleting.

Further investigation shows that even simply messing with the $ContextPath is enough to do it.

BeginPackage["foo`a`"];
ggg::usage = "asdasd";
EndPackage[];

(* again, split these into two cells *)

PrependTo[$ContextPath, "foo`"];

Next, this does not seem to have to do with the standard way the FE handles autocompletions (as I know it at least).

I say this because the following doesn't prevent it from auto-completing (working from a Quit kernel):

BeginPackage["foo`a`"];
ggg::usage = "asdasd";
EndPackage[];

FrontEndExecute@
 FrontEnd`UpdateKernelSymbolContexts[
  "foo`",
  Join[
   {"foo`", "foo`a`"},
   $ContextPath
   ],
  {{"foo`", {"aaa"}, {}, {"aaa"}, {}}}
  ]

And even more to the point, this isn't enough to preserve the autocompletion:

BeginPackage["foo`a`"];
ggg::usage = "asdasd";
EndPackage[];

(* Split the cell here *)

Internal`SymbolList[False]

(* And further split here *)

BeginPackage["foo`"]; EndPackage[]

(* Make sure to call Internal`SymbolList[True] after trying this *)

why this happens is something I haven't been able to figure out. This behavior is a lot like the behavior involved in developerFunctions.m. There doesn't seem to be anything in NeedCurrentFrontEndSymbolsPacket to explain it.

---

Workaround

Since this triggers on $ContextPath edits, you simply need to edit the $ContextPath again.

This will recover the auto-completion:

BeginPackage["foo`a`"];
EndPackage[];

So after you load your packages, you can use that to get your autocompletions back.