How to hide functions in a sub package from user level?

Question

The only packages I did before were simple one file packages.

I am now learning how to make a package that has a sub package. The main package calls functions in the subpackage. But user is not supposed to be able to call functions directly in the subpackage.

Here is an example below. I need to have the subpackage in separate file, so I can't sub package code in the private part of the main package, at least I do not know how to.

What Am I doing wrong in the following? Under

 C:\Users\me\AppData\Roaming\Mathematica\Applications

I have

Applications/
   mypkg/
      Kernel/
         init.m
      mypkg.m
      sub.m

init.m is

 Get ["mypkg`mypkg`"]
 Get ["mypkg`sub`"]

mypkg.m is

BeginPackage["mypkg`"]
main::usage="main[]"
Begin["`Private`"]
main[]:=Module[{},  
  mypkg`sub`foo[1]
]
End[] (* End Private Context *)
EndPackage[]

And sub.m is

BeginPackage["mypkg`sub`"]
Begin["`Private`"]
  foo[n_]:=Module[{},
     Print["in mypkg`sub`foo[]"];
     n+1
  ]
End[] (* End Private Context *)
EndPackage[]

But now in my notebook, I find I am able to call

 mypkg`sub`foo[]          

How is this possible? Since foo has no foo::usage entry to export it to outside and it is in the private context of sub?

<< mypkg`
mypkg`sub`foo[1]

I want only functions in mypkg.m to be able to call functions in sub.m. What to fix in the above code for this to happen?

I am finding working with packages with multiple files in Mathematica confusing so far, as it is not clear how to separate scope of things.

V12.

Answer 1

Mathematica does not have a concept of private symbols in the sense of most other programming languages. Given its full name, you can call any function, assuming it has been loaded. However, there is still a scoping mechanism available to us: Contexts and $ContextPath. In short, every symbol lives in a context (see Context). If a symbol without a explicit context is encountered, the following happens (see e.g. @Szabolcs' answer here):

• If the symbol exists in $Context, this one is taken
• All contexts on $ContextPath are searched in order (sub-contexts not included). If a symbol with the right name is found, this one is taken
• If the symbol is not in any of these contexts, a new symbol is defined in the current context (see $Context)

This means that we can hide symbols by not adding their context to $ContextPath. This is what is usually done with Begin["`Private`"]:

• BeginPackage["foo`"] sets $Context to "foo`", and $ContextPath to {"foo`","System`"}. Any symbol that is encountered after BeginPackage is therefore defined in foo` (assuming it's not already in System`).
• Begin[`"Private`"] sets $Context to foo`Private` but leaves $ContextPath alone. So any symbol that is already defined from before is taken, and new symbols are created in the foo`Private` context.
• End[] resets $Context
• EndPackage[] resets $Context and $ContextPath, and adds "foo`" to $ContextPath. This means that only symbols that have been encountered before the Begin["`Private`"] part are directly visible to the user.

Next, I will demonstrate how we can have a package and a sub-package, where the package has access to the "private" symbols of the sub-package. (Note that in most cases, simply putting internal stuff in a private context should be enough)

This is using the same package layout as shown in the question.

init.m: (if this is the only context of the init file, it can also be omitted)

Get["mypkg`mypkg`"]

mypkg.m:

BeginPackage["mypkg`"]
Needs["mypkg`sub`"] (* load the sub-packge. Remember that BeginPackage will 
  reset $ContextPath again, so it doesn't appear loaded from the outside *)
AppendTo[$ContextPath,"mypkg`sub`Private`"] (* make the private context visible as well*)
main::usage="main[]"
Begin["`Private`"]
main[]:=Module[{},  
  foo[1] (* since mypkg`sub`Private` is on $ContextPath, we don't need the full name here *)
]
End[] (* End Private Context *)
EndPackage[]

sub.m:

BeginPackage["mypkg`sub`"]
Begin["`Private`"]
  foo[n_]:=Module[{},
     Print["in mypkg`sub`Private`foo[]"]; (* this is the actual full name now *)
     n+1
  ]
End[] (* End Private Context *)
EndPackage[]

This is how it looks now:

<< mypkg`

main[]
(* in mypkg`sub`Private`foo[] *)
(* 2 *)

foo[1] (* foo is not visible *)
(* foo[1] *)

MemberQ[$ContextPath, "mypkg`sub`"] (* sub-packege is not loaded *)
(* False *)

<< mypkg`sub`

foo[1] (* foo is still not visible *)
(* foo[1] *)

MemberQ[$ContextPath, "mypkg`sub`"] (* sub-packege is now loaded *)
(* True *)

mypkg`sub`Private`foo[1] (* the full name will always work *)
(* in mypkg`sub`Private`foo[] *)
(* 2 *)

Answer 2

An answer to a similar question regarding package dependencies described the directory structures needed to implement sub-functions but here the details of the function structure are illustrated. To first backtrack; an ideal package framework should enjoy the following properties with function names inserted from this example but which apply more generally:

1. The end-user's name-space should not be polluted by foo.
2. The mypkg` package should "see" the unadulterated foo without needing a full context specification.
3. There should be no potential conflicts in mypkg`and mypkg`sub`.
4. To seamlessly scale, there should be no need for (multiple) Needs's calls given the introduction of a new "Developer's DeclarePackage" idiom.

The OP's implementation satisfies 1) The accepted answer satisfies 1) and 2); WL's "idiomatic method" (to be illustrated shortly) satisfies 1), 2) and 3); and finally, I will mention the syntax for an extended package framework which satisfies 1), 2), 3) & 4).

Explanations for the above claims:

1) An end-user's namespace is not polluted because an unadulterated foo (one without full context specification) is still available for use in the end-users environment (foo's full contextual name is mypkg`sub`Private`foo but mypkg`sub`Private` doesn't appear on $ContextPath).

2) In the OP's example, any appearance of the full context form mypkg`sub`foo means accessing its definition wherever it is placed but what may initially seem surprising here is that an apparently encapsulated definition has attached itself to an apparently external symbol. Consider however, the following snippet:

mypkg`sub`foo

BeginPackage["mypkg`sub`"];
Begin["`Private`"];
foo[n_] := Module[{}, Print["in mypkg`sub`foo[]"];
  n + 1]
End[]; (*End Private Context*)
EndPackage[];

mypkg`sub`foo[1]

One might have expected foo to be created (and hence encapsulated) as mypkg`sub`Private`foo with the final mypkg`sub`foo[1] remaining unevaluated but a previous creation of the mypkg`sub`foo symbol (anywhere) ensures that the internal foo now instead refers to mypkg`sub`foo and hence attaches its subsequent definition to this symbol. As noted, this is because mypkg`sub` remains on $ContextPath when the control flow passes through Begin["`Private`"] with new symbols only being created (and hence encapsulated) in mypkg`sub`Private` if they don't already appear in $ContextPath. But we are now, at least, in a position to see why the idiomatic practice works:

The idiomatic way to define this sub-package

BeginPackage["mypkg`sub`"];
usage::foo = "foo[n] adds one to n";
Begin["`Private`"];
foo[n_] := Module[{}, Print["in mypkg`sub`foo[]"];
  n + 1]
End[]; (*End Private Context*)
EndPackage[];

ensures firstly that all symbols created in foo's definition are encapsulated in mypkg`sub`Private` (and hence don't pollute elsewhere, either in end-users' namespaces or in developers' other package namespaces). In terms of the foo symbol itself; it is defined in mypkg`sub` and hence accessible in the private context given that mypkg`sub` is the only other context whose symbols the private context "sees" (we will cover foo's exposure shortly). This similarly applies to any other symbols defined in usage commands placed between BeginPackage["mypkg`sub`"] and Begin["`Private`"];

Hence following idiomatic practice we see that usage commands serve a dual purpose; ostensibly to provide documentation but actually much more importantly, to create the symbols in mypkg`sub` that are to be seen in the following private context (when being defined) and in other packages (when exported).

For years I used these idioms without giving a second thought to the actual context manipulations occurring behind the scenes that give them effect. It was only when I came to extend this BeginPackage/EndPackage framework (to implement (4) described later) that I came to appreciate the superb design that has underpinned their implementation (I'd like to know who came up with it?) over all these years. In fact, IMO this is the very marker and epitome of brilliant language design; end-users do things that just seem natural (place to-be-public symbols in idiomatic positions; idiomatically wrap to-be-encapsulated code in Begin/End etc.) while the design takes care of the rest. It does also mean though, that you deviate from designed idioms at your peril.

To be continued ...

Query 1: Why is the accepted answer (satisfying OP's specific request) risky? Query 2: Why doesn't WL's current package idiom "seamlessly scale"?

Returning to the example at hand, our idiomatic definition of foo in mypkg`sub` has, while encapsulating foo's definitions (in mypkg`sub`Private`) apparently also made the symbol foo public as it is exported as a symbol from mypkg`sub`. But therein lies the horns of an apparent dilemma, a tension between the imperatives of end-user and developer--on the one hand, foo needs to be private from an an end-users perspective but on the other, public from a developer's perspective in exporting it for servicing the over-arching package mypkg`. In fact, it is actually this tension that lies at the root of a design challenge, one eventually confronted by all developers.

It is readily met idiomatically (but IMO ultimately unsatisfactorily) by the following practice of internally placing Needs["mypkg`"]

BeginPackage["mypkg`"]
main::usage = "main[]"
Begin["`Private`"]
Needs["mypkg`sub`"];
main[] := foo[1] // moreThanfoo;
End[] (*End Private Context*)
EndPackage[]

This "solves" our dilemma because it makes foo "temporarily public" in the service of main's definition but then returns foo to its encapsulated, private status as the control flow exits EndPackage[] in preparation for end-user application. It is natural in the sense of mimicking an interactive session by, in a developer's environment (within the package definitions), making available functionality in the same familiar way Needs imports functionality in an interactive session.

Let's see the Context manipulation going on behind the scenes to give this effect--Needs puts mypkg`sub` on $ContextPath thereby giving access to all of mypkg`sub`'s exported functions but what Needs giveth EndPackage taketh as the latter removes mypkg`sub` from $ContextPath and with it end-users access to any of mypkg`sub` exported symbols as desired.

We can immediately see why this is preferable to the accepted answer and the approach initially tried by the OP: in this idiomatic approach, within mypkg` only foo is exposed from mypkg`sub` and not any of the symbols involved in foo's definition which is what occurs when mypkg`sub`Private` is added to $ContextPath as in the accepted answer (Note that Needs also ensures the package is a priori loaded in addition to adjusting $ContextPath and it turns out that de-coupling this action will key to a new "Developer's DeclarePackage").

Now there can be a bit of a trade-off between encapsulation and convenience. For example, I'm not a big fan of routinely including helper functions within the body of a Module/Block/Let as I feel that the loss of readability is rarely offset by any (local) encapsulating gains particularly when backed by the encapsulation still taking place at the package level. Modularization therefore, not only occurs at run-time but also occurs at the code organization level so I prefer to have helper definitions tucked away in other collapsible cells within a package preventing them from clogging up the logic of my main function (but see other viewpoints and this is restricted to functions not variables).

I've also never actually stumbled across a naming/variable conflict within a single "standard" package/file although of course when mission-critical functionality arises "total encapsulation" can be adopted in a similar way that a helper function can command its own package if it morphes into something more general.

Potential naming conflicts with helper functions/etc across packages however, is a very different matter and IMO real risks accrue in scaling a code-base by file encapsulation alone (effectively the approach of the accepted answer) and in particular, without including the context encapsulation provided by fine-grained package disaggregation.

Ok, so the described idiomatic method seems fit-for-purpose, can handle the OP's request for a single subsidiary function, so what about our earlier claim that it doesn't "seamlessly scale" or that it is in some sense inherently unsatisfactory. Well, first, this needs to be put in perspective; clearly the idiomatic method is an elegant, bullet-proof and proven base and it evidently does scale given the numerous large, successful packages/systems that have already been built on top of it. Hence for someone starting out in package development you can indeed go a long way before any of the current limitations start to make their presence felt---for most package development therefore, what follows may not be relevant for a considerable amount of time.

Nonetheless, it is nice to at least be aware of the existence of frameworks for managing the burgeoning complexity that inevitably accompanies a package's growth, a quest, to which we now turn.

To be continued ... (or not as this is becoming inappropriately saga-like for SE)