Declaring Package with dependencies in multiples files?

Question

I am new to using Mathematica where I write most of my code in packages.

So to get the hang of it I am trying to set up an example of what a project might actually look like.

/TestPackage
/TestPackage.m (* top level package code goes here *)
/Component1
/Component1/foo.m (* some functions belonging to component 1*)
/Component1/bar.m (* some other functions belonging to component 1*)
/Component1/Component1.m (* mid level package code goes here*)
/Component2/Component2.m (* mid level package code goes here*)

Where TestPackage needs functions from both of the components, and Component1 requires functions that have been split into the smaller files foo.m and bar.m

For a single file package I can do:

BeginPackage[<pkg-name>`]
(* stuff here *)
EndPackage[]

but how can I do the relative depedencies?

Aside

Why ask this question?

Well looking at this documentation:

https://reference.wolfram.com/workbench/index.jsp?topic=/com.wolfram.eclipse.help/html/tasks/applications/packages.html

If you want to develop several applications which refer to each other, you can do this using project references. Suppose that you have two applications, App1 and App2. Furthermore, App1 needs functions from App2, for example, it contains the declaration BeginPackage["App1", "App2"].

(What I suggest above)

When you run or debug the project for App1 the Workbench will add the source for App2 so that this gets used as well.

Which sounds odd, as often times a package is split into different levels and the lower levels are sequestered away. From the above documentation it sounds like those lower levels should be elevated, or given Smurf naming e.g. MyPackageSubComponentUtilities rather than MyPackage/SubComponent/Utilities.m

In the same documentation, under "Multiple Packages":

is the recommended way to do multiple packages.

Which to me means that perhaps my question is due to a lexical misunderstanding. Maybe what I am trying to write is a level lower than a Mathematica Package, more akin to a python module...

Now it could be that these are just outdated docs that persist, so I have also gone through the circular links in the new docs, with closest thing I have found being the scantly linked to:

Files For Packages

which doesn't really say anything (I can understand in context of answering my question) but the brief allude to the context searching for /dir last.

and the tutorial Setting Up Wolfram Language Packages does not address anything more advanced than a single file package.

any help would be appreciated.

The answer provided for sub-context's here suggests it involves subtlety and workbench :/

Answer 1

Preamble

I will interpret your question as a general question on how one can organize multi-file projects in Mathematica / WL. There are two main axes in this problem, and they are

• Loading procedure / loading sequence
• Namespace / context management

To the first approximation, one can treat these independently. However, any complete solution will provide some solutions for both which might be interdependent, so I will consider them together.

Simplest case: multi-fragment (multi-file) project, single namespace

This is more or less what is suggested in the tutorial you mentioned. In this scheme, there is a single public context MyProject` and (usually) a single private context MyProject`Private`, just like in a single-file package. However, the code is split into several files / fragments.

While there is a single public and private namespace, a split to several fragments requires one to have a fragment that would declare all public / exported symbols, even if their implementation will live in a different fragment, and to load that fragment before any others.

Since init.m file is present in the Kernel sub-folder, it is an entry point, and you are free to customize everything in that init.m. There are two main variations one can employ in this scheme

Namespace-aware fragments

You load all fragments within init.m using Get, one after another.

This method has the following advantages:

• Project's loading sequence is explicit and declarative.
• It will be easier to statically analyze, if such a need arises
• One doesn't need to use the $InputFileName variable and absolute paths

But every fragment would need to separately wrap the code into BeginPackage - EndPackage, as well as Begin["`Private`"] - End[] for the private section, which one may consider as a disadvantage.

One master fragment and N slave fragments

You only load your main fragment within init.m with Get, and inside the main fragment Private` section you load all other fragments.

This method couples project's loading with execution of implementation code, but then you don't need to assign a namespace for each fragment - if you load it inside the first / main fragment Private` section, other fragments may contain just the code, and will automatically be parsed into Private` context - so that in this approach only the main fragment needs the structure BeginPackage - EndPackage, as well as Begin["`Private`"] - End[].

It is however less declarative, and one would need to use the $InputFileName and absolute paths to load other fragments.

Problems solved and not solved in this approach

The main problems that this approach solves:

• It is much more convenient to work with smaller source files
• One can split code according to the different functionality, making it easier to reason about

The problems this approach does not solve

• Name collisions between private symbols: since there is still a single public and single private namespaces, one may forget that a given private symbol has been defined in one fragment and define it again for a completely different purpose.
• No real modularization: even though code is split into fragments, the developer is not encouraged to minimize fragment's interdependencies, avoid circular dependencies, etc. Moreover, even if the code has pieces which are completely independent from each other, single namespace does not allow one to really separate them well enough.

New-style packages: undocumented modern way of structuring multi-fragment packages

With a usual disclaimer that one can't rely on undocumented functionality, and can use it at one's own risk only, let me briefly describe the new package format (AFAIK, available since V10).

Project's structure

In this format, the project typically has a flat structure, similar to the structure you mentioned:

MyProject    
  Kernel
    init.m
  Main.m
  ExtraTools.m
  Utilities.m

The folder Kernel/init.m is optional, if one of the fragments is named the same as the project's folder, in the above case would be MyProject.m. One can, however, still use Kernel/init.m, if one wants some other fragment to run first.

Note that one doesn't need to load manually all the fragments - this is done automatically by Get, for a new-style package, so it is enough to only indicate the fragment that you want to run first.

Syntax and scoping

In this format, you split your package into several fragments, each of which has to start with a declaration:

Package["MyPackage`"]

In this format, you don't need to wrap the code in BeginPackage - EndPackage or Begin - End. Instead, if you want to make your symbol public (i.e. having the context MyPackage`), you write

PackageExport["MySymbol"]

anywhere in the file / fragment (in fact, it does not matter in which fragment even, although it is more logical usually to do that in the one where you implement that symbol).

There is one additional level of scope in this format, which is package scope. Package-scoped symbols are available to all fragments in the package, but not exported to the end user. You declare them as

PackageScope["MyInnerSharedSymbol"]

Package-scoped symbols live in MyProject`PackageScope` context.

If you want to import other packages, you use PackageImport, which is an analog of Needs:

PackageImport["DatabaseLink`"]

typically one does that at the start of the fragment, after Package declaration. You shouldn't use Needs in place of PackageImport in this format.

All symbols that are not exported, not package-scope, and not in any of the imported packages, are considered private. The difference with the standard format, however, is that each fragment has its own private context, so that private contexts of different fragments are different and don't collide. For example, for a fragment named ExtraTools.m, the private context will be MyPackage`ExtraTools`PackagePrivate`, while for fragment Utilities.m, it will be MyPackage`Utilities`PackagePrivate`. This separation of private sub-contexts is also why package-scoped symbols become necessary, as a way to privately communicate between different fragments.

How it works, and a few gotchas

How it works (brief summary)

The way this format works is different from the standard one, where all contexts of all symbols were determined at run-time. Here, all fragments are first analyzed statically, to determine which symbols should be created in which contexts. During that analysis, dependencies (packages imported using PackageImport) are loaded dynamically, so one probably can't call it fully static. After all symbols get resolved, the fragments are actually run.

Why Needs can not be used

The above makes it more understandable, why Needs should not be used in place of PackageImport - the code using Needs will execute too late, after all static analysis would have been already performed, and so imported symbols won't have a chance to resolve in time, and in code of the package would be instead considered private symbols).

Loading order

Unfortunately, one can't easily control the fragments execution order, which is alphabetical in fragments names. One can, however, point Get in the init.m to a specific fragment, to ensure that that one will be executed first - and then all the rest in the alphabetical order of their names.

If you must enforce specific loading order, you can mangle fragments names, adding letters A, B etc. in front, to ensure that their alphabetical order corresponds to the order in which you actually want them to be loaded. However, one might argue that dependence of very specific loading order for several fragments is evil, and a sign of bad design.

A pitfall with declaration statements

It is important to point out is that statements PackageExport, PackageScope, PackageImport are actually tokens for a static analyzer, rather than real Mathematica code. In particular, wrapping them into anything won't work:

PackageImport["MyOtherPackage`"];

or

If[var === val, PackageExport["MySymbol"]]

won't work (the semicolon at the end is a very common error).

Per-fragment nature of PackageImport

Regarding PackageImport, one non-obvious thing is that a package, imported in one fragment, is not considered imported in others. This means that, if you use some package in several fragments of your new-style package, you need to PackageImport it in all of them, even though they are parts of the same big package you are developing. This behavior has both pros and cons, although it may often seem an inconvenience.

Problems solved and not solved in this approach

The main problems that this approach solves:

• All of the problems that the simpler approach solves
• Better encapsulation: no collisions between private symbols of different fragments
• Somewhat better modularization: the fact that one can't use private symbols of one fragment in the other fragment, forces one to more carefully separate the interface from implementation. One is given the tool of package-scoped symbols to use for such communication, but you will be better off reducing their number to a necessary minimum

Problems still not solved in this approach

• Still better level of modularization can only be achieved when symbols belonging to different structural parts, live in different namespaces
• The new-style package format as such does not scale well when the amount and complexity of code grows. This is so both because it does not provide a true multi-level modularization, and because individual modules are not separately reloadable or otherwise self-contained (for example, can't be tested in isolation).

Still, this method is a huge step in the right direction, and for most small to medium or even large projects can be enough.

Multi-package projects

If your code base really becomes large and complex, you may want to upgrade to a more flexible and powerful structure. The basic idea is to make different modules of your project independent full-fledged packages. For each individual module, one can use any of the methods I described earlier. One would also need a few additional pieces such as custom loader and probably an interface section.

I will describe this method in more detail, if there is enough interest.

Conclusions

Depending on one's needs and also stylistic preferences, one can use a number of different schemes to scale up the code and create multi-file projects. I have described a few methods that should do it for most user-level projects.

Answer 2

This answer adheres to the outline in the section "Multi-package projects" of Leonid's answer (emphasis mine):

If your code base really becomes large and complex, you may want to upgrade to a more flexible and powerful structure. The basic idea is to make different modules of your project independent full-fledged packages. [...] One would also need a few additional pieces such as custom loader and probably an interface section.

Below is (briefly) described the multi-package dependency, loading, and code generation of the package "MonadicContextualClassification.m", [1].

The goal of the package [1] is to provide a Domain Specific Language (DSL) for rapid specification of machine learning classification workflows.

Package file structure

1. Packages import code section.

   • Here packages are imported from GitHub with Import.

     • (If the corresponding definitions are not already in the context.)

   • These are the mentioned above "independent full-fledged packages".

     • (They were developed before the package [1].)

     • (The package [1] provides a DSL that combines those "full-fledged packages.")

2. Package interface definitions code section.

   • Here after BeginPackage the ::usage definitions are given.

   • This is the "interface section" mentioned above.

3. The private context declaration.

4. The declaration of the package contexts with Needs code section.

   • (For the contexts of the packages in the imports code section 1.)

5. Code generation section.

   • In this case with
     GenerateStateMonadCode["MonadicContextualClassification`ClCon",...]

   • This code generation code section together with the imports code section 1, correspond to the "custom loader" mentioned above.

     • (The package [2] generates the definitions of some of the functions of the package [1].)

6. Function implementations code section.

Testing

The development of the package was done with two unit test files. (One with hand-written unit tests, the other with randomly generated tests. The latter is specific to the methodology behind the package functionality.)

The point is that unit tests are crucial when dealing with this kind of complex package dependencies. (And packages that deal with complex subjects.)

Summary diagram

Below is given a diagram that summarizes the development of "MonadicContextualClassification.m". For more details see the end sections of the document "A monad for classification workflows".

Example run

The following screenshot is of an example run that demonstrates the package import prompts and the utilization of the loaded packages in a classification pipeline. (The packages have functionalities for training classifier ensembles, making ROC plots, and finding importance of variables.)

References

[1] Anton Antonov, Monadic contextual classification Mathematica package, (2017), MathematicaForPrediction at GitHub.

[2] Anton Antonov, State monad code generator Mathematica package, (2017), MathematicaForPrediction at GitHub.

Answer 3

I tend to do this with an autogenerated loader file for all of my packages. This allows me to set up a single directory in my project called "Packages" and then the loader runs through the packages in that and sets up autoloading for their symbols.

This allows me to have a package structure like:

Packages
  + Pkg1.m
  + Pkg2.m
  Sub1
    + SubPkg1.m
  Sub2
    + SubPkg2.m
    SubSub1
      + SubSubPkg1.m
 etc.

And everything in Pkg1 and Pkg2 will be exposed at top level, everything in SubPkg1 is in the context Sub1, everything in SubPkg2 is in Sub2, and everything in SubSub1 is in SubSubPkg1.

The key here, though, is that the loader first scans the head portion and declares all new symbols there as to be autoloaded. Then I expose all of the top-level package contexts to every other package.

The nice thing about this is that every package sees every other package so I don't have to manually manage dependencies. As long as you're not reusing symbol names across packages (I'd recommend against that anyway) you're fine.

The private implementations aren't shared across contexts.

One final thing before I dump a bunch of code is that this is very similar to how python rolls, except with less import management.

For those interested, here's a template of this kind of load-file (and since it's only ~300 LOC I've copied it below):

(* ::Package:: *)

(* ::Section:: *)
(*$Name Loader*)


BeginPackage["$Name`"]


$Name::usage="A head for the package (useful for messages)";


(* ::Subsection:: *)
(*Package Level Symbols*)


BeginPackage["`PackagePrivate`"]


$PackageName::usage="The name of the package";
$PackageDirectory::usage="The directory for the package";


$PackageListing::usage="The listing of packages";
$PackageContexts::usage="The list of contexts exposed to all packages";
$PackageDeclared::usage="Whether the package has been auto-loaded or not";


$PackageFileContexts::usage="The contexts for files in the package";
$DeclaredPackages::usage="The set of packages found and declared via the autoloader";
$LoadedPackages::usage="The set of loaded packages";


PackageExecute::usage="Executes something with the package contexts exposed";
PackageLoadPackage::usage="Loads a package via PackageExecute";
PackageLoadDeclare::usage="Declares a package";


PackageAppLoad::usage="Loads the entire package";
PackageAppGet::usage="";


(* ::Subsubsection::Closed:: *)
(*Begin*)


Begin["`Private`"]


(* ::Subsubsection::Closed:: *)
(*Constants*)


$PackageName="$Name";
$Name["Directory"]:=
  $PackageDirectory;
$PackageDirectory=
  DirectoryName@$InputFileName;


$PackagePackagesDirectory="Packages"


$Name["PackageListing"]:=$PackageListing;
$PackageListing=<||>;
$Name["Contexts"]:=$PackageContexts;
If[!ListQ@$PackageContexts,
  $PackageContexts=
    {
      "$Name`",
      "$Name`PackagePrivate`"
      }
  ];
$PackageDeclared=TrueQ[$PackageDeclared];


$Name["FileContexts"]:=$PackageFileContexts;
If[Not@AssociationQ@$PackageFileContexts,
  $PackageFileContexts=
    <||>
  ];
$Name["DeclaredPackages"]:=$DeclaredPackages;
If[Not@AssociationQ@$DeclaredPackages,
  $DeclaredPackages=
    <||>
  ];
$Name["LoadedPackages"]:=$LoadedPackages;
If[Not@ListQ@$LoadedPackages,
  $LoadedPackages={}
  ];


(* ::Subsubsection::Closed:: *)
(*PackageFilePath*)


PackageFilePath[p__]:=
  FileNameJoin[Flatten@{
    $PackageDirectory,
    p
    }];


(* ::Subsubsection::Closed:: *)
(*PackageExecute*)


PackageExecute[expr_]:=
  Internal`WithLocalSettings[
    Begin[$PackageContexts[[1]]];
    System`Private`NewContextPath@
      Prepend[
        $PackageContexts,
        "System`"
        ];,
    expr,
    System`Private`RestoreContextPath[];
    End[];
    ];
PackageExecute~SetAttributes~HoldFirst


(* ::Subsubsection::Closed:: *)
(*PackagePullDeclarations*)


PackagePullDeclarationsAction//Clear
PackagePullDeclarationsAction[
  Hold[
    _Begin|_BeginPackage|
      CompoundExpression[_Begin|_BeginPackage,___]
    ]
  ]:=
  Throw[Begin];
PackagePullDeclarationsAction[e:Except[Hold[Expression]]]:=
  Sow@e;


PackagePullDeclarations[pkgFile_]:=
  pkgFile->
    Cases[
        Reap[
          With[{f=OpenRead[pkgFile]},
            Catch@
              Do[
                If[
                  Length[
                    ReadList[
                      f,
                      PackagePullDeclarationsAction@Hold[Expression],
                      1
                      ]
                    ]===0,
                    Throw[EndOfFile]
                  ],
                Infinity
                ];
            Close[f]
            ]
        ][[2,1]],
      s_Symbol?(
        Function[Null,
          Quiet[Context[#]===$Context],
          HoldAllComplete
          ]
          ):>
          HoldPattern[s],
      Infinity
      ]


(* ::Subsubsection::Closed:: *)
(*PackageLoadPackage*)


PackageLoadPackage[heldSym_,context_,pkgFile_->syms_]:=
  Block[{
    $loadingChain=
      If[ListQ@$loadingChain,$loadingChain,{}],
    $inLoad=TrueQ[$inLoad]
    },
    Internal`WithLocalSettings[
      System`Private`NewContextPath@$ContextPath,
      If[!MemberQ[$loadingChain,pkgFile],
        AppendTo[$loadingChain, pkgFile];
        With[{$$inLoad=$inLoad},
          $inLoad=True;
          Replace[Thread[syms,HoldPattern],
            Verbatim[HoldPattern][{s__}]:>Clear[s]
            ];
          If[Not@MemberQ[$ContextPath,context],
            $ContextPath=Prepend[$ContextPath,context];
            ];
          PackageAppGet[context,pkgFile];
          Unprotect[$LoadedPackages];
          AppendTo[$LoadedPackages, pkgFile];
          Protect[$LoadedPackages];
          ReleaseHold[heldSym]
          ]
        ],
      System`Private`RestoreContextPath[]
      ]
    ];


(* ::Subsubsection::Closed:: *)
(*PackageDeclarePackage*)


PackageDeclarePackage[pkgFile_->syms_]:=
  With[{c=$Context},
    $DeclaredPackages[pkgFile]=syms;
    $PackageFileContexts[pkgFile]=c;
    Map[
      If[True,
        #:=PackageLoadPackage[#,c,pkgFile->syms]
        ]&,
      syms
      ]
    ];


(* ::Subsubsection::Closed:: *)
(*PackageLoadDeclare*)


PackageLoadDeclare[pkgFile_String]:=
  If[!MemberQ[$LoadedPackages,pkgFile],
    If[!KeyMemberQ[$DeclaredPackages,pkgFile],
      PackageDeclarePackage@
        PackagePullDeclarations[pkgFile]
        ],
      PackageAppGet[pkgFile]
    ];


(* ::Subsubsection::Closed:: *)
(*PackageAppLoad*)


$Name["Load", args___]:=
  PackageAppLoad[args]


packageAppLoad[dir_, listing_]:=
  With[
    {
      fileNames=
        Select[
          FileNames["*", dir],
          DirectoryQ@#||MatchQ[FileExtension[#], "m"|"wl"]&
          ]
      },
    Replace[
      Select[fileNames, 
        StringMatchQ[
          ToLowerCase@FileNameTake[#],
          "__pre__."~~("m"|"wl")
          ]&
        ],
      {f_}:>Get[f]
      ];
    PackageAppLoad[
      $PackageListing[listing]=
        Select[fileNames, StringFreeQ["__"]@*FileBaseName]
      ];
    Replace[
      Select[fileNames, 
        StringMatchQ[
          ToLowerCase@FileNameTake[#], 
          "__Post__."~~("m"|"wl")
          ]&
        ],
      {f_}:>Get[f]
      ];
    ];


PackageAppLoad[dir_String?DirectoryQ]:=
  If[StringMatchQ[FileBaseName@dir,(WordCharacter|"$")..],
    Internal`WithLocalSettings[
      Begin["`"<>FileBaseName[dir]<>"`"],
      AppendTo[$PackageContexts, $Context];
      packageAppLoad[dir, FileNameDrop[dir, FileNameDepth[$PackageDirectory]+1]],
      End[]
      ]
    ];
PackageAppLoad[file_String?FileExistsQ]:=
  PackageLoadDeclare[file];
PackageAppLoad[]:=
  PackageExecute@
    packageAppLoad[
      FileNameJoin@{$PackageDirectory, $PackagePackagesDirectory}, 
      $PackageName
      ];
PackageAppLoad~SetAttributes~Listable;


(* ::Subsubsection::Closed:: *)
(*PackageAppGet*)


$Name["Get", f__]:=
  PackageAppGet[f];
PackageAppGet[f_]:=
  PackageExecute@
    With[{fBase = 
      If[FileExistsQ@f,
        f,
        PackageFilePath[$PackagePackagesDirectory, f<>".m"]
        ]
      },
      With[{cont = 
        Most@
          FileNameSplit[
            FileNameDrop[fBase, 
              FileNameDepth[PackageFilePath[$PackagePackagesDirectory]]
              ]
            ]},
        If[Length[cont]>0,
          Begin[StringRiffle[Append[""]@Prepend[""]@cont, "`"]];
          (End[];#)&@Get[fBase],
          Get[fBase]
        ]
      ]
    ];
PackageAppGet[c_,f_]:=
  PackageExecute[
    Begin[c];
    (End[];#)&@
      If[FileExistsQ@f,
        Get@f;,
        Get@PackageFilePath[$PackagePackagesDirectory, f<>".m"]
        ]
    ];


(* ::Subsubsection::Closed:: *)
(*End*)


End[]


(* ::Subsubsection::Closed:: *)
(*EndPackage*)


EndPackage[]


(* ::Subsection:: *)
(*Load*)


If[!TrueQ[`PackagePrivate`$PackageDeclared],
  `PackagePrivate`PackageAppLoad[];
  `PackagePrivate`$PackageDeclared
  ]


EndPackage[]

Name that file TestPackageLoader, change $Name to TestPackage, then set up a directory that looks like this:

TestPackage
  + TestPackageLoader.wl
  Kernel
    + init.m
  Packages
    Pkg1.m
    Sub1
      Sub1.m
      Sub2
        Sub2.m

Put

<<TestPackage`TestPackageLoader`

in init.m

Put something like

Pkg1::usage="...";
Begin["`Private`"];
Pkg1[]:=Sub1[]
End[]

in Pkg1.m

And put a similar type of thing in Sub1.m and Sub2.m

When you call <<TestPackage` the following will happen:

• Autoloading will be configured for Sub1, Sub2, and Pkg1 for when they're used
• When loaded, each symbol will know about the others
• Only Pkg1 will be exposed in the top-level context

I've used this to good effect here, here, here, and here, among many places, although my actual load files are much more fully featured (they include exception handling helpers, autocompletion helpers, dependency loaders, etc.).

Hopefully this at least provides inspiration for what can be done, even if it isn't directly useful

Answer 4

Yes, I agree that the documentation for packages could be significantly improved as much of it seems to be folklore or scattered in various places. This really is a pity since once settling on a workflow I have found the whole package idiom extremely powerful and IMO, the fact that it has changed so little for so long is testament to the soundness of the original design. Having said this, there are a couple of tweaks and mindsets needed to make the design fly:

1. Abandon all second argument usages of BeginPackage
2. Get into the habit of aligning package namespaces with directory layouts and then exploit mercilessly

My experience is that doing so allows your code to emerge thoroughly modularized together with flexible deployment opportunities. The following describes the set up and idiom I believe is intended by the WW file structure in your example (but with some minor personal customizations):

TestPackage
   Kernel
       init.m
   TestPackage.m
   Component1
      Component1.nb   (* foo, bar code defined here *)
      Component1.m
   Component2
        Component2.nb   (* foo2, bar2 code defined here *)
        Component2.m
        Subcomponent2 
            Subcomponent2.nb (* deeperFoo2, deeperBar2 code defined here  *)
            Subcomponent2.m

I have taken the liberty of adding a third component, Subcomponent2, in order to emphasise the possibility of increasingly nested folders as part of unlimited modularization. Curiously, deeply nested examples seem rare (the deepest I have observed is 3 deep in the Parallel`Queue`FIFO package) but IMO this is what ultimately gives the package idiom its scalability and flexibility (I've often wondered if Mathematica could significantly reduce its run-time footprint with more granulated packaging in its System` context).

The important point to note is that this directory structure directly maps to the contexts of the public functions when the package is loaded. So for example, the following shows the full context names of the public functions when the "relevant" package is loaded

TestPackage`Component1`foo
TestPackage`Component1`bar
TestPackage`Component2`foo2
TestPackage`Component2`bar2
TestPackage`Component2`Subcomponent2`deeperFoo2
TestPackage`Component2`Subcomponent2`deeperBar2

A user won't need to use the full context names as illustrated shortly after fleshing out some details (with any clarifications if needed later):

---

• The top level is identical to what WW generates except that instead of a file Component1.m I prefer to have a folder Component1 that contains Component1.m. This is because I like to leave the option open for these subpackages to contain their own Kernel folder, Tests, DataRepositories etc. Further, these .m files I like to generate from initialized cells in corresponding .nb files since I prefer to develop code in the front-end. Of course many other workflows edit the .m file directly in the frontend, WolframWorkbench, IntelliJIDEA or your favourite editor.

• Note these extra files/folders don't affect the calling of these packages and to see this it helps to internalize Needs control flow: Firstly, for Needs to recognize these packages the TestPackage directory needs to be on $Path or else called with a second argument : as in Needs["TestPackage`", "path-to-TestPackageDirectory"]. Needs resolves any named package to a directory before first trying to load a contained Kernel/init.m file or if that fails, a .m file with the same name (FindFile can be used to check if the correct file is going to be loaded). For example, assuming $Path is appropriately set, the following invocations loads the corresponding file.

  Needs["TestPackage`"]
  ________{TestPackage/Kernel/init.m}

  Needs["TestPackage`Component1`"]
  ________{TestPackage/Component1/Component1.m}

  Needs["TestPackage`Component2`"]
  ________{TestPackage/Component2/Component2.m}

  Needs["TestPackage`Component2`Subcomponent2`"] ________{TestPackage/Component2/Subcomponent2/Subcomponent2.m}

In practice, these subpackages don't need to be loaded individually (but having the option to do so adds flexibility in setting up certain dependencies) since this is done automatically through the magic of DeclarePackage - IMO one of the most underrated and important functions in Mathematica. On invoking Needs["TestPackage`"] the file TestPackage/Kernel/init.m is loaded which, containing Get["TestPackage`TestPackage`"] loads in TestPackage/TestPackage.m which hopefully contains the following DeclarePackage commands (there used to be, I think[?] , an old Roman script that produced these DeclarePackage blocks automatically)

DeclarePackage["TestPackage`Component1`", {"foo","bar"}];
DeclarePackage["TestPackage`Component2`", {"foo2","bar2"}];
DeclarePackage["TestPackage`Component2`Subcomponent2`", {"deeperFoo2","deeperBar2"}]}

These DeclarePackage calls load the subpackages on a needs basis by setting the Stub attribute to the variables and adding the sub packages' contexts to $ContextPath (thereby making the entering of full context names unnecessary). So, for example, entering deeperFoo2 now automatically calls Needs["TestPackage`Component2`Subcomponent2`"] which loads TestPackage/Component2/Subcomponent2/Subcomponent2.m and deeperFoo2's definition(s).

The reason this is so worthwhile is that having swathes of code loaded on a just-in-time basis means reaping (compounding) run-time benefits from all your modularizing efforts (not to mention making any end-user Needs usage redundant).

In a well-known white paper, Building Large Software Systems in Mathematica, a comprehensive list of the benefits of modularization is provided but without mentioning the extra benefits obtained by having finely granulated packages (backed by DeclarePackage) significantly lesson a run-time footprint. It seems inevitable that this will become increasingly important for new applications in the cloud, on Raspberry Pi and parallel kernels etc.

• An example of the aforementioned alignment between directories and BeginPackage usage is Component2.nb containing initialization cells with the sequence (Note, Needs is expecting this package structure when loading, Get doesn't)

  BeginPackage["TestPackage`Component2`"]
  
    usage::foo2="foo2[var] does this";
  
      Begin["Private`"]
  
         foo2[_]:="does this"
  
      End[]
  
  EndPackage[]
  

• Internal Needs calls (not using the second argument of BeginPackage) can then be used to handle all required package dependencies.

• I have used this set up to implement various use cases (e.g. using/exposing common subpackages, versioning etc), without "unweildy" manipulations of $Path, $ContextPath etc . At least, I have never found this set-up wanting with what seem like common and previously described requirements (but I suspect I might be missing something or have misinterpreted these other use-cases given an apparent need for a "next generation" packaging system).
• There is actually a further extension I have in my development set-up that redefines Needs mainly to handle the different imperatives of end-users and developers. This extension is for development and doesn't make it into deployed packages ( [or posts] given that packages etc really ought not to modify system functions). Maybe future built-in functions will include these developer capabilities but irrespective, IMO the current package idiom can already take you a very long way.