16
$\begingroup$

There was a recent question asking about how to design modular packages:

This is an important issue for which unfortunately there is no One True Solution. People use various approaches, some better than others. Please check the answers to the linked question.

My question ties into this, but is more specific:

  • What is the best way to design reusable sub-modules?

Most packages are meant for end-users. Some other packages are useful for developing other packages built on top of them. One example is Kuba's recently proposed error handling library. Another is my LTemplate package. Both of these are useful only for building other packages on top of them.

Suppose I want to use an error handling library in my own package. How do I deal with this dependency? Do I instruct the users of my own package to also install this other library? This is a sure recipe for failure: Some people will certainly mess up the installation. If they don't, the version of the error handling library they install may not be compatible with my package. But this incompatible version may be required by a different package they are also using ...

I believe that the only robust solution is to bundle the error handling library (the sub-module) with my package in such a way that it won't conflict with any other installations. This question is about how to design the sub-module to make this bundling easy and possible. The eventual goal is to develop a standard for doing this. The standard will hopefully be flexible enough that the design of the sub-module won't constrain the design of the main package too much.

I did try to design LTemplate this way, and it can be used as a case study to identify the problems and requirements when using such sub-modules, and to come up with a better design. This is how LTemplate currently works:

  • It can be loaded as a standalone package. This is important so that it can be used interactively while prototyping a new package. In this case, it lives in the LTemplate` context.

  • It can be embedded as a sub-module of a main package, say, MyPack. In this case, it lives in the context MyPack`LTemplate` so that it wouldn't conflict with the global installation (LTemplate`) or with another embedded instance (MyOtherPack`LTemplate`). Note that these may be different versions, or may be configured differently.

  • Embedding is achieved by placing the complete LTemplate directory within the MyPack package directory, and adding Get["`LTemplate`LTemplatePrivate`"] after BeginPackage["MyPack`"]. This resolves to the MyPack/LTemplate/LTemplatePrivate.m file, even if MyPack lives in a paclet. See an example here.

  • A usual package has the following structure: BeginPackage, usages/declarations, Begin["`Private`"], definitions, End[]; EndPackage[]. LTemplate puts the usage/declaration and the Begin/End part in a separate file called LTemplateInner.m, which is loaded by Get in several "wrapper" files containing different context setups. LTemplate is normally loaded through one of these wrappers. The LTemplate.m wrapper is for standalone loading, and places symbols into the LTemplate` context. The LTemplatePrivate.m wrapper is for loading when LTemplate is embedded in another package, and will set up the contexts so everything is in MyPack`LTemplate`.

  • Finally, practical needs made it necessary to allow for "configuring" LTemplate by the main package that embeds it. Thus it is also required for it to evaluate a ConfigureLTemplate function at loading time to set certain options.

As you can see, all this is pretty complicated, and it is not clear to me that it would play well with some of the possibilities for modularizing packages (e.g. the new style packages that Leonid described). It is also not clear that it is the best solution for a complex, multi-module package, where several of the modules may depend on LTemplate.

The goal of this question is to come up with a design (or standard) that is as simple as possible, can be used as a guide by anyone to create reusable modules, and will play well with the best suggested designs for modular packages.

|improve this question
$\endgroup$
  • 1
    $\begingroup$ I am afraid that the robust solution would require a lot of custom infrastructure built on top of currently existing functionality. The biggest problem is the emulation of lexical scoping by a single flat symbol table, which makes it impossible to have two versions of the same package running simultaneously in isolation from each other. This makes current namespace management of Mathematica fundamentally incompatible with versioning. Custom infrastructure can solve it, but that would require also a custom build / distribution system. Personally I would much prefer to keep custom extra stuff .. $\endgroup$ – Leonid Shifrin Jul 4 '18 at 20:38
  • 1
    $\begingroup$ ... to a minimum. But in terms of modularity, contexts as they exist represent a too low level of abstraction. If we create a new package format that would operate on higher level (I made a few half-hearted attempts before), this would be something very non-standard (mental overhead, hurts adoption), and also interop with existing infrastructure can be challenging. But if we don't, then we can forget about versioning, and without it one can't have a robust solution. Your solution for LTemplate to have special branch for embedding, does not look scaleable for nested dependencies, as is. $\endgroup$ – Leonid Shifrin Jul 4 '18 at 20:44
  • 4
    $\begingroup$ I will probably have more to say on this matter soon. The core idea for the potential solution is to not use contexts at all in code / package declaration, and leave context management entirely to the loader. That would allow one to solve the versioning problem. I already made an attempt of implementing this, some time ago, but I was missing a few critical ideas. Now I have a better preliminary overall design, which I will try to materialize into an answer as soon as I get a long enough spare moment, hopefully in the next day or two. $\endgroup$ – Leonid Shifrin Jul 4 '18 at 23:21
  • $\begingroup$ @Leonid It looks like the new-style packages could not really accommodate what I am currently doing with LTemplate, right? Breaking up IGraph/M into smaller parts if way overdue, and new-style packages would make this quite easy. But I can't seem to make LTemplate play nicely with it. For now, forget about nested dependencies and other problems with the current design of LTemplate—I'm just trying to make things work in a practical way. $\endgroup$ – Szabolcs Oct 22 '18 at 9:15
  • $\begingroup$ What are the problems that currently make LTemplate not play well with new-style packages? I have created a new room where we could discuss this further. $\endgroup$ – Leonid Shifrin Oct 22 '18 at 11:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.