What are recommended guidelines for developing packages?

Question

This might be of general interest – I have different questions regarding naming conventions, contexts, subcontexts, shadowing, etc., but I do feel that they are closely related, thus I don't really want to split this post into three.

1. Naming
   What is the best method to name the package file, the package context, the directory of package files (or a more complex hierarchy of these files)? Which of these names must be the same? I got confused several times before, and though I always manage to solve a situation, I don't feel like I have a good understanding on how these things work.

2. Shadowing
   When there are interrelated packages with different contexts where some symbols appear in all of these contexts then - when calling contexts in the same session - usually shadowing messages appear. This is useful, when such symbols have different definitions and are unintentionally named the same way, but not, in the following case. If someone has a newly introduced function option, like Verbose, which doesn't have an OwnValue, then it is totally unnecessary to invoke shadowing messages, as no call of Verbose could do any harm. There still might be difference in the overall description of Verbose in two packages (even when all OwnValues, DownValues, etc. are the same), for example their usage messages might differ, as different functions would utilize the Verbose option in the different packages.

   What is the best way to deal with these things? Should a Common.m package be introduced, and all the related packages be moved under a common context-name and/or directory? Do they have to be in the same directory?

3. Grouping and sub-contexts
   Following point 2, when is it useful to introduce sub-contexts (e.g. myContext`format` and myContext`content`)? Should these be split into different files? How should these files be named? Is it necessary then to include a Common.m too or is it just for convenience? What should be kept in Common.m?

Answer 1

Part of what you are asking is of course a matter of taste and habits, but here are my 2 cents:

1. if you want Mathematica to find your package files with a Needs or Get their context names must agree with the hierarchy of directories and filenames. I don't see any good reasons to diverge from that standard convention. For complex packages with many files you will also typically have a Kernel-subdirectory with an init.m, but I think these things are relatively well documented.

2. My personal opinion is that using symbols for option names is asking for exactly these kind of problems. Obviously at least some of the WRI personnel thinks the same, since in later versions there are more and more options that accept strings as names and the new way to work with options also full supports this. If you are worried about cluttering your code with too many pairs of "", note that this will work alright:

   Options[f] = {"Verbose" -> False}

   f[OptionsPattern[]] := (If[OptionValue["Verbose"], Print["I'm so verbose!"]]; RandomReal[])

   f[Verbose -> True]

or even:

f[someothercontext`Verbose -> True]

What you loose is the possibility to have a usage message bound to the option name, but as you have noticed if there are more than one function using the same option name, the usage message is of limited use anyway and the details must be explained in the documentation of the function, not the option. WRI has the same problem, obviously: At least I don't think that this usage is of very much help:

?Method

Method is an option for various algorithm-intensive functions that specifies what internal methods they should use.

3. Introducing sub-contexts is useful when things get more complex and parts of the whole can be split up in more or less independent parts. Of course giving these parts names that make it easy to recognize what they provide is a good idea, but I think that's so obvious that I doubt I fully understand that part of your question. If you want these parts to be loadable without the other parts, you must split them in different files, otherwise it's up to you from the technical viewpoint. From the code organization point of view I would think that if it makes sense to split your packages in separate contexts, it usually is also a good idea to split them into separate files. That becomes even more important if several people work on the various parts, but I feel there is not much Mathematica code written that way (except within WRI). Of course it's not necessary to include a Common.m, but as you have mentioned it's a good approach to collect all symbols that the various parts share into one common context/file, and Common.m (myPackage`Common` ) is a common convention that is also used by WRI, so I'd stick with it. On the other hand I would consider it as a good design of your package when you don't need a Common.m, since then you obviously managed to really split your package in independent parts.

EDIT 2023

Considering Shadowing I think it is necessary to mention that there is new functionality I think since version 13 that allows to use context aliases when loading packages with Needs. I think that allows to mitigate almost all problems with shadowing. Please refer to the documentation of Needs for details (you also might find some answers about that topic on this site over time)...

Answer 2

While @Albert gave an excellent answer, here are my two cents:

Naming

One thing which is true almost always: make your context name for the package the same as the file name for that package (without .m, of course). While you technically can use a different name, this will make your life harder.

A lot depends on the complexity of the project. If it fits well into a single package, and does not have many supporting files, then what I usually do is to have just a package plus a notebook with tests, examples and development notes. If it is more complex, I'd really recommend Wolfram Workbench, because it will create and help you manage the project's infrastructure. In that case, you will have a Kernel sub-directory in your project folder, with init.m file there, and this gives you additional control over things.

Shadowing

I do not totally agree here either with you or with @Albert. In my opinion, it is useful to have options (at least main ones, not sub-options perhaps) to be symbols rather than strings. I expressed my opinion on this matter more fully in my answer to this question. Even if the symbol is inert (as for options), I think it is important to know that it is shadowed, since otherwise you are in for very subtle bugs.

Generally, shadowing is a problem of the user of your package, not you. You can, however, take some steps to minimize the probability of shadowing, such as:

• Give your functions reasonably long and descriptive names
• Export minimal interface
• When using other packages inside yours, import them privately, when you can.

That said, I feel that we really need a number of tools to help dealing with shadowing when it happens.

Sub-contexts

Occasionally, I find them very useful. Here is what I do: when I develop some project, I tend to break it into several packages (files) - usually I work in Workbench. However, when the project is ready, I try to minimize the number of resulting packages. What I do often is to include / refactor some of the separate packages as sub-contexts in the final large package. There are a few subtleties involved here, but I find that generally this pays off.