What are recommended guidelines for developing packages?

This might be of general interest – I have different questions regarding naming conventions, contexts, subcontexts, shadowing, etc., but I do feel that they are closely related, thus I don't really want to split this post into three.

1. Naming
What is the best method to name the package file, the package context, the directory of package files (or a more complex hierarchy of these files)? Which of these names must be the same? I got confused several times before, and though I always manage to solve a situation, I don't feel like I have a good understanding on how these things work.

When there are interrelated packages with different contexts where some symbols appear in all of these contexts then - when calling contexts in the same session - usually shadowing messages appear. This is useful, when such symbols have different definitions and are unintentionally named the same way, but not, in the following case. If someone has a newly introduced function option, like Verbose, which doesn't have an OwnValue, then it is totally unnecessary to invoke shadowing messages, as no call of Verbose could do any harm. There still might be difference in the overall description of Verbose in two packages (even when all OwnValues, DownValues, etc. are the same), for example their usage messages might differ, as different functions would utilize the Verbose option in the different packages.

What is the best way to deal with these things? Should a Common.m package be introduced, and all the related packages be moved under a common context-name and/or directory? Do they have to be in the same directory?

3. Grouping and sub-contexts
Following point 2, when is it useful to introduce sub-contexts (e.g. myContextformat and myContextcontent)? Should these be split into different files? How should these files be named? Is it necessary then to include a Common.m too or is it just for convenience? What should be kept in Common.m?

-
I'd say a Common.m file is useful when there are a lot of symbols that your system of packages share... then all your other packages can have something like BeginPackage["myContextpackage1", "myContextCommon"]. –  Ｊ. Ｍ. Jan 19 '12 at 16:19

Part of what you are asking is of course a matter of taste and habits, but here are my 2 cents:

1) if you want Mathematica to find your package files with a Needs or Get their context names must agree with the hierarchy of directories and filenames. I don't see any good reasons to diverge from that standard convention. For complex packages with many files you will also typically have a Kernel-subdirectory with an init.m, but I think these things are relatively well documented.

2) My personal opinion is that using symbols for option names is asking for exactly these kind of problems. Obviously at least some of the WRI personnel thinks the same, since in later versions there are more and more options that accept strings as names and the new way to work with options also full supports this. If you are worried about cluttering your code with too many pairs of "", note that this will work alright:

Options[f] = {"Verbose" -> False}

f[OptionsPattern[]] := (If[OptionValue["Verbose"],
Print["I'm so verbose!"]]; RandomReal[])

f[Verbose -> True]


or even:

f[someothercontextVerbose -> True]


What you loose is the possibility to have a usage message bound to the option name, but as you have noticed if there are more than one function using the same option name, the usage message is of limited use anyway and the details must be explained in the documentation of the function, not the option. WRI has the same problem, obviously: At least I don't think that this usage is of very much help:

?Method


Method is an option for various algorithm-intensive functions that specifies what internal methods they should use.

3) Introducing sub-contexts is useful when things get more complex and parts of the whole can be split up in more or less independent parts. Of course giving these parts names that make it easy to recognize what they provide is a good idea, but I think that's so obvious that I doubt I fully understand that part of your question. If you want these parts to be loadable without the other parts, you must split them in different files, otherwise it's up to you from the technical viewpoint. From the code organization point of view I would think that if it makes sense to split your packages in separate contexts, it usually is also a good idea to split them into separate files. That becomes even more important if several people work on the various parts, but I feel there is not much Mathematica code written that way (except within WRI). Of course it's not necessary to include a Common.m, but as you have mentioned it's a good approach to collect all symbols that the various parts share into one common context/file, and Common.m (myPackageCommon) is a common convention that is also used by WRI, so I'd stick with it. On the other hand I would consider it as a good design of your package when you don't need a Common.m, since then you obviously managed to really split your package in independent parts.

-
Hi Albert! Welcome to SE Mathematica site - it's great to see you here! And +1, of course. –  Leonid Shifrin Jan 19 '12 at 17:04
thanks @LeonidShifrin -- for both –  Albert Retey Jan 19 '12 at 17:30
@AlbertRetey, that was quick! Excellent ;-) –  user21 Jan 19 '12 at 18:38
Thanks Albert, that was helpful. I don't think that it's only the usage message that I loose by introducing string options instead of symbols. For one, strings do not have their own reference pages in the Documentation, which is somewhat of a drawback on understanding their use. Though I guess even string options could have their entries... –  István Zachar Jan 22 '12 at 10:34
I think that must be possible: you can find all the import/export formats in the documentation although there are no symbols related to them. To my current knowledge there are two main disadvantages for strings: a) no usage message, b) no auto completion. As soon as more than one function uses the same option name, the advantage of the usage message seems less convincing. Other than that there is the difference about how a symbol from another context is handled, I'm not sure whether one or the other is causing more problems, though. –  Albert Retey Jan 22 '12 at 18:20

While @Albert gave an excellent answer, here are my two cents:

Naming

One thing which is true almost always: make your context name for the package the same as the file name for that package (without .m, of course). While you technically can use a different name, this will make your life harder.

A lot depends on the complexity of the project. If it fits well into a single package, and does not have many supporting files, then what I usually do is to have just a package plus a notebook with tests, examples and development notes. If it is more complex, I'd really recommend Wolfram Workbench, because it will create and help you manage the project's infrastructure. In that case, you will have a Kernel sub-directory in your project folder, with init.m` file there, and this gives you additional control over things.

I do not totally agree here either with you or with @Albert. In my opinion, it is useful to have options (at least main ones, not sub-options perhaps) to be symbols rather than strings. I expressed my opinion on this matter more fully in my answer to this question. Even if the symbol is inert (as for options), I think it is important to know that it is shadowed, since otherwise you are in for very subtle bugs.

Generally, shadowing is a problem of the user of your package, not you. You can, however, take some steps to minimize the probability of shadowing, such as:

• Give your functions reasonably long and descriptive names
• Export minimal interface
• When using other packages inside yours, import them privately, when you can.

That said, I feel that we really need a number of tools to help dealing with shadowing when it happens.

Sub-contexts

Occasionally, I find them very useful. Here is what I do: when I develop some project, I tend to break it into several packages (files) - usually I work in Workbench. However, when the project is ready, I try to minimize the number of resulting packages. What I do often is to include / refactor some of the separate packages as sub-contexts in the final large package. There are a few subtleties involved here, but I find that generally this pays off.

-
good points, as expected. I was reading your arguments for symbols as option, but couldn't understand why you think that introspection is harder for strings: wouldn't you just see them with Options[function] just as well as symbols? Just looking at symbols in the same context doesn't actually give me any information about whether these are options and if so for which functions, or do I miss something? –  Albert Retey Jan 19 '12 at 20:34
@Albert Well, it will be hard for me to give a definitive argument here - I just have some intuitive dislike for string names for options (less so for "Sub-options"). It is of the similar nature as my dislike for XML when used say in Java for various purposes. I do use strings as option names for sub-options though. I think this is more a matter of personal taste, I don't have very strong arguments against strings for options. –  Leonid Shifrin Jan 19 '12 at 20:47
@Leonid: thanks for the two cents, I found them very useful. I do agree with you on symbol-options. I think a lot of "hostility" against strings comes from the fact that many of the (sub)options for e.g. Method and others are simply cannot be found in the Documentation (easily), which caused a lot of frustration so far for users. –  István Zachar Jan 22 '12 at 10:39