137
$\begingroup$

I'm building a package to help me write packages and their documentation. In this post I explained how to make a package and its documentation. In the answer I provided I describe how to build a very simple package. However, I have been looking around the extra packages that come with Mathematica and in some packages I see many .m files. I see this as a good way of dividing the application. Can someone describe the structure of a package?

To do this lets try to make a package out of the next simple functions. Suppose that we have the following in a notebook:

AddTwo::usage = "AddTwo[a, b] returns a+b";
AddThree::usage = "AddThree[a, b, c] returns a+b+c";
DotTwo::usage = "DotTwo[a, b] returns a*b";
DotThree::usage = "DotThree[a, b, c] returns a*b*c";
AddTwo[a_, b_] := a + b;
AddThree[a_, b_, c_] := a + b + c;
DotTwo[a_, b_] := a*b;
DotThree[a_, b_, c_] := a*b*c;

I would like to put these functions in a package. They all seem to be very simple arithmetic operations so let us make a package named SimpleArithmetic. This package is perfect to be divided into sections. One for additions and one for products, so we can make "subpackages" Addition and Product. If we follow some of the examples in the Mathematica installation we can create a folder called SimpleArithmetic in say $UserBaseDirectory. Inside SimpleArithmetic we can create two other files Addition.m and Product.m. The code for the additions would be placed in Addition.m and the code for multiplications would be placed in Product.m.

The question now is, how would these files look like? There is also a folder called Kernel which contains Init.m.

Could someone please just explain the best practices to create packages? I've read over the documentation and the whole "context" and "packages" keywords have already confused me. The code in the files I have described would be very appreciated.

$\endgroup$
7
  • $\begingroup$ There is no simple answer to this. In the Java world there is a tool just to create quality metrics for this topic. Read about it here: clarkware.com/software/JDepend.html. This might give you an idea of the complexity of the subject you touched. $\endgroup$ Commented Jul 9, 2011 at 7:15
  • 1
    $\begingroup$ @ndroock, Is the creation of packages in Mathematica that complicated? So far I can just create a simple package, which is a collection of functions. I might be overthinking things because of the notation <<PackageName'Somethingelse' where the ' is of course `. What exactly does this mean? $\endgroup$
    – jmlopez
    Commented Jul 9, 2011 at 7:27
  • 11
    $\begingroup$ This is a large topic. A standard reference for writing packages is "Programming in Mathematica" by Roman Maeder, 3d edition - 1996 - still very good and clear. David Wagner has a nice chapter on packages in his book ("Power programming with Mathematica - the kernel"). If you want to start with something light-weight, this answer may get you started: stackoverflow.com/questions/6633180/making-mathematica-packages/… $\endgroup$ Commented Jan 19, 2012 at 17:53
  • 3
    $\begingroup$ The conventional package structure (and some of the reasoning behind it) is described in the documentation here. This won't answer your question in full, but familiarity with this is a must if you are going to develop packages. $\endgroup$
    – Szabolcs
    Commented Jan 19, 2012 at 18:00
  • $\begingroup$ @Leonid Your comment looks like a great answer (and it has six up votes!). How about making it official? $\endgroup$
    – JxB
    Commented Jan 20, 2012 at 8:12

9 Answers 9

156
$\begingroup$

Package creation is a large topic indeed. I will still attempt to give a minimal clarification of the encapsulation mechanism behind packages, since in my experience it pays off to understand it.

What constitutes a package

Basically, a piece of Mathematica code (usually containing a number of variable and function definitions), which is placed inside

Begin[someContext]

code

End[]

can be called a package. Usually, however, at least some more structure is present. In particular, to separate interface from implementation, the typical package looks like

BeginPackage[someContext]

public-functions-usage-messages 

Begin["`Private`"]

code

End[]

EndPackage[]

Contexts and symbol names

The context here is a namespace. The convention is that context name is a string ending with "`". At any given moment, the value for the current working namespace is stored in the system variable $Context, and can also be queried by calling Context[]. Begin["test`"] will simply add the current context to the context stack, and then change it to "test`", while End[] will exit the current context by making the previous one current.

Every symbol must belong to some context. The system commands belong to the "System`" context, and the default working context for interactive FrontEnd sessions is "Global`". When mma code is parsed, the symbols are given their "true" (long) names, which contain both a symbol name and a context where the symbol is. For example, Map is really System`Map, and if I define a function f[x_]:=x^2 in the FE session, it will be Global`f. For any symbol, one can call Context[symbol] to determine the context where that symbol belongs. To "export" a symbol defined in a package, it is sufficient to simply use it in any way in the "public" part of the package, that is, before "`Private`" or other sub-contexts are entered. Usage messages is just one way to do it, one in principle could just write sym; and the sym would be created in the main package context just the same (although this practice is discouraged).

Every symbol can be referenced by its long name. Using the short name for a symbol is acceptable if the context where it belongs belongs to the list of contexts currently on the search path, stored in a variable $ContextPath. If there is more than one context on the $ContextPath, containing the symbol with the same short name, a symbol search ambiguity arises, which is called shadowing. This problem should be avoided, either by not loading packages with conflicting public (exported) symbols at the same time, or by referring to a symbol by its long name. I discussed this mechanics in slightly more detail in this post.

Contexts can be nested. In particular, the "`Private`" above is a sub-context of the main context someContext. When the package is loaded with Get or Needs,only its main context is added to the $ContextPath. Symbols created in sub-contexts are therefore inaccessible by their short names, which naturally creates the encapsulation mechanism. They can be accessed by their full long names however, which is occasionally handy for debugging.

Storing and loading packages

Packages are stored in files with ".m" extension. It is recommended that the name of the package coincides with the name of the package context. For the system to find a package, it must be placed into some of the locations specified in the system variable $Path. As a quick alternative (useful at the development stage), $Path can be appended with the location of a directory that contains a package.

When the Needs or Get command are called, the package is read into a current context. What is meant by this is that the package is read, parsed and executed, so that the definitions it contains are added to the global rule base. Then, its context name is added to the current $ContextPath. This makes the public symbols in a package accessible within the current working context by their short names. If a package A is loaded by another package B, then generally the public symbols of A will not be accessible in the context C which loads B - if needed, the A package must generally be explicitly loaded into C.

If the package has been loaded once during the work session, its functions can be accessed by their long names even if it is not currently on the $ContextPath. Typically, one would just call Needs again - if the package has been loaded already, Needs does not call Get but merely adds its context name to the $ContextPath. The internal variable $Packages contains a list of currently read in packages.

The case at hand

Here is how a package might look like:

BeginPackage["SimpleArithmetic`"]

AddTwo::usage = "AddTwo[a, b] returns a+b";
AddThree::usage = "AddThree[a, b, c] returns a+b+c";
TimesTwo::usage = "TimesTwo[a, b] returns a*b";
TimesThree::usage = "TimesThree[a, b, c] returns a*b*c";

Begin["`Private`"]

plus[args___] := Plus[args];
times[args___] := Times[args]

AddTwo[a_, b_] := plus[a, b];
AddThree[a_, b_, c_] := plus[a, b, c];
TimesTwo[a_, b_] := times[a, b];
TimesThree[a_, b_, c_] := times[a, b, c];

End[]
EndPackage[]

The functions AddTwo, AddThree, TimesTwo,TimesThree are public because these symbols were used in the public part of the package. Their long names would be then SimpleArithmetic`AddTwo, SimpleArithmetic`AddThree, SimpleArithmetic`TimesTwo, SimpleArithmetic`TimesThree. The functions plus and times are private to the package, since they are in the sub-context `Private`, which is not added to the ContextPath when the main package is loaded. Note that this is the only reason they are private. Should I call AppendTo[$ContextPath,SimpleArithmetic`Private`], and they'd become as "public" as the main functions (practice that should of course be discouraged by which should clarify the encapsulation mechanism).

With regards to splitting a package into several packages, this is a normal practice, but usually an individual mma package contains much more functionality than say a typical Java class, more like Java package. So, in the case at hand, I'd not split it until you get a much more functionality in it.

Of course, I only discussed here a very small subset of things related to packages. I will hopefully update this tutorial soon. An excellent reference for writing packages is a book of Roman Maeder "Programming in Mathematica". It is pretty old, but still one of the most (if not the most) useful accounts on the matter.

$\endgroup$
9
  • $\begingroup$ thank you for taking the time to explain this important topic. I think I got this part covered. I'll be waiting for your update. What I really want is to know how to use SimpleArithmetic'Addition' and SimpleArithmetic'Product'. So far I can sort of do this but I always end up messing the links to the documentation I write. I could make everything as part of one whole package (documentation and package works) but I think it is very important to know how add subsidary packages to the application that work with the documentation. $\endgroup$
    – jmlopez
    Commented Jul 9, 2011 at 9:23
  • $\begingroup$ @jmlopez. Mr. Wizard's reply to this question may be of interest $\endgroup$
    – user1066
    Commented Jul 9, 2011 at 10:49
  • 1
    $\begingroup$ David Park's reply to this message is also very informative. (A new version of DP's Presentations package for Mma 8 is now available, BTW) $\endgroup$
    – user1066
    Commented Jul 9, 2011 at 11:00
  • $\begingroup$ @TomD, I think David Park's explanation is what I needed. Thank you. $\endgroup$
    – jmlopez
    Commented Jul 9, 2011 at 14:18
  • 1
    $\begingroup$ @TomD, I have tested what Park said, I indeed needed this piece of undocumented code in the PacletInfo.m file under Extensions: {"Kernel", "Context"->{"SimpleArithmetic'Addition'","SimpleArithmetic'Product'"}}. Now that that's resolve I can simply things by a lot. $\endgroup$
    – jmlopez
    Commented Jul 9, 2011 at 14:32
59
$\begingroup$

I think what confuses most users who are new to packages is the larger question of where to put them and how to use them. I am going to discuss this in a larger context.

Suppose you are working on some significant or extended topic, which we will call TopicX. This topic might include many notebooks of various kinds and several packages, and perhaps later WRI style paclet documentation.

First you need a place to collect all your work on TopicX. The best place to collect this is in a TopicX folder in your private Applications folder. You can find this Applications folder by evaluating $UserBaseDirectory in Mathematica and then looking for the pre-existing Applications folder. Many users find some reason to put their applications elsewhere, but I think this is the best and standard location for a number of reasons, which I won't expound on here.

Within the TopicX folder you could build a folder structure for your own notebooks and other files associated with the topic, according to your own preferences. So far, no package.

As you work on the topic you will find it convenient to develop various routines associated with the project. You might develop them in a notebook proper and then move them to a Routines section at the top of the notebook. You might leave a routine there for a while and even copy it from notebook to notebook until you are satisfied that it works properly. I often call this "package purgatory". For these routines write usage messages, a SyntaxInformation statement, Attributes if any, Options definitions if any, error messages if the routine checks for errors. If all this is done, the routine is ready for "package heaven".

An application may have more than one package associated with it. I am going to assume that this is the case, or a future possibility, and give the packages names other than TopicX. So let's assume that your first package will be named Package1. In the TopicX folder create a new file named Package1.m. You could do this by opening Mathematica, using Create New> Other> Package, and then saving the file as Package1.m in your TopicX folder.

Package files can have sectional organization just as regular notebooks. You may wish to create sectional organization for the BeginPackage and Usage messages, and for the Private section, and for an End section. You may also want subsections for individual routines. According to your taste. Package files can also contain Text cells for annotation or notes.

The actual Mathematica code in a Package file is contained in Code cells. These are automatically Initialization cells and they are evaluated when the package is loaded. Cells that have the Input Style are not part of the package. (Converting a Code cell to an Input cell is a way to save an old version of a routine.) You can copy your routines from the notebook where they were developed to the package file. Usage messages to the Usage section and code to the Private section. Depending on how you copy you may have to switch Input cells to Code cells using the context Style menu. Code cells, especially usage messages often do not conveniently break and require horizontal scrolling. Sometimes it helps to temporarily switch them to Input cells for editing.

Following the folder structure, the BeginPackage statement will be:

BeginPackage["TopicX`Package1`"]

and the package could be loaded from anywhere with:

<< TopicX`Package1`

However, there is another very convenient feature that WRI has implemented. If a user executes the load statement without the package name as follows:

<< TopicX`

then Mathematica looks for an init.m file within a TopicX/Kernel folder and evaluates it. So create a Kernel folder within TopicX and an init.m file within it, and include the statements:

Get["TopicX`Package1`"]
Get["TopicX`Package2`"]  

if there are other packages in the application.

That's it. I won't discuss the details of package code since that is pretty well discussed elsewhere.

Later, if you want to add WRI paclet documentation, you could obtain Wolfram Workbench. You could just transfer the package files to Workbench and start writing Guide and Function pages. One important thing to remember is that all the routines from all the packages in TopicX are included in a single documentation paclet for TopicX.

$\endgroup$
2
  • 4
    $\begingroup$ Welcome to the site, David! $\endgroup$ Commented Oct 2, 2012 at 17:16
  • 3
    $\begingroup$ Hi David, nice to see you here! $\endgroup$ Commented Oct 2, 2012 at 19:05
21
$\begingroup$

You'll want to at least look at chapters 1-2 of Roman Maeder's Programming in Mathematica for starters. That was the walkthrough I used when I was starting out with package writing.

In particular, the book provides the listing of a template package file, named Skeleton.m. Here is what it looks like:

(* :Title: Skeleton.m -- a package template *)

(* :Context: ProgrammingInMathematica`Skeleton` *)

(* :Author: Roman E. Maeder *)

(* :Summary:
   The skeleton package is a syntactically correct framework for package
   development.
 *)

(* :Copyright: © <year> by <name or institution> *)

(* :Package Version: 2.0 *)

(* :Mathematica Version: 3.0 *)

(* :History:
   2.0 for Programming in Mathematica, 3rd ed.
   1.1 for Programming in Mathematica, 2nd ed.
   1.0 for Programming in Mathematica, 1st ed.
*)

(* :Keywords: template, skeleton, package *)

(* :Sources:
   Roman E. Maeder. Programming in Mathematica, 3rd ed. Addison-Wesley, 1996.
*)

(* :Warnings:
   <description of global effects, incompatibilities>
*)

(* :Limitations:
   <special cases not handled, known problems>
*)

(* :Discussion:
   <description of algorithm, information for experts>
*)

(* :Requirements:
   ProgrammingInMathematica/Package1.m
   ProgrammingInMathematica/Package2.m
   ProgrammingInMathematica/Package3.m
*)

(* :Examples:
   <sample input that demonstrates the features of this package>
*)


(* set up the package context, including public imports *)

BeginPackage["ProgrammingInMathematica`Skeleton`", "ProgrammingInMathematica`Package1`", "ProgrammingInMathematica`Package2`"]

(* usage messages for the exported functions and the context itself *)

Skeleton::usage = "Skeleton.m is a package that does nothing."

Function1::usage = "Function1[n] does nothing."
Function2::usage = "Function2[n, (m : 17)] does even more nothing."

(* error messages for the exported objects *)

Skeleton::badarg = "You twit, you called `1` with argument `2`!"

Begin["`Private`"]    (* begin the private context (implementation part) *)

Needs["ProgrammingInMathematica`Package3`"]    (* read in any hidden imports *)

(* unprotect any system functions for which definitions will be made *)

protected = Unprotect[ Sin, Cos ]

(* definition of auxiliary functions and local (static) variables *)

Aux[f_] := Do[something]

staticvar = 0

(* definition of the exported functions *)

Function1[n_] := n

Function2[n_, m_ : 17] := n m /; n < 5 || Message[Skeleton::badarg, Function2, n]

(* definitions for system functions *)

Sin /: Sin[x_]^2 := 1 - Cos[x]^2

Protect[ Evaluate[protected] ]     (* restore protection of system symbols *)

End[ ]         (* end the private context *)

Protect[ Function1, Function2 ]    (* protect exported symbols *)

EndPackage[ ]  (* end the package context *)

For your own package, just modify, replace and/or delete stuff from the template as needed.

$\endgroup$
16
$\begingroup$

Quick start tutorial

This is a quick start guide on how to set up a package following the typical structure. It does not explain why it is usually done that way, and it does not discuss other possible way to do it. That is left to the other answers here (especially Leonid's), as well as the official documentation.

Follow this guide to quickly set up your first package, so you have something concrete to experiment with. After that you must read the other answers and the references below, to gain a more complete understanding.

What is a package?

It is a text file with the .m (or .wl) extension that contains function definitions, and adheres to certain conventions. It can be loaded with Needs or Get to make the functions available for use.

How to create a basic package?

A basic package consists of a single file. More complex, multi-file packages won't be discussed here.

  1. Choose a name for your package. For this example I will assume the name MyPack.

  2. Type the source code into a file named MyPack.m.

  3. The file must adhere to the following structure:

    BeginPackage["MyPack`"];
    
    (* Export section *)
    
    (* Every function that the package provides will have a usage message here. *)
    (* There will be no function definitions in this section, only usage messages. *)
    (* Public functions will have names starting with capitals, by convention. *)
    
    MyFunction::usage = "MyFunction[x]";
    
    Begin["`Private`"]; (* note ` character both before and after Private *)
    
    (* Implementation section *)
    
    (* Function definitions will go into this section *)
    
    MyFunction[x_] := helper[x]
    
    (* All functions which are not public, and are only used in the 
       internal implementation of the package, go into this section.
       These have non-capital names by convention. *)
    
    helper[z_] := z^2
    
    End[]; (* `Private` *)
    
    EndPackage[]; (* MyPack` *)
    
  4. The file must be placed into a directory which is in $Path.

    Packages are typically installed into FileNameJoin[{$UserBaseDirectory, "Applications"}]

How to load and use a package?

Evaluate

<< MyPack`

If MyPack.m is in $Path, it will be loaded.

Now the function MyFunction is available for use.

References


Notes

The description in this guide is oversimplified on purpose, to make it easy to follow. When I stated things in absolute terms, I "lied" a little bit here and there: you don't need to strictly follow this exact structure. However, this structure does represent the best practices, and going beyond it does require an understanding of contexts. The other answers here should be considered required reading after you have set up your first package.

$\endgroup$
14
$\begingroup$

Another thing that isn't clear from the documentation. If you want to make a .m file from .nb file, you need to change the cell properties of anything that you want in it to Initialization Cell before you save. Otherwise, the saved module file will have no active content!

$\endgroup$
1
  • 1
    $\begingroup$ Thank you for this! I had carefully followed what seemed to be all the instructions, including copy and pasting others' code snippets. I understand why users here might not have mentioned it, but it seems that the Mathematica documentation itself might! $\endgroup$
    – LSpice
    Commented Jan 16, 2017 at 16:18
14
$\begingroup$

For me, the best way to create packages in based on Maeder's book. That gets things done quickly & semi ... dirtily(!) so that usages work just fine, but there's no fancy documentation center entries. The way to create such a (completely doc. center compatible package is expertly covered here.

On to the question at hand ...

If you think you need to separate functionality, then one approach is (as a template)

BeginPackage["YourPackageDirectory`YourPackageName`", {"YourPackagesDependencys`"}];
(* usages go here *)
Begin ["Private`"];
(* Function definitions go here *)
End[]; (* private *)
(* protect what you want *)
EndPackage[];

So what you might want to do with your wrapper package is

BeginPackage[`Lopez`SimpleArithmetic`", {"Lopez`Addition`", "Lopez`Multiplication`"}];
Begin["`Private`"];
End[];
EndPackage[];

You'd then start another package like

BeginPackage["Lopez`Addition`"];
AddTwo::usage = "AddTwo[a, b] returns a+b";
AddThree::usage = "AddTwo[a, b, c] returns a+b+c";
Begin["`Private`"];
AddTwo[a_, b_] := a + b;
AddThree[a_, b_, c_] := a + b + c;
End[];
EndPackage[];

And a third along similar lines called Lopez`Multiplication. All packages live in $UserAddOnsDirectory/Lopez which you'll probably need to create.

You use them by loading Lopez`SimpleArithmetic (Needs["Lopez`SimpleArithmetic`"]), though you can also load individual packages for debugging.

As another guide, I'd suggest studying source code in the Statistics legacy as a complicated package with many dependencies all seamlessly resolved when you load things. Your mileage may vary.

D.

$\endgroup$
4
  • $\begingroup$ thank you for the comment "expertly covered here". I'm a little confused about the structure of the application. Do you mean to say that we have a directory called "Lopez" which contains m files with names "SimpleArithmetic.m", "Addition.m" and "Multiplication.m"? At this moment I'm not so concerned with dependencies. All I want is to have subsidary packages (Addition and Multiplication) in the main application (SimpleArithmetic). The whole packageDirectory'subdidarypackage' is confusing at this moment. I can't find an actual example in Mathematica that uses this format. $\endgroup$
    – jmlopez
    Commented Jul 9, 2011 at 9:33
  • $\begingroup$ Exactly. And you can find an example that uses this format. As I suggested, the legacy statistics package is a good place to start. Have a look in MultiDescriptiveStatistics.m. This requires StatisticsCommonMultivariateCommon which is in the directory Statistics/Common. $\endgroup$
    – dwa
    Commented Jul 9, 2011 at 12:30
  • 1
    $\begingroup$ the problem with this legacy package is that it does not give you the file structure with the documentation. In any case, I think I got the whole file structure straighten up now. I'll be writing up another answer for the whole documentation with the plain installation of Mathematica soon and this question has helped me a lot. $\endgroup$
    – jmlopez
    Commented Jul 9, 2011 at 14:39
  • 1
    $\begingroup$ Very important to notice the difference between Begin["Private`"] and Begin["`Private`"] in the above. The first one puts symbols into a single, global, Private namespace where they can collide with symbols from other packages. The second one scopes your symbols and will be critical for multi-package development, in my opinion. The wolfram site discusses only the first form here reference.wolfram.com/workbench/index.jsp?topic=/… $\endgroup$
    – Reb.Cabin
    Commented Jul 20, 2015 at 16:29
11
$\begingroup$

One useful approach might be to use Wolfram Workbench, if you have access to it. Whether you use Workbench or not, a good way to organise your application is to have an overarching package that calls the subsidiary packages. For example, in the large project I am developing at the moment, there is a package for general utilities (data massaging), a package for the main plotting functions (highly customised version of the usual ones), and a package that provides a more general version of one of the plotting functions (DateListBarChart). The general package calls the first two of these, and the general as well as the second package call the third. The general package could be as simple as the following. In fact, this is the main package for my application, with the acronym for my employer changed to XYZ.

(* Mathematica Package *)
(* Created by the Wolfram Workbench May 20, 2010 *)
BeginPackage["XYZ`" ,{"XYZ`DateListBarChart`","XYZ`XYZGraphs`","XYZ`XYZUtilities`"}]
 (* Exported symbols added here with SymbolName::usage *) 
Begin["`Private`"]
 (* Implementation of the package *)
End[]
EndPackage[]

The way one organises these (for deployment) is, on Mac OS X at least, to put them in /Users/username/Library/Mathematica/Applications/. Inside this would be a folder for your main application name (e.g. XYZ), containing the main package, XYZ.m and any subsidary packages. Documentation and Kernel are subfolders of this folder.

You might find some useful tips in Wolfram's white paper on large projects, available from this page.

$\endgroup$
1
  • $\begingroup$ @Verbeia, I used the Wolfram Workbench to see the directory and file structure but the examples they provided were done in the most simple case. You should check out this link which TomD provided. I hope that updating Java doesn't mess with my plain mathematica installation. That's all I'm using to make the documentation. $\endgroup$
    – jmlopez
    Commented Jul 9, 2011 at 14:43
8
$\begingroup$

I like to use the Mathematica plugin for Eclipse (or the Workbench) for development. It's very simple to build packages, you can write the documentation for your functions (as well as Guides and Tutorials) and you can deploy your package to your Mathematica installation so both your functions and the documentation are integrated with the built-in ones.

It works very nicely (and the same for webMathematica if you're interested).

$\endgroup$
1
5
$\begingroup$

There is an extremly nice tutorial "Building Packages: A basic tutorial" by David Reiss (Scientific Arts).

It is in a notebook format and a the link to it can be found at http://community.wolfram.com/groups/-/m/t/214901

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.