Getting serious about Mathematica programming

Question

I'm at a point in my Mathematica journey where I need to be a little bit more strategic about my coding. Presently, I have a bunch of notebook files organized using the filesystem of the OS and tossed on to a cloud drive (Ubuntu One, Google Drive, whatever) so that I have access to the code from the 3-4 machines upon which I use Mathematica.

Since I'm not a programmer by trade, I suspect a number of the following questions will be fairly generic, but since we've discussed using Mathematica as a first language (multiple times) my issues may arise with other relatively new users of the software.

I am trying to address the following concerns:

• What are some best practices for maintaining notebooks and packages across multiple computers?
• How does one effectively implement version control for notebooks?
• What strategies are useful for managing tasks in large projects?
• How can packages be distributed efficiently, especially to a user-base that may have limited Mathematica knowledge.

Multicomputer use: As alluded to above, I'm using a cloud service to maintain my notebooks; however I've run in to syncing problems enough times that I wonder if there aren't better solutions. I understand that CVS is an option for team-based programming, but I've never used this system before, have no prior knowledge of how hard it is to set up a server, and wonder if this would be overkill for projects that are effectively written by me on multiple computers. My experience with git is limited to downloading the SEUploader and getting scared away by rm-rf's answer to using git.

Version control: I have no idea how to do this properly. Presently whenever I start a project, I name it project-v01.nb and move on from there. I arbitrarily change the version number, typically when my cloud drive service throws a syncing error. There's not a chance in the world that my current system can provide differences between versions.

Task management: In an ideal world, I would finish every project at one, uninterrupted sitting. Presently I've tried two approaches: I use the heading styles of a notebook to create a type of outline. These notebooks get convoluted quickly because they are a mix of project design and project application. I then thought to move the project design components to a new notebook, which I named currentproblems-v01.nb. Then, like bunnies, currentproblems-v02, v03, ... popped up. I have no idea which current problem is, uh, current.

Distribution: I suspect other programmers have delusions of grandeur as well, thinking that their code will be used by millions. At the very least, I'd like to port my projects to the computers used by my research students, who have to use my code or I can fire them... I suspect this will involve using the Wolfram Workbench, but then I've heard some folks prefer Eclipse. I don't have a preference since I've used neither; I think the only real requirements of the software are (a) make my life easier and (b) maintain development so I don't become dependent on abandoned software. I envision creating notebooks and packages that undergraduate students (in Chemistry) will use in research and the classroom. Given that my students (the first-years, anyway) will have limited experience with Mathematica, I wonder if asking them to install packages is the best option to distribute code.

Answer 1

I know that you are thinking ahead and considering how to make the work flow easiest for yourself. Nevertheless, in reading over your question I suspect you are pointed in the wrong direction and will only make things more difficult and burdensome for yourself.

What you are talking about is writing Mathematica Applications. (I would start with a single Application.) An Application is something that you would normally distribute to others and it represents relatively finished work, generated and accumulated knowledge accessible to readers and various tools such as packages, palettes and style sheets. The only thing that counts is the content of the Application, how clear the presentations are, and how easy and natural the interface is. Nothing else much matters.

As a file structure a master copy of an Application would reside on the author's (i.e., your computer) in the $UserBaseDirectory/Applications folder. Distributed versions would exist in the same location on a user's computer, probably downloaded from a server as a zip file. Perhaps there is a way it could be put on a $BaseDirectory/Applications folder on a server that also had Mathematica and user's could access it from there. (I'm not an expert on that.) Otherwise all of the multiple computers business is a diversion.

Here is what the file structure for the master copy of an Application might look like. I'm not an expert on chemistry and am just making up names here but I hope it will convey the idea.

$UserBaseDirectory/Applications
     CarbonChemistry
    Documentation (maybe added later using Workbench)
    Kernel
         init.m (controls loading of the packages)
    FrontEnd
         StyleSheets
              CarbonChemistry
                   (any style sheets)
         Palettes
              CarbonChemistry
                    (any palettes)
    ChemDisplayRoutines.m (packages)
    PentaRings.m
    HexaRings.m
    RNA.m
    Tutorials&Papers (folder structure, notebooks and other material)
    CourseLessons
    PrivateNotebooksBob (folder structure and notebooks)
         ApplicationDevelopmentNotebooks
         ChemistryDevelopmentNotebooks
         Communications

$UserBaseDirectory/Applications is the place to put Applications because it survives Mathematica updates and because Mathematica automatically looks there for all the resources that go with the Application. Also it is a single place where you and users can keep all their work on CarbonChemistry.

When you distribute the Application, or updates, you simply zip the application and then delete the PrivateNotebooksBob folder. Users, on their end, could add their own PrivateNotebooks section. A user might even develop and add his own sub-package, which he might later share with you. This form of Application can be a great medium of collaboration.

A BeginPackage statements would, for example, look like:

BeginPackage["CarbonChemistry`PentaRings`", {"CarbonChemistry`ChemDisplayRoutines`"}]

The init.m file would look like:

Get["CarbonChemistry`ChemDisplayRoutines`"]
Get["CarbonChemistry`PentaRings`"]
Get["CarbonChemistry`HexaRings`"]
Get["CarbonChemistry`RNA`"]

The Application would be loaded with:

<< CarbonChemistry`

The reason I put extra level CarbonChemistry folders in the Palettes and StyleSheets folders is that it will create only a single entry on these Mathematica Menus. Then if you have a number of style sheets or palettes they are all grouped (and identified) with your Application in a single entry and not scattered all over the menu. Our Applications are guests on other people's computers and it's really not polite to fill up their Mathematica menus with scattered and unidentified items.

Version control: Forget it. The only thing that counts is the living tip of the coral. Sure, you can periodically save dated copies but you will very seldom, most likely never use them. (You could always file a FOI request with NSA to obtain a complete copy of everything you ever typed, dated and probably cross indexed, but it will be of no more use to you then it is to them.) The more useful type of version control is either short term or more limited. For example, if I am developing a routine or discussion in a notebook I will do it in a Section. If I'm dissatisfied, but not completely certain which way to go, I'll duplicate the Section and label it Try 2. I might get 6 or 7 Tries before I'm happy but then I delete all the initial tries. Similarly, if I have a notebook, PaletteGeneration.nb say, that generates a palette and I want to make a major change I might save it with a suffix PaletteGeneration1.nb and then continue developing with the original notebook.

Task management: Do one incremental addition or improvement to the Application at a time and it will accumulate. Of course, it helps to have a good concept of the ultimate structure of the Application and sometimes it might be necessary to completely reorganize an initial attempt.

My overall advice for Application design is to make it a seamless addition to Mathematica and follow the WRI documentation scheme. Don't put the user in a box that puts a wall between the user and Mathematica and don't design your own documentation scheme. The WRI scheme may not be perfect, but it's good enough and it's what the user knows. And make darn sure that a user has a completely positive initial experience.

The tools are already there. It's the technical content and presentation that are important. Don't get tangled in computer science and systems. Science is more fun. You'll have enough work as it is.

Answer 2

While some have pointed out that many of your questions apply to all programming environments. I do think Mathematica's immediacy provides some interesting ways to approach development projects at every scale. Some ideas follow.

Wolfram Workbench -- If memory serves, Wolfram Workbench sits on top of Eclipse.

Version control -- CVS provides a useful open source version control system, with which you can distribute versions of your and your student's code to multiple developers. Of course, IBM now has the whole Rational Rose suite of tools. Reviewing their documentation can help one think about this in a clear manner. Some links:

Rational Rose wiki

IBM's Rational Rose

Rational Rose has (is) the full UML paradigm. Well worth reviewing, even if you don't buy into all of it.

Project organization -- For me I think of several kinds of code:

• Simple calculations
• Development of functions
• Packages still in development
• Packages for production or deployment of solutions

When I maintain my discipline, I begin each project with 2 notebook files and 2 package files corresponding to these.

I use one notebook merely for development and calculations.

The second notebook becomes a repository of functions and especially of functions of functions and program control functions (I typically set up a main[] function (holdover from the C++ paradigm) in it). I may copy functions or lines of the code from them into the development notebook, but the function repository in the midst of development, may only get changed and update once a day.

As soon as I have a function or set of them locked down, they go into a package, but this typically happens on a longer time frame, maybe weekly. As I distribute code, I may have identical functions or even associated packages in several packages, so I typically pay attention to full path names and contexts. In my case, once deployed I won't have conflicts.

Packages for deployment require a variety of things including documentation, copy-write, security, encryption, association with a front end notebook or CDF or whatever, and the like. I've got these mostly buttoned down, so I can apply them readily when I need them.

webMathematica -- webMathematica, especially if your university will spring for it, might give you a great solution to both make your packages available to your students and sharing packages among students themselves.

Refactoring -- Out and out the best thing I do. Whenever I have any time or even when I feel blocked in solving a problem or writing some code, I refactor. Simplify code. Make it more self documenting. Improve naming of variables and functions. This better focuses my mind than anything else I do in programming. IT just helps everything.

IMHO, Martin Fowler's book Refactoring remains an indispensable classic.

Answer 3

This may seem unsophisticated, but I have developed a few tools and practices that help me keep my fairly large notebooks manageable and help me keep track of what I've done with projects that sometimes go on for months.

Specifically, I use:
1. A style sheet with several levels of headings to identify notebook sections.
2. Group openers to make it easy to hide and expose areas.
3. A standardized organizational scheme with sections for archived code, functions, working code, and, if projects goes on for a really long time, a journal.
4. A toolbar that makes it easy to do things like toggle frames, set background colors, and that identifies the notebook.
5. Color coding for functions, archives, code, etc.

I second Jagra's comment about re-factoring.

I'm not sure this really addresses your issues, though, and if others think it not useful, I'll delete it.