# Are there suitable versioning systems for Mathematica notebooks?

I am considering a project using Mathematica and openCL. I know that the openCL C source code can be tracked.

Has anyone used a versioning site or software so multiple developers can modify a common notebook? Is it possible to:

• check out a notebook non-exclusively?
• check out a notebook exclusively?
• change the notebook?
• merge a notebook locally?
• check in a notebook?
• I use Eclipse + Workbench plugin + SVN. – b.gates.you.know.what Sep 29 '12 at 9:48
• @FredDanielKline there is already a proposal for the community to migrate that question on SO. If that is supported, then we can merge your question with that one once it's migrated. – Verbeia Sep 29 '12 at 11:15
• I've experienced that under Linux, if you check in a notebook to an SVN archive while it is open in Mathematica, Mathematica has problems (I don't remember exactly because it's some time since I experienced it, but IIRC I had to restart Mathematica). I have no idea what SVN does to the file on checkin (I would have expected it to only read the file for checking in), but whatever it is, Mathematica doesn't like it. With Mercurial, I didn't have such problems. Note that a checkout while the notebook is open in Mathematica is never a good idea. – celtschk Sep 29 '12 at 16:07
• I thought about such a solution for a while, never came up with a satisfactory answer, however. There is just too much metainformation in nb files, which is of no real relevance for the code you actually want to check into a VCS. I resorted to keeping packages I write (as m files only) in git, while keeping regular backups of all nb files. Of course you could also save the nb as m file (therefore discarding all metainformation) and check that into your VCS. I just never found that a practical approach. – sebhofer Sep 29 '12 at 17:14

First, if you want to have a team working on Mathematica code, then you really should do it properly and use Wolfram Workbench. As pointed out by Leonid, notebooks are not the right tool for software development.

This said, I have used version control (git) with a notebook and successfully merged versions. To do it, I minimized the amount of metadata in the notebook by:

1. Turning off Notebook History (and clearing any existing history using the dialog)
2. Turning off the Notebook Cache (in the option inspector).

You can turn off both notebook options with with the single command

SetOptions[InputNotebook[],
PrivateNotebookOptions -> {"FileOutlineCache" -> False},
TrackCellChangeTimes -> False]


but clearing the existing notebook history (removing all of the CellChangeTimes cell options) is easiest using the provided dialog.

Outputs can be long and messy and normally don't want to be tracked by your VCS. Some input/ouput combinations I did want to keep, so I set the output cell option GeneratedCell->False and then both cells' options to make them non-deletable and non-editable. The rest of the output cells were removed using the Delete All Output menu option.

Finally, keep your notebook(s) well organised with sections and subsections so that work and changes are clearly localised, which will make possible merges easier.

• Very good points,+1. Good to see you answering, it's been a long time. – Leonid Shifrin Sep 30 '12 at 13:34
• @Leonid Thanks! – Simon Oct 2 '12 at 8:48

If you plan to start a project with several developers involved, I would not base it on notebooks, because, as was mentioned by others, notebooks contain too much metadata. If you open a notebook file in a text editor, you will see that the actual code in the input cells only occupies a small fraction of the total file's size.

Besides, notebooks are aimed at a very different goal: explorations, interactive visualizations, and perhaps rapid prototyping. They are not aimed at structuring and maintaining code bases, at least not without some extra framework built on top of the core notebook interface.

I would definitely recommend the toolchain containing Wolfram Workbench (which is based on Eclipse and integrates with CVS and SVN), Mathematica packages,and the MUnit package for unit tests. Packages contain just code, are in plain ASCII files, and are well-suited for versioning. Developers can still use notebooks for interactive development, but I would not do shared work with notebooks. I would rather use them in two other capacities where they shine:

• Exploration and prototyping medium for individual developers (don't have to be shared)
• A medium of communication: one developer can create a notebook with illustrations, and send to others or commit that under his / her own sub-project. Others can use that to quickly see things, but would not modify it.

Of course, I don't mean to suggest that notebooks should never be shared and jointly modified, but if you reduce that to a minimum, you will avoid most of the merging problems involving them.

The bottom line of my message here is that code should be separated from the environment which has nothing to do with the problem that the code solves directly. The less metadata you will have hanging around the better it will be, because this metadata creates serious mental overhead for the developers. For any project except most trivial ones, this overhead will actually be much larger than the one associated with learning and using Workbench and related tools. And, the Workbench is integrated with Mathematica /FrontEnd sufficiently tightly that you don't lose the advantages notebooks give you - interactivity, rapid prototyping, etc.

• @NasserM.Abbasi That depends on the definition of useful work. If you have a project with more than one developer, it is likely to eventually have a moderately large code base. In such a case, different modules developed by different developers should have clear interfaces separated from their implementations. Those interfaces must be agreed upon, by the developers. In any case, my point is that the notebooks should not contain serious work to be shared. If the interface of a given module changes, it is dev's responsibility to update the notebook, if that is at all necessary. – Leonid Shifrin Sep 29 '12 at 21:35
• @NasserM.Abbasi The whole point of separation between interface and implementation, which is central to software development, is exactly to decouple implemetation changes from the use cases. If someone checks out some module and changes implementation, that should not affect the usage (notebook). While if someone changes also the interface, then it is a bad separation of duties: any given interface should IMO be "owned" (in the sense of write access) by only the developer who works on that module. – Leonid Shifrin Sep 29 '12 at 21:38

I lack the reputation to comment on a different answer so I must write my own answer (how perverse!).

Let me add to what simon wrote.

First, there need to be quotes around "TrackCellChangeTimes" in the code he wrote here or else it enters the Notebook's options as something like CellContextTrackCellChangeTimes.

Second, it is possible to programmatically clear the CellChangeTimes from a notebook. Here is one approach:

CleanNotebook[nb_NotebookObject] := Module[{},
SetOptions[nb,
PrivateNotebookOptions -> {"FileOutlineCache" -> False},
"TrackCellChangeTimes" -> False];
SetSelectedNotebook[nb];
SelectionMove[nb, All, Notebook];
FrontEndExecute[{FrontEndRemoveOptions[FrontEndSelectionObject,
"CellChangeTimes"]}]
]


The last three commands come from inspecting the system file ./SystemFiles/FrontEnd/SystemResources/HistoryOverview.nb in the Mma distribution; that is the code for the palette which lets you inspect the CellChangeTimes.

A new possibility is to use mathematica-notebook-filter which parses Mathematica notebooks and strips all output cells and metadata so that these are not committed into the version control system.

In the specific case of git, it is quite easy to integrate mathematica-notebook-filter so that git automatically cleans the output and metadata when calculating diffs through the use of gitattribute filters. You will need to have mathematica-notebook-filter filter installed and added to your path variable (or adapt the configuration below to point to the binary) and add the following line to your ~/.gitattributes file:

*.nb    filter=dropoutput_nb


This instructs git to parse all files matching *.nb with the dropoutput_nb filter which is defined in your ~/.gitconfig as:

[filter "dropoutput_nb"]
clean = mathematica-notebook-filter
smudge = cat


If, for some reason, you want to have a specific Mathematica notebook committed with all output and metadata, you can disable the filter in the project's .gitattributes file by adding:

notebook_file.nb    !filter


Disclaimer: I am the author of this tool. It is open source and feedback (both good and bad) is appreciated. Contributions are welcome on Github.

• +1 I was thinking about this approach but since I don't have skills to write it outside WL it was far away on my todo. I was thinking about using something like NotebookImport[EvaluationNotebook[], "Input" -> "Text"] as a filter, should be possible to call it as a script. What do you think? – Kuba Sep 6 '17 at 6:44
• I see you've added the same answer to related topic. It fits there better but can't deny it is on topic here too. Wouldn't it be more clear / convenient to post here only the comment with a link so that only one answer needs to be maintained? – Kuba Sep 6 '17 at 6:58
• I definitely think it's relevant to both questions, but I also don't really like have a link to other answers (unless it is an outright duplicate). So in the end I opted to double up. If you want, I could abbreviate the answer here and link to the other answer perhaps. I'm not familiar enough with NotebookImport[], so I don't really know (also I'm not sure how it would fit into a script). – JP-Ellis Sep 6 '17 at 8:41
• wolframscript -code 'UsingFrontEnd @ NotebookImport["path/to/nb.nb", "Input" -> "Text"]'` this works for example. – Kuba Sep 6 '17 at 8:53

Not a full answer, but too long to fit in a comment, will delete when this is settled. Also notice that Eclipse > 3.7.2 doesn't work with the Workbench plugin. The setup Eclipse+Workbench plugin+SVN works well when sharing packages (.m) but I have little experience doing it with notebooks.

I setup a quick example trying to answer the comment by @Nasser but I'm not sure if this is satisfactorily; please see first screenshot.

Regarding merging, I think there is at least a basic functionality; again, not sure if this is exhaustive enough; please see second screenshot.