Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Background

I'm relatively new to git — currently overseeing an offshore .NET-based development project on GitHub within a private business account, but little experience beyond that.

I'd like to set up repos for storing my Mathematica projects (including one for my dissertation work). Initial tests revealed that git will make an unusable mess out of .nb files. Searching lead me to resources mentioning that notebooks contain too much metadata to diff/merge cleanly (without some specialized logic).

Question:

What git customizations are appropriate when Mathematica files and projects are involved (i.e., Workbench projects)? I'm a git noob, so be as explicit as possible with any instructions/advice! Are there any specific considerations to be made when using GitHub to host my Mathematica projects?


Resources that might help:

  1. Pro Git Book:

  2. Are there suitable versioning systems for Mathematica notebooks?

share|improve this question
    
I think there is a way to make .nb files be treated as binary files. Even the smallest change makes the whole .nb file "dirty" Maybe you could do this for .nb files, .mx files etc. and have .m files be treated normally. I don't think this is in .gitignore, though… will look and see if I can find the settings. –  Eric Brown May 31 '13 at 1:02
    
I made some edits to your question — GitHub is just a service that uses git, which is the underlying VCS technology. Your questions are all about using git and not GitHub. Think of GitHub as being to your data/source as what Facebook/Flickr are to your photos. Just a place to store and share something and "like"/"follow" others (except, without the FB creepiness). There is nothing Mathematica specific about using GitHub. Secondly, the title mentioned .gitignore, while the body asked about other stuff, which was a disconnect... Looks like you aren't after just the .gitignore afterall –  rm -rf May 31 '13 at 1:16
    
I think he has a valid Mathematica ecosystem question. It would probably be a disaster to merge two .nb files, but it would seem appropriate to merge .m files. I think everything else (wdx, mx) is a binary. If nb could be treated as binary, then I think everything would be fine. –  Eric Brown May 31 '13 at 1:40
    
@rm-rf - I understand all the Git vs. GitHub stuff... I explicitly mentioned both git and GitHub in the question because while git may be the underlying vcs, GitHub is bound to be a more likely search term. –  telefunkenvf14 May 31 '13 at 3:02
    
@EricBrown - Yes, I think the treat .nb as binary is one tweak MMA users should be using---but I'm not exactly sure how/where it should be used. And are there consequences for how this and other settings are applied for team/public projects (with multiple participants)? –  telefunkenvf14 May 31 '13 at 3:07
add comment

1 Answer

Preamble:

Using git to version control your Mathematica projects is a good choice and you will not regret it. However, like with most tools, it has its own learning curve, the difficulty of which will depend on how comfortable you are with using unix style command line tools. While the basics of git are easy to learn and use (especially if you're a single user, using it to simply "save state"), it takes more ninja-fu to do more complicated tasks when working in a team environment (such as resolving merge conflicts, rebasing, rewriting history, etc.).

In this answer, I will not go over the details of using git — the Git – Book that you linked to is an excellent resource and it is a waste of time to regurgitate that. Instead, I'll try to focus on certain aspects of git that are useful in solo/collaborative work, some Mathematica specific settings, and also briefly discuss project hosting on GitHub. Most of my answer is based on personal experience, although I'm far from being a git ninja.

General Mathematica + git advice:

1. Avoid using Notebooks as your primary source code

As you noted in the question, there is a lot of meta data bundled inside each notebook. Although you can turn these off with FileOutlineCache -> False in the options settings, I have never managed to satisfactorily place notebooks under version control. Notebooks can contain In/Out labels, cell grouping rules, modification times, front end version info, etc. some of which can change by simply opening and re-saving without any modifications. In a team environment, this can lead to disasters and you'll be spending all your time resolving merge conflicts. The worst part — you'll be doing this with the ugly internal cell expressions in the notebook!

Strictly speaking, git should be used only for actual source code i.e. version control of code that you/someone else personally wrote/modified and not something that's modified in possibly unknown/undetermined ways by an external program. A one letter change should not result in a 10 line diff output. Placing notebooks under VCS is orthogonal to the goals of a VCS and should be avoided as much as possible. As I mentioned earlier, if you're a single user, working in a single branch all the time and are using git solely as a daily backup, then you could get by with adding notebooks to your repo. Marking a notebook as a binary with gitattributes is another option (and in fact, recommended if you must place it under VCS).

2. Get comfortable with packages

As a continuation from the previous point, if you want to use git effectively, then you should start maintaining your code in Mathematica packages (or plain m files). This does not mean that you should give up the notebook entirely. The notebook still serves as a useful, interactive tool to quickly develop/test/modify functions. Just remember to copy them over to the package when you're done with it, so that they can be placed under version control.

As you start writing more packages and become familiar with programming in Mathematica without the front end, you'll get better at writing your functions and programs directly in the .m file and only using the front end for final testing and improvements.

3. Your development environment is not the same as the deployment environment!

If you create a project in the Workbench, your project structure looks something like this:

Project/
   |-- Project/
   |      |-- Project.m
   |      |-- Kernel/
   |             |-- init.m
   |-- Project.nb
   |-- Paclet.info
   |-- Test/
         |-- Test1.mt
         |-- Test2.mt

whereas your deployed environment looks like this

Project/
   |-- Kernel/
   |      |-- init.m
   |
   |-- Project.m
   |-- Paclet.info

These two directory and file structures are different, so trying to deploy the development setup directly will give you an error. I must note — by "deploy", I mean copying over to $UserBaseDirectory as in the case of a simple application. For more complicated projects with documentation, you'll have additional complexities, but the concerns remain the same.

When hosting your project on GitHub, you should decide – do you want the repo to be for development purposes only (i.e., you don't expect anyone to pull and install it to $UserBaseDirectory) or do you plan on making it a combined repo which has both the deployed and development branches? If it's the latter, then I suggest maintaining a separate development branch with the first structure and a distinct master branch with the second structure and never let the two mix.

The easiest way to do this would be with an orphan branch (which, as the name suggests, has no parent). Let's say you have your development repo in ~/dev/project and remote repo (up-to-date) at github.com/user, then the following steps will help you establish a separate master

cd ~/path/to/mma/apps/dir/
git clone github.com/user/project.git
git checkout --orphan master
rm -rf ./*

Now in your Workbench project, deploy (and build, if necessary) your project to ~/path/to/mma/apps/dir/project, verify the files and then commit to master. You now have a master branch that is clean and immediately deployable. Make sure to never attempt to merge from the development branch to the master and vice versa. It simply won't work because they have no parents in common (remember, we used an orphan branch).

This means that all your new code/modifications must originate in the development, be versioned and pushed/pulled to the remote(s), from others, etc. and deployed and committed to master when ready (it doesn't make sense to push minor commits to master) only via the Workbench (or a custom deploy script). Note that the Workbench does not preserve file permissions when deploying, so if that is critical to your application, then you must use a custom deploy script. The master branch serves merely as the face of the application to the end user, whereas all the gory history is in the development branch.

General git + team work advice:

1. Use branches!

First, if you're using git, then you already know (or now you do) that branches are light weight (meaning, files are not copied like in SVN) and only contain a reference to the parent commit. This means that all your trials/experimentations/goofs should be done in a temporary branch which can then be merged to the main development branch when you're satisfied. If you're not, simply delete the branch and create a new one. As simple as that! Learn to make extensive use of branches on your local copy, but don't do this on the remote.

In addition to the main development line, it is very helpful if the team members each maintain their own remote branch that is up-to-date at all times, so that others can pull changes from them. This allows one to work in their own "private space" (the branch), while allowing others to publicly view and comment/discuss code changes before pulling. GitHub also allows you to issue pull requests from one branch to the other in the same repo, so that's also a possibility to let someone (say, project manager/team member) know that your changes are ready to be reviewed and merged.

2. Use blame and diff to track down changes

When working in teams, you might find that a certain line of code is causing you trouble and you want to find out who introduced it in which commit, so that you can track that person down and discuss why the change was made and how to work around the present situation. Use git blame and git diff for that.

Although "blame" is a rather strong term (we're not trying to blame anyone; just trying to fix the problem), the blame tool is useful because it annotates each line with the author that last modified it and the commit SHA that introduced the modification. git diff is also needed here to track down what was changed (or perhaps deleted).

3. Squash/rebase your history to keep it clean

This is more of a personal preference for you/your team on how you want to maintain your repo. Some users create a commit for every small change, but it quickly gets boring to see 10 checkins from one user that are variants of

 -- fixed comma
 -- changed indentation
 -- fixed typo
 -- missing ; inserted

While this is fine in a local repo, it is better to send just a single commit upstream. Using git squash, you can "squash" or collapse all of those to a single commit.

Rebasing is a more complicated procedure (but a very useful one) that lets you rewrite history as it didn't happen. For example, suppose feature A logically comes before feature B, but due to the twisted nature of development, was implemented after B. You can use the rebase tool to change the order of the commits. Look up rebasing – it's worth learning.

4. Don't rewrite history on the remote!

This is a very important warning! Every time you change history, you are changing the SHA of the commit. Rebasing, squashing, etc. (and other complicated changes with the git filter) all change the commit SHAs. While this is easily fixed (or not an issue) on local branches, once you've pushed to the remote, you should not attempt to rebase or make any such modifications, as you'll end up breaking your team mates' code (if they've already pulled your changes).

If you must modify the history of the remote commit, do it only if you know no one has pulled yet (e.g., within a minute of pushing). If someone has pulled it and you still need to modify, then let the other person know so that they can work around it. All in all, history modification is a messy operation and try to avoid it in shared code.


This is all I have time for now, but hopefully it provides a broad overview of using git for Mathematica projects and a little bit on using it for team work. There is not much use in focusing on GitHub alone, as the issues you raise (and I've tried to address) are fundamentally about git and not the specific online service you use.

As to what the .gitignore should have — well, just about anything you don't want added to VCS.

share|improve this answer
    
very sound advice, thanks. –  Tuku May 31 '13 at 14:51
    
Thanks for all the info. I'm already thinking of follow-ups regarding some team workflow examples and how best to contribute ideas/improvements/fixes to projects initiated by others on GitHub, without pi$$ing anyone off. (issues centered around 'what constitutes an improvement' come to mind---Mathematica's flexibility makes this somewhat of a challenging issue, especially given that lack of a standard 'planning tool' for MMA development, and the fact that most (?) of us don't practice test driven development.) I'll give ya the ol'check mark in a couple of days, in case you want to edit more. –  telefunkenvf14 Jun 5 '13 at 15:14
    
@telefunkenvf14 I can edit (later) to address some of that, but the remaining are highly dependent on the project lead and there's not much I can say. For instance, some folks accept pull requests for small modifications, whereas others expect the feature to be fleshed out in detail, some are anal about how the git commit message is written/formatted (Tim Pope is a good example), some base it on the experience/ability of the user (expect pros to contribute more), etc. Sometimes a simple comma might be the source of a huge bug... (contd.) –  rm -rf Jun 8 '13 at 16:19
    
... so while it looks like a "minor change" from a diff view, it might be the change that allowed your product to ship in time. Discuss in advance with your project/team lead and other devs as to what constitutes a decent contribution and what things will piss them off. There are some common sense rules as well — for instance, don't go around changing all the indentation and formatting in someone else's repo to your liking or get into a tab vs spaces holy war, etc. If they have bad practices, discuss with them and show an improvement in your fork and confirm before initiating a pull request. –  rm -rf Jun 8 '13 at 16:19
1  
In summary, Mathematica was not prepared to be in a version control system... Not even the "undo" feature they made it right. –  J. C. Leitão Aug 20 '13 at 10:18
show 14 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.