Preamble:
Using git to version control your Mathematica projects is a good choice and you will not regret it. However, like with most tools, it has its own learning curve, the difficulty of which will depend on how comfortable you are with using unix style command line tools. While the basics of git are easy to learn and use (especially if you're a single user, using it to simply "save state"), it takes more ninja-fu to do more complicated tasks when working in a team environment (such as resolving merge conflicts, rebasing, rewriting history, etc.).
In this answer, I will not go over the details of using git — the Git – Book that you linked to is an excellent resource and it is a waste of time to regurgitate that. Instead, I'll try to focus on certain aspects of git that are useful in solo/collaborative work, some Mathematica specific settings, and also briefly discuss project hosting on GitHub. Most of my answer is based on personal experience, although I'm far from being a git ninja.
General Mathematica + git advice:
1. Avoid using Notebooks as your primary source code
As you noted in the question, there is a lot of meta data bundled inside each notebook. Although you can turn these off with FileOutlineCache -> False in the options settings, I have never managed to satisfactorily place notebooks under version control. Notebooks can contain In/Out labels, cell grouping rules, modification times, front end version info, etc. some of which can change by simply opening and re-saving without any modifications. In a team environment, this can lead to disasters and you'll be spending all your time resolving merge conflicts. The worst part — you'll be doing this with the ugly internal cell expressions in the notebook!
Strictly speaking, git should be used only for actual source code i.e. version control of code that you/someone else personally wrote/modified and not something that's modified in possibly unknown/undetermined ways by an external program. A one letter change should not result in a 10 line diff output. Placing notebooks under VCS is orthogonal to the goals of a VCS and should be avoided as much as possible. As I mentioned earlier, if you're a single user, working in a single branch all the time and are using git solely as a daily backup, then you could get by with adding notebooks to your repo. Marking a notebook as a binary with gitattributes is another option (and in fact, recommended if you must place it under VCS).
2. Get comfortable with packages
As a continuation from the previous point, if you want to use git effectively, then you should start maintaining your code in Mathematica packages (or plain m files). This does not mean that you should give up the notebook entirely. The notebook still serves as a useful, interactive tool to quickly develop/test/modify functions. Just remember to copy them over to the package when you're done with it, so that they can be placed under version control.
As you start writing more packages and become familiar with programming in Mathematica without the front end, you'll get better at writing your functions and programs directly in the .m file and only using the front end for final testing and improvements.
3. Your development environment is not the same as the deployment environment!
If you create a project in the Workbench, your project structure looks something like this:
Project/
|-- Project/
| |-- Project.m
| |-- Kernel/
| |-- init.m
|-- Project.nb
|-- Paclet.info
|-- Test/
|-- Test1.mt
|-- Test2.mt
whereas your deployed environment looks like this
Project/
|-- Kernel/
| |-- init.m
|
|-- Project.m
|-- Paclet.info
These two directory and file structures are different, so trying to deploy the development setup directly will give you an error. I must note — by "deploy", I mean copying over to $UserBaseDirectory as in the case of a simple application. For more complicated projects with documentation, you'll have additional complexities, but the concerns remain the same.
When hosting your project on GitHub, you should decide – do you want the repo to be for development purposes only (i.e., you don't expect anyone to pull and install it to $UserBaseDirectory) or do you plan on making it a combined repo which has both the deployed and development branches? If it's the latter, then I suggest maintaining a separate development branch with the first structure and a distinct master branch with the second structure and never let the two mix.
The easiest way to do this would be with an orphan branch (which, as the name suggests, has no parent). Let's say you have your development repo in ~/dev/project and remote repo (up-to-date) at github.com/user, then the following steps will help you establish a separate master
cd ~/path/to/mma/apps/dir/
git clone github.com/user/project.git
git checkout --orphan master
rm -rf ./*
Now in your Workbench project, deploy (and build, if necessary) your project to ~/path/to/mma/apps/dir/project, verify the files and then commit to master. You now have a master branch that is clean and immediately deployable. Make sure to never attempt to merge from the development branch to the master and vice versa. It simply won't work because they have no parents in common (remember, we used an orphan branch).
This means that all your new code/modifications must originate in the development, be versioned and pushed/pulled to the remote(s), from others, etc. and deployed and committed to master when ready (it doesn't make sense to push minor commits to master) only via the Workbench (or a custom deploy script). Note that the Workbench does not preserve file permissions when deploying, so if that is critical to your application, then you must use a custom deploy script. The master branch serves merely as the face of the application to the end user, whereas all the gory history is in the development branch.
General git + team work advice:
1. Use branches!
First, if you're using git, then you already know (or now you do) that branches are light weight (meaning, files are not copied like in SVN) and only contain a reference to the parent commit. This means that all your trials/experimentations/goofs should be done in a temporary branch which can then be merged to the main development branch when you're satisfied. If you're not, simply delete the branch and create a new one. As simple as that! Learn to make extensive use of branches on your local copy, but don't do this on the remote.
In addition to the main development line, it is very helpful if the team members each maintain their own remote branch that is up-to-date at all times, so that others can pull changes from them. This allows one to work in their own "private space" (the branch), while allowing others to publicly view and comment/discuss code changes before pulling. GitHub also allows you to issue pull requests from one branch to the other in the same repo, so that's also a possibility to let someone (say, project manager/team member) know that your changes are ready to be reviewed and merged.
2. Use blame and diff to track down changes
When working in teams, you might find that a certain line of code is causing you trouble and you want to find out who introduced it in which commit, so that you can track that person down and discuss why the change was made and how to work around the present situation. Use git blame and git diff for that.
Although "blame" is a rather strong term (we're not trying to blame anyone; just trying to fix the problem), the blame tool is useful because it annotates each line with the author that last modified it and the commit SHA that introduced the modification. git diff is also needed here to track down what was changed (or perhaps deleted).
3. Squash/rebase your history to keep it clean
This is more of a personal preference for you/your team on how you want to maintain your repo. Some users create a commit for every small change, but it quickly gets boring to see 10 checkins from one user that are variants of
-- fixed comma
-- changed indentation
-- fixed typo
-- missing ; inserted
While this is fine in a local repo, it is better to send just a single commit upstream. Using git squash, you can "squash" or collapse all of those to a single commit.
Rebasing is a more complicated procedure (but a very useful one) that lets you rewrite history as it didn't happen. For example, suppose feature A logically comes before feature B, but due to the twisted nature of development, was implemented after B. You can use the rebase tool to change the order of the commits. Look up rebasing – it's worth learning.
4. Don't rewrite history on the remote!
This is a very important warning! Every time you change history, you are changing the SHA of the commit. Rebasing, squashing, etc. (and other complicated changes with the git filter) all change the commit SHAs. While this is easily fixed (or not an issue) on local branches, once you've pushed to the remote, you should not attempt to rebase or make any such modifications, as you'll end up breaking your team mates' code (if they've already pulled your changes).
If you must modify the history of the remote commit, do it only if you know no one has pulled yet (e.g., within a minute of pushing). If someone has pulled it and you still need to modify, then let the other person know so that they can work around it. All in all, history modification is a messy operation and try to avoid it in shared code.
This is all I have time for now, but hopefully it provides a broad overview of using git for Mathematica projects and a little bit on using it for team work. There is not much use in focusing on GitHub alone, as the issues you raise (and I've tried to address) are fundamentally about git and not the specific online service you use.
As to what the .gitignore should have — well, just about anything you don't want added to VCS.
git, which is the underlying VCS technology. Your questions are all about using git and not GitHub. Think of GitHub as being to your data/source as what Facebook/Flickr are to your photos. Just a place to store and share something and "like"/"follow" others (except, without the FB creepiness). There is nothing Mathematica specific about using GitHub. Secondly, the title mentioned.gitignore, while the body asked about other stuff, which was a disconnect... Looks like you aren't after just the.gitignoreafterall $\endgroup$