For the first part of your question please see a very recent Q&A: 

["How is + as an infix operator associated with Plus?"](http://mathematica.stackexchange.com/a/31377/187)

There you will find everything about defining operators yourself. For the second part, lets use your definition and bind it to [`CircleTimes`](http://reference.wolfram.com/mathematica/ref/CircleTimes.html). 

    CircleTimes[a_, b_] := a + a*b + b;

Now you can verify some of your assumptions. I use images instead of code-blocks, so that you see the rendered output. The operator can be inserted with the combination <kbd>Esc</kbd>+<kbd>c</kbd>+<kbd>*</kbd>+<kbd>Esc</kbd>. Verify commutativity:

> ![Mathematica graphics](https://i.sstatic.net/X1Eq3.png)

associativity 

> ![Mathematica graphics](https://i.sstatic.net/rjSEh.png)

Inverse element which does not necessarily exist in $\mathbb{Z}$

> ![Mathematica graphics](https://i.sstatic.net/oZWfp.png)

and the identity can be calculated likewise. I don't know whether you can show with *Mathematica* that this operator is closed in $\mathbb{Z}$, but you could easily make this clear, since you only use + and * which is closed in $\mathbb{Z}$.

I hope this gives you some point to start.