For the first part of your question please see a very recent Q&A:

"How is + as an infix operator associated with Plus?"

There you will find everything about defining operators yourself. For the second part, lets use your definition and bind it to CircleTimes.

CircleTimes[a_, b_] := a + a*b + b;

Now you can verify some of your assumptions. I use images instead of code-blocks, so that you see the rendered output. The operator can be inserted with the combination Esc+c+*+Esc. Verify commutativity:

associativity

Inverse element which does not necessarily exist in $\mathbb{Z}$

and the identity can be calculated likewise. I don't know whether you can show with Mathematica that this operator is closed in $\mathbb{Z}$, but you could easily make this clear, since you only use + and * which is closed in $\mathbb{Z}$.

I hope this gives you some point to start.

For the first part of your question please see a very recent Q&A:

"How is + as an infix operator associated with Plus?"

There you will find everything about defining operators yourself. For the second part, lets use your definition and bind it to CircleTimes.

CircleTimes[a_, b_] := a + a*b + b;

Now you can verify some of your assumptions. I use images instead of code-blocks, so that you see the rendered output. The operator can be inserted with the combination Esc+c+*+Esc. Verify commutativity:

associativity

Inverse element which does not necessarily exist in $\mathbb{Z}$

and the identity can be calculated likewise. I don't know whether you can show with Mathematica that this operator is closed in $\mathbb{Z}$, but you could easily make this clear, since you only use + and * which is closed in $\mathbb{Z}$.

I hope this gives you some point to start.