Assuming that you meant
$$
\langle n\lvert M \rvert m \rangle = m^2 \delta_{m,n}+(1-m)(\delta_{m,n+2}+\delta_{m,n-2})
$$
and ranges of $n,m\in\{1,2,3,\ldots,a\}$, you can define this matrix with
M[a_] := Table[m^2 KroneckerDelta[n, m] +
(1 - m) (KroneckerDelta[m, n + 2] + KroneckerDelta[m, n - 2]),
{n, a}, {m, a}]
or, more efficiently,
M[a_] := SparseArray[{{m_,m_} -> m^2, {n_,m_} /; Abs[n-m]==2 -> 1-m}, {a,a}]
or even
M[a_] := SparseArray[{Band[{1, 1}] -> Range[a]^2,
Band[{3, 1}] -> -Range[0, a - 3],
Band[{1, 3}] -> -Range[2, a - 1]}]
Test:
M[6] // MatrixForm
(* {{ 1, 0, -2, 0, 0, 0},
{ 0, 4, 0, -3, 0, 0},
{ 0, 0, 9, 0, -4, 0},
{ 0, -1, 0, 16, 0, -5},
{ 0, 0, -2, 0, 25, 0},
{ 0, 0, 0, -3, 0, 36}} *)
Considering that this matrix isn't Hermitian and that you are using Dirac-notation matrix elements, I suspect that there is something wrong in these definitions.