Let's denote a matrix by a simple notation:
M[a, b, c, d, e, f, g, h, q]
such that the matrix form is recovered by the substitution:
subM = {M[x__] :> MatrixForm[Partition[{x}, Sqrt[Length[{x}]]]]};
M[a, b, c, d, e, f, g, h, q] /. subM

Rotations and reflections of rows is one dihedral group $D_{2n}$ with $2n$ elements. Rotations and reflections of columns is another copy of dihedral group $D_{2n}$. Finally, transposition along the diagonals (or, when combined with a row or column reflection, equivalently simply rotations by 90 degrees) are a superimposed $S_2$ group with 2 elements. This means we have a semi-direct product $S_2\rtimes D_{2n}$ with $2(2n)^2$ elements in total.
In our example $n=3$, so that we get $72$ elements. One can generate an invariant object from an M
as follows:
invar[m_] := Block[{tmp, res, n, pref},
pref = m /. M[xx__] -> 1;
n = Sqrt[Length[m /. M -> List]];
tmp = Partition[m/pref /. M -> List, n];
res = Table[M[RotateRight[tmp, i]], {i, 0, n - 1}] //DeleteDuplicates;
res = ((Table[ M[Transpose[RotateRight[Transpose[(#[[1]])], i]]], {i, 0, n - 1}] &) /@ res) // Flatten // DeleteDuplicates;
res = {res, (Reverse@# & /@ # & /@ res)} // Flatten // DeleteDuplicates;
res = {res, (Transpose@Reverse@Transpose@# & /@ # & /@ res)} // Flatten // DeleteDuplicates;
res = {res, (Transpose@# & /@ # & /@ res)} // Flatten // DeleteDuplicates;
pref ( res /. M[x__] :> (M[x] /. List -> Sequence) /. List -> Plus) // Expand
]
So that we properly get 72 distinct objects in the most general case, collectively furnishing an invariant set:
invar[M[a, b, c, d, e, f, g, h, q]] /. subM

However, the function invar[m_]
is too hacky for my taste. Especially when n
grows bigger, I suspect it will become very slow. That is why I'd prefer to have a more elegant solution.