Take the $C_{3v}$ point group for example:
Rot[θ_] := RotationMatrix[θ]
E0 = IdentityMatrix[2];
C1 = Rot[2π/3];
C2 = C1.C1;
σ1 = ReflectionMatrix[Rot[2π/3].{1, 0}];
σ2 = C1.σ1.Inverse[C1];
σ3 = C2.σ1.Inverse[C2];
C3v = {E0, C1, C2, σ1, σ2, σ3};
So $C_{3v}$ has the following 6 elements: $ \left\{ \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ \end{pmatrix}, \begin{pmatrix} -\frac{1}{2} & -\frac{\sqrt{3}}{2} \\ \frac{\sqrt{3}}{2} & -\frac{1}{2} \\ \end{pmatrix}, \begin{pmatrix} -\frac{1}{2} & \frac{\sqrt{3}}{2} \\ -\frac{\sqrt{3}}{2} & -\frac{1}{2} \\ \end{pmatrix}, \begin{pmatrix} \frac{1}{2} & \frac{\sqrt{3}}{2} \\ \frac{\sqrt{3}}{2} & -\frac{1}{2} \\ \end{pmatrix}, \begin{pmatrix} \frac{1}{2} & -\frac{\sqrt{3}}{2} \\ -\frac{\sqrt{3}}{2} & -\frac{1}{2} \\ \end{pmatrix}, \begin{pmatrix} -1 & 0 \\ 0 & 1 \\ \end{pmatrix} \right\} $
For given basis functions $\{f_n(\mathbf{r})\mid n=1\cdots N\}$, there is a representation $\{D(g)\mid g\in C_{3v}\}$, in which the $N\times N$ matrix $D(g)$ is defined as
$$ f_i(g^{-1}\mathbf{r})=\sum_{j=1}^N f_j(\mathbf{r})D_{ji}(g)$$
where $\mathbf{r}=${x,y}
.
If the basis functions are just monomials, e.g. $$f_1(\mathbf{r})=x^2,~~f_2(\mathbf{r})=y^2,~~f_3(\mathbf{r})=xy$$
I can compute the representation matrices using the Coefficient
function
f1[{x_, y_}] := x^2
f2[{x_, y_}] := y^2
f3[{x_, y_}] := x y
basis = {f1, f2, f3};
Rep[g_] := Block[{i, j, ig, x, y, r},
ig = Inverse[g]; r = {x, y};
Table[Coefficient[basis[[j]][ig.r], basis[[i]][r]], {i, 3}, {j, 3}]
]
repC3v = Rep /@ C3v;
MatrixForm /@ repC3v
This gives the following results: $$ \left\{\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \\ \end{pmatrix},\begin{pmatrix} \frac{1}{4} & \frac{3}{4} & \frac{\sqrt{3}}{4} \\ \frac{3}{4} & \frac{1}{4} & -\frac{\sqrt{3}}{4} \\ -\frac{\sqrt{3}}{2} & \frac{\sqrt{3}}{2} & -\frac{1}{2} \\ \end{pmatrix},\begin{pmatrix} \frac{1}{4} & \frac{3}{4} & -\frac{\sqrt{3}}{4} \\ \frac{3}{4} & \frac{1}{4} & \frac{\sqrt{3}}{4} \\ \frac{\sqrt{3}}{2} & -\frac{\sqrt{3}}{2} & -\frac{1}{2} \\ \end{pmatrix},\begin{pmatrix} \frac{1}{4} & \frac{3}{4} & \frac{\sqrt{3}}{4} \\ \frac{3}{4} & \frac{1}{4} & -\frac{\sqrt{3}}{4} \\ \frac{\sqrt{3}}{2} & -\frac{\sqrt{3}}{2} & \frac{1}{2} \\ \end{pmatrix},\begin{pmatrix} \frac{1}{4} & \frac{3}{4} & -\frac{\sqrt{3}}{4} \\ \frac{3}{4} & \frac{1}{4} & \frac{\sqrt{3}}{4} \\ -\frac{\sqrt{3}}{2} & \frac{\sqrt{3}}{2} & \frac{1}{2} \\ \end{pmatrix},\begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & -1 \\ \end{pmatrix}\right\} $$
However, if the basis functions are general functions, not monomials, how can I get the coefficient $a_i$ of function $f_i(\mathbf{r})$ in the transformed function $f_k(g^{-1}\mathbf{r})=\sum_i a_i f_i(\mathbf{r})$?
For example, if
F1[{x_, y_}] := (x + I y)^2
F2[{x_, y_}] := (x - I y)^2
F3[{x_, y_}] := (x + y)^2
How can I get the representation matrices under this basis?
NOTE: I mean a general method, not just a transformation from basis f1~f3 to F1~F3.
TeXForm[]
? $\endgroup$