Yes, you can. This would involve transferring the matrix to Mathematica, invoking the multiplication function, then transferring the result back.
Doings this for the kinds of small (4 by 4 and 3 by 3) matrices that come up in your application area is going to be
Therefore I would strongly discourage doing this. Matrix math is simple and there are lots of excellent libraries for it in all low-level languages. Save Mathematica for calculations that are not easily done in C++.
If you still want to do it, you need to learn about the MathLink.
Please see here and in particular here on how to call Mathematica from C (or C++).
I'd recommend you familiarize yourself with how MathLink works first by playing with it in pure Mathematica code. It's going to be easier to write the C version once you're comfortable with how MathLink the protocol works.