David's answer is correct and the one you need to solve your specific problem. I thought nonetheless that it is worth providing some additional information that might help explain how to diagnose similar issues.
Matrix/tensor operations like
Inverse are designed to work with lists, that is, expressions with a
List. It also works with
SparseArray objects. From the documentation:
When its arguments are not lists or sparse arrays,
Dot remains unevaluated.
You can check whether your expressions have compatible
FullForm. It is common for people to use postfix (
//) notation to check this. As you can see, in your version of the code,
a has head
b has head
MatrixForm. So they can't combine.
If you discover you have erroneously created a matrix wrapped in
MatrixForm, you can change it back to a list using
As an aside, I don't see any point assigning the variable
inv to represent the
a. Unless your real problem uses $a'$ more than once (especially if it is expensive to calculate), you can just as easily do:
Note that I have mixed using of
 style notation for pedagogical reasons.