I am trying to detect outliers in a big dataset using the Mahalanobys distance method. To calculate the Mahalanobys distance for a given vector $x$ from a mean vector $\mu$ you have to compute the following quadratic form:

$(x-\mu)^T S^{-1}(x-\mu)$

Where $S^{-1}$ is the inverse of the covariance matrix, or precision matrix. This is all quite simple in Mathematica, the problem is that the covariance matrix is a 592x592 matrix and is ill-conditioned (the condition number is approximately $10^{26}$ and the Mathematica function Inverse issues a warning in this sense) and I often obtain negative values from the quadratic form above, which shouldn't happen if the inversion had been performed correctly since $S^{-1}$ should be positive definite.

I have read that the function LinearSolve can be used to circumvent this problem avoiding the matrix inversion with something like:

m = Covariance[dataset];
ls = LinearSolve[m];
(x-mean) . ls[x-mean];

With this I don't get negative values (but still get the bad conditioned warning). My questions are:

  1. How can I know how big the uncertainty is on the result of this calculation?
  2. Is there a better way to do this? Either a robust algorithm to invert the matrix or a way to avoid it like with LinearSolve would be fine. (Infinite precision calculation is not feasible I think and I have tried up to 70 digit precision without achieving better results)
  3. Do you happen to know other methods to detect outliers in a big dataset that would avoid dealing with the covariance matrix inversion?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.