1
$\begingroup$

I am trying to detect outliers in a big dataset using the Mahalanobys distance method. To calculate the Mahalanobys distance for a given vector $x$ from a mean vector $\mu$ you have to compute the following quadratic form:

$(x-\mu)^T S^{-1}(x-\mu)$

Where $S^{-1}$ is the inverse of the covariance matrix, or precision matrix. This is all quite simple in Mathematica, the problem is that the covariance matrix is a 592x592 matrix and is ill-conditioned (the condition number is approximately $10^{26}$ and the Mathematica function Inverse issues a warning in this sense) and I often obtain negative values from the quadratic form above, which shouldn't happen if the inversion had been performed correctly since $S^{-1}$ should be positive definite.

I have read that the function LinearSolve can be used to circumvent this problem avoiding the matrix inversion with something like:

m = Covariance[dataset];
ls = LinearSolve[m];
(x-mean) . ls[x-mean];

With this I don't get negative values (but still get the bad conditioned warning). My questions are:

  1. How can I know how big the uncertainty is on the result of this calculation?
  2. Is there a better way to do this? Either a robust algorithm to invert the matrix or a way to avoid it like with LinearSolve would be fine. (Infinite precision calculation is not feasible I think and I have tried up to 70 digit precision without achieving better results)
  3. Do you happen to know other methods to detect outliers in a big dataset that would avoid dealing with the covariance matrix inversion?
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.