Bug introduced in 8.0 or earlier and fixed in 11.2.0

CorrelationDistance[{-53.0, 4.3}, {-23.0, 5.4}] is a negative number. It is close to zero, to be sure, but negative. Shouldn't all distance functions guarantee a non-negative result?

  • $\begingroup$ CorrelationDistance[{-53.0, 4.3},{-23.0, 5.4}] Does that help? $\endgroup$ Nov 8 '17 at 0:05
  • 1
    $\begingroup$ Use exact numbers to get an exact result: CorrelationDistance @@ Rationalize[{{-53.0, 4.3}, {-23.0, 5.4}}, 0] $\endgroup$
    – Bob Hanlon
    Nov 8 '17 at 13:06

This is again a situation where one needs to use a stable formula for computations.

Let's look at the result of CorrelationDistance[] again for reference:

v1 = {-53.0, 4.3}; v2 = {-23.0, 5.4};

CorrelationDistance[v1, v2] // InputForm

Note that there is a simple relationship between CorrelationDistance[] and CosineDistance[]:

CosineDistance[v1 - Mean[v1], v2 - Mean[v2]] // InputForm

It's the same result. Let's look at the result of using explicit formulae:

c1 = v1 - Mean[v1]; c2 = v2 - Mean[v2];
1 - c1.c2/(Norm[c1] Norm[c2])

1 - Normalize[v1 - Mean[v1]].Normalize[v2 - Mean[v2]]

The second explicit formula gives the correct result.

Still, the need to subtract two quantities that are very nearly equal should give one much reluctance. Thus, here is a stable algorithm, derived from work by Velvel Kahan:

cosDistance[v1_?VectorQ, v2_?VectorQ] := 
   Module[{n1 = Normalize[v1], n2 = Normalize[v2], y},
          y = Norm[n1 - n2]^2; 2 y/(Norm[n1 + n2]^2 + y)]

correlationDistance[v1_?VectorQ, v2_?VectorQ] :=
cosDistance[v1 - Mean[v1], v2 - Mean[v2]]

and thus

correlationDistance[v1, v2]
  • $\begingroup$ (This is somewhat related.) $\endgroup$
    – J. M.'s torpor
    Nov 15 '17 at 10:38

The expected result is exactly zero.

Simplify[CorrelationDistance[{x, y} , {p, q}],
 Assumptions -> {Element[{x, y, p, q}, Reals], x < y, p < q}]


The result , -2.22045*10^-16 is zero to within machine precision. Use Chop if you like.

  • 1
    $\begingroup$ It will cause problems when used with some functions (FindClusters) that expect a strictly non-negative value though. For this reason I would consider it a bug, or at least worth reporting. $\endgroup$
    – Szabolcs
    Nov 7 '17 at 16:23
  • 1
    $\begingroup$ Another problem is that it may return a Complex result with complex input. The imaginary part is 0., but the head is Complex, which may again cause trouble. $\endgroup$
    – Szabolcs
    Nov 7 '17 at 16:26
  • $\begingroup$ Another point of view is that if we were to supply symbolic input, then substitute machine numbers for the symbolic result, then the same thing could happen. There is no way around this. (I would still expect the function to give a strictly non-negative result when given machine precision input.) $\endgroup$
    – Szabolcs
    Nov 7 '17 at 16:28
  • $\begingroup$ obviously there is a performance trade if you want to handle the special case. $\endgroup$
    – george2079
    Nov 7 '17 at 16:55
  • $\begingroup$ Isn't that negligible though? Suppose this function is internally optimized for machine numbers. The check for a negative value (and replacing it with 0.0) would likely take less time than a C-language function call (!), not to mention just initiating a Mathematica evaluation. $\endgroup$
    – Szabolcs
    Nov 7 '17 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.