I have two data sets. When I do a QuantilePlot[data1, data2], I get the graph below. However, KolmogorovSmirnovTest[data1, data2] returns 0. While clearly the distributions of the two data sets are somewhat different, it is equally clear that they are not that far apart. Now, to add to the confusion, while the examples for KolmogorovSmirnovTest[] do give an example of two data sets being compared, the actual documentation seems to tell us how to see if data matches a symbolic distribution. So, it all seems rather goofy. Anybody have thoughts?

EDIT I withdraw the "bug" theory, since I have now compared the same data in Matlab (thanks to @Szabolcs's excellent MATLink (matlink.org)) and MATLAB is in agreement with Mathematica (except that Mathematica's 0 is MATLAB's 1)

enter image description here

  • 1
    $\begingroup$ I suspect the answer is simply that KolmogorovSmirnovTest goes very quickly to zero for distributions the "look" fairly similar, especially if you have a large number of points. $\endgroup$ – george2079 Sep 10 '14 at 20:56
  • $\begingroup$ @george2079 that's certainly possible, though the qq plot seems to indicate that the distributions are largely the same away from the tails, which I would think would making them somewhat K-S friendly. But I really don't know enough about the subject (though trying to learn). $\endgroup$ – Igor Rivin Sep 10 '14 at 21:04
  • $\begingroup$ @george2079 By the way, the data sets had 100000 and 200000 points, for what it's worth, and the K-S value was EXACTLY 0., no fuzz at all, which seems like a possible bug. $\endgroup$ – Igor Rivin Sep 10 '14 at 21:05
  • 1
    $\begingroup$ Regarding the name of this thread, I will mention that 6 out of every 5 people suffer from statistics confusion. $\endgroup$ – Daniel Lichtblau Sep 12 '14 at 19:29

The KS test looks for the maximum absolute deviation between empirical cdfs and so it is sensitive to departure in distribution anywhere.

In your case there is very significant deviation in the tails. The null hypothesis is that the distributions are equivalent and, since the empirical distribution should well represent the underlying population distribution with large sample sizes it is highly unlikely that you would see a deviation of that magnitude due to chance.

For p-values the KS test in Mathematica only supports machine precision and the true p-value is likely smaller than can be represented by machine numbers. This is why you are seeing a zero rather than some absurdly tiny number.

  • $\begingroup$ Thanks! In my case, I have two families of distributions converging to (putatively) the same distribution, and I want to see if this is true, so am comparing two distributions far down the sequence in both cases, and seeing that they are NOT the same, which is not very useful somehow. Do you have any idea for the right thing? Maybe some sort of Kullback-Leibler thing? $\endgroup$ – Igor Rivin Sep 12 '14 at 19:36
  • $\begingroup$ The main problem with using a hypothesis test for this is that they all become increasingly sensitive to even small deviations with sample size. You at least want one that looks at the bigger picture like Cramer Von Mises which integrates along the whole cdf. $\endgroup$ – Andy Ross Sep 12 '14 at 19:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.