KolmogorovSmirnovTest[] On Frequency Data

Question

I have two observed frequency data, let say

f1 = {100, 200, 150, 100, 50};
f2 = {110, 195, 160, 105, 40};

and i want to know whether these random samples belong to the same distribution using KolmogorovSmirnovTest[]. Since KolmogorovSmirnovTest[] takes raw data, i simulate them based on f1 and f2.

data1 = {};
data2 = {};
k = 1;
Do[data1 = Join[data1, Table[k n, {f1[[n]]}]], {n, 1, Length[f1]}]
Do[data2 = Join[data2, Table[k n, {f2[[n]]}]], {n, 1, Length[f2]}]

The test gives the result

KolmogorovSmirnovTest[data1, data2]
0.99998

Is this correct procedure to perform KS on frequency data? I have changed k with any other number and got the same result. Thank you.

Answer 1

Does re-creating the original data yield the same result?

What you are doing would indeed be a very intuitive way of getting the same result as with the original data. This follows from the definition of the Kolmogorov-Smirnov test statistic which is the maximum difference between the empirical distribution functions $F$ of the two data sets. $F$ is defined as:

$F_n(x)=\frac{\sum _{i=1}^n I_{X_i\leq x}}{n}$

where $X_i$ is the $i$-th data element and $I_{X_i\leq x}$ is the indicator function which is 1 if $X_i\leq x$ is true and 0 otherwise. Crucial here is that in the calculation of $F$ the data elements are ordered. This makes your reconstruction method possible, because there's no difference between the original unordered data set (that you don't know) and your reconstructed ordered data set as far as the KS statistic is concerned.

Let's test this.

Generate three sample test sets, two with an identical distribution and a third with a slightly different distribution:

d1 = RandomVariate[TruncatedDistribution[{0, 6}, PoissonDistribution[3]], 1000];
d2 = RandomVariate[TruncatedDistribution[{0, 6}, PoissonDistribution[3]], 1000];
d3 = RandomVariate[TruncatedDistribution[{0, 6}, PoissonDistribution[3.4]], 1000];

The test results:

KolmogorovSmirnovTest[d1, d2]

0.9882610776

KolmogorovSmirnovTest[d1, d3]

0.0224637083

Turn the data into frequencies:

f1 = (Sort@Tally[d1])[[All, 2]]
f2 = (Sort@Tally[d2])[[All, 2]]
f3 = (Sort@Tally[d3])[[All, 2]]

Turn this, using your procedure, into the original (but sorted) data:

dd1 = Flatten@MapIndexed[ConstantArray[#2[[1]], #1] &, f1];
dd2 = Flatten@MapIndexed[ConstantArray[#2[[1]], #1] &, f2];
dd3 = Flatten@MapIndexed[ConstantArray[#2[[1]], #1] &, f3];

I used a more functional approach to re-create the data, but in principle it is the same as yours.

As you can see, the test result are the same:

KolmogorovSmirnovTest[dd1, dd2]

0.9882610776

KolmogorovSmirnovTest[dd1, dd3]

0.0224637083

Two sample Kolmogorov-Smirnov test on discrete data

The above treatment assumed that it is correct to perform the KS test on discrete data sets (with lots of ties). The documentation, however, states that KS is intended for continuous data.

Using the KS function as it is would yield the following on the data you provided:

f1 = {100, 200, 150, 100, 50};
f2 = {110, 195, 160, 105, 40};

dd1 = Flatten@MapIndexed[ConstantArray[#2[[1]], #1] &, f1];
dd2 = Flatten@MapIndexed[ConstantArray[#2[[1]], #1] &, f2];

KolmogorovSmirnovTest[dd1, dd2]

0.9999803563

A better test would be a Monte Carlo simulation. KolmogorovSmirnovTest has this built-in:

KolmogorovSmirnovTest[dd1, dd2, Method -> "MonteCarlo"]

0.859

What it does is presumably something like the following:

Pool the two data sets. $H_0$ is that the two sets are from the same distribution; this one:

pooled = Join[dd1, dd2]; 

Calculate the KS statistic for the original sets:

uniqueDataValues = Union[pooled];
da1 = #/#[[-1]] &[Accumulate[Count[dd1, #] & /@ uniqueDataValues]];
da2 = #/#[[-1]] &[Accumulate[Count[dd2, #] & /@ uniqueDataValues]];    
ksstat = Max[Abs[da1 - da2]]

Now randomly draw new sets from the pooled set and calculated the new statistic. Count how often its value is greater or equal to the original value. This gives the p-value:

Table[
   dr = RandomSample[pooled, Length@pooled];
   uniqueDataValues = Union[dr];
   dr1 = #/#[[-1]] &[Accumulate[Count[Take[dr, Length@dd1], #] & /@ uniqueDataValues]];
   dr2 = #/#[[-1]] &[Accumulate[Count[Drop[dr, Length@dd1], #] & /@ uniqueDataValues]];
   Boole[Max[Abs[dr1 - dr2]] >= ksstat],
   {100000}
 ] // Mean // N

0.85361

Corresponds nicely to the built-in test value (and the one in Ray's answer). I note that Ray's answer is very slow for large datasets as its complexity is $O[n_1\times n_2]$. The built-in MonteCarlo method seems to be slower as well (for the same amount of samples specified using Method -> {"MonteCarlo", "MonteCarloSamples" -> 100000}).

Answer 2

The K-S test is appropriate here if the five categories are ordered. Then the exact p-value, conditional on the distribution of ties in the pooled data, is .8535 (rounded to 4 places). See here for background and code, and here for a corrected version that fixes a special-case error noted by the OP.