Does re-creating the original data yield the same result?
What you are doing would indeed be a very intuitive way of getting the same result as with the original data. This follows from the definition of the Kolmogorov-Smirnov test statistic which is the maximum difference between the empirical distribution functions $F$ of the two data sets. $F$ is defined as:
$F_n(x)=\frac{\sum _{i=1}^n I_{X_i\leq x}}{n}$
where $X_i$ is the $i$-th data element and $I_{X_i\leq x}$ is the indicator function which is 1 if $X_i\leq x$ is true and 0 otherwise. Crucial here is that in the calculation of $F$ the data elements are ordered. This makes your reconstruction method possible, because there's no difference between the original unordered data set (that you don't know) and your reconstructed ordered data set as far as the KS statistic is concerned.
Let's test this.
Generate three sample test sets, two with an identical distribution and a third with a slightly different distribution:
d1 = RandomVariate[TruncatedDistribution[{0, 6}, PoissonDistribution[3]], 1000];
d2 = RandomVariate[TruncatedDistribution[{0, 6}, PoissonDistribution[3]], 1000];
d3 = RandomVariate[TruncatedDistribution[{0, 6}, PoissonDistribution[3.4]], 1000];
The test results:
KolmogorovSmirnovTest[d1, d2]
0.9882610776
KolmogorovSmirnovTest[d1, d3]
0.0224637083
Turn the data into frequencies:
f1 = (Sort@Tally[d1])[[All, 2]]
f2 = (Sort@Tally[d2])[[All, 2]]
f3 = (Sort@Tally[d3])[[All, 2]]
Turn this, using your procedure, into the original (but sorted) data:
dd1 = Flatten@MapIndexed[ConstantArray[#2[[1]], #1] &, f1];
dd2 = Flatten@MapIndexed[ConstantArray[#2[[1]], #1] &, f2];
dd3 = Flatten@MapIndexed[ConstantArray[#2[[1]], #1] &, f3];
I used a more functional approach to re-create the data, but in principle it is the same as yours.
As you can see, the test result are the same:
KolmogorovSmirnovTest[dd1, dd2]
0.9882610776
KolmogorovSmirnovTest[dd1, dd3]
0.0224637083
Two sample Kolmogorov-Smirnov test on discrete data
The above treatment assumed that it is correct to perform the KS test on discrete data sets (with lots of ties). The documentation, however, states that KS is intended for continuous data.
Using the KS function as it is would yield the following on the data you provided:
f1 = {100, 200, 150, 100, 50};
f2 = {110, 195, 160, 105, 40};
dd1 = Flatten@MapIndexed[ConstantArray[#2[[1]], #1] &, f1];
dd2 = Flatten@MapIndexed[ConstantArray[#2[[1]], #1] &, f2];
KolmogorovSmirnovTest[dd1, dd2]
0.9999803563
A better test would be a Monte Carlo simulation. KolmogorovSmirnovTest
has this built-in:
KolmogorovSmirnovTest[dd1, dd2, Method -> "MonteCarlo"]
0.859
What it does is presumably something like the following:
Pool the two data sets. $H_0$ is that the two sets are from the same distribution; this one:
pooled = Join[dd1, dd2];
Calculate the KS statistic for the original sets:
uniqueDataValues = Union[pooled];
da1 = #/#[[-1]] &[Accumulate[Count[dd1, #] & /@ uniqueDataValues]];
da2 = #/#[[-1]] &[Accumulate[Count[dd2, #] & /@ uniqueDataValues]];
ksstat = Max[Abs[da1 - da2]]
Now randomly draw new sets from the pooled set and calculated the new statistic. Count how often its value is greater or equal to the original value. This gives the p-value:
Table[
dr = RandomSample[pooled, Length@pooled];
uniqueDataValues = Union[dr];
dr1 = #/#[[-1]] &[Accumulate[Count[Take[dr, Length@dd1], #] & /@ uniqueDataValues]];
dr2 = #/#[[-1]] &[Accumulate[Count[Drop[dr, Length@dd1], #] & /@ uniqueDataValues]];
Boole[Max[Abs[dr1 - dr2]] >= ksstat],
{100000}
] // Mean // N
0.85361
Corresponds nicely to the built-in test value (and the one in Ray's answer). I note that Ray's answer is very slow for large datasets as its complexity is $O[n_1\times n_2]$. The built-in MonteCarlo method seems to be slower as well (for the same amount of samples specified using Method -> {"MonteCarlo", "MonteCarloSamples" -> 100000}
).
KolmogorovSmirnovTest[]
? $\endgroup$ – stathisk Oct 16 '13 at 8:12