3
$\begingroup$

I am attempting to become more familiar with the DistributionFitTest, specifically the distributions it yields and their fittedness. What I have found is that when I generate random data, fit it, and analyze it, I can repeat the process exactly and get histograms and fits that look very similar, as expected. But, the P-Values from trial to trial (determined by TestDataTable) range from nearly 0 to almost 1.

By running the code shown below multiple times, I produce histograms and PDF's that are quite similar each time, but whose P-Value table varies wildly (eg. Pearson Chi2 has taken on values from 0.04 to 0.96)

data=RandomVariate[NormalDistribution[],10^5];  

fit=DistributionFitTest[data,NormalDistribution[m,s],"HypothesisTestData"];  

fit["TestDataTable",All]  

Show[Histogram[data,30,"PDF"],Plot[PDF[fit["FittedDistribution"],x],{x,-6,6}]]

My question is why this would happen? Based on the histogram/PDF plot, I would expect the P-Values for any given method to be consistently high, or at least consistent.

To be clear, I am not interested in why the different methods of computing P-Values differ among one another (eg, why the P-Value found from Cramer-von Mises is different than the Pearson Chi2 P-Value). Rather, I am curious as to why executing this code a dozen or so times doesn't return similar values for any particular method. The results from all methods seem to change greatly from trial to trial.

I have perused the Mma documentation of DistributionFitTest and other SE questions concerning this function to no avail. One titled Inconsistent DistributionFitTest results informed me of the different behavior DistributionFitTest has when you specify mu and sigma for the Normal Distribution, but the variability of P-Values continues whether or not I specify my mu/sigma when I create and/or fit the data.

Thank you for your time.

Improve this question
$\endgroup$
3
  • $\begingroup$ This is just natural variability in the computed CramerVonMises statistic. DistributionFitTest chooses that statistic for your data (as the "AutomaticTest" property of the HypothesisTestData reveals). Try running this code: Do[ data=RandomVariate[NormalDistribution[],10^5]; fit=CramerVonMisesTest[data]; Print@fit ,{10}] and you'll see similar results. $\endgroup$
    – Stefan R
    Jun 18, 2015 at 21:01
  • $\begingroup$ @StefanR Thanks for the code, that's a much more compact way of describing the situation. My question then boils down to how, with 100,000 data points, can the P-Value range from nearly perfect to quite the opposite? It seems like an awful lot of natural variation. Or am I misinterpreting how well these fit vis a vie the P-Value? Better yet, how can I use Mathematica to consistently find a fit for data that matches well. $\endgroup$
    – Marten Thompson
    Jun 18, 2015 at 21:34
  • 3
    $\begingroup$ Under the null hypothesis the p-value has a uniform distribution on (0,1). So, you would expect 5% of the values to fall at or below 0.05 for example. $\endgroup$
    – Andy Ross
    Jun 18, 2015 at 21:43

1 Answer 1

Reset to default
9
$\begingroup$

As I said in the comments, under the null hypothesis (in this case that the data was drawn from a particular distribution family) the p-value should follow a uniform distribution on (0,1). Let me illustrate with a simple z-test.

ztest[data_, mu0_, sigma_] := Block[{z, p, d},
  z = (Mean[data] - mu0)/(sigma/Sqrt[Length[data]]);
  d = NormalDistribution[];
  p = 2*Min[{CDF[d, z], SurvivalFunction[d, z]}];
  {z, p}
  ]

We will generate some data that follows a normal distribution with given mean and standard deviation...

mu = 2.;
sigma = 3;
rvs = RandomVariate[NormalDistribution[mu, sigma], {10000, 100}];

We can test against a null distribution with the same mean and standard deviation...

{z0, p0} = Transpose[ztest[#, mu, sigma] & /@ rvs];

And then against an alternative distribution...

{z1, p1} = Transpose[ztest[#, mu + .5, sigma] & /@ rvs];

Notice that the distribution of the test statistic differs whether the null is true or not.

Histogram[{z0, z1}, ChartLegends -> {"Null", "Alt"}]

enter image description here

Under the null, the p-value is uniformly distributed but under the alternative it is skewed towards smaller values.

Histogram[{p0, p1}, ChartLegends -> {"Null", "Alt"}]

enter image description here

Improve this answer
$\endgroup$
2
  • 3
    $\begingroup$ This is always surprising to undergrad students, and sadly it seems it's often surprising to post-doc researchers.... +1 $\endgroup$
    – ciao
    Jun 18, 2015 at 22:12
  • $\begingroup$ @Andy Thank you for your help, your example made it very clear. $\endgroup$
    – Marten Thompson
    Jun 19, 2015 at 21:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.