I am attempting to become more familiar with the DistributionFitTest, specifically the distributions it yields and their fittedness. What I have found is that when I generate random data, fit it, and analyze it, I can repeat the process exactly and get histograms and fits that look very similar, as expected. But, the P-Values from trial to trial (determined by TestDataTable) range from nearly 0 to almost 1.
By running the code shown below multiple times, I produce histograms and PDF's that are quite similar each time, but whose P-Value table varies wildly (eg. Pearson Chi2 has taken on values from 0.04 to 0.96)
data=RandomVariate[NormalDistribution[],10^5];
fit=DistributionFitTest[data,NormalDistribution[m,s],"HypothesisTestData"];
fit["TestDataTable",All]
Show[Histogram[data,30,"PDF"],Plot[PDF[fit["FittedDistribution"],x],{x,-6,6}]]
My question is why this would happen? Based on the histogram/PDF plot, I would expect the P-Values for any given method to be consistently high, or at least consistent.
To be clear, I am not interested in why the different methods of computing P-Values differ among one another (eg, why the P-Value found from Cramer-von Mises is different than the Pearson Chi2 P-Value). Rather, I am curious as to why executing this code a dozen or so times doesn't return similar values for any particular method. The results from all methods seem to change greatly from trial to trial.
I have perused the Mma documentation of DistributionFitTest and other SE questions concerning this function to no avail. One titled Inconsistent DistributionFitTest results informed me of the different behavior DistributionFitTest has when you specify mu and sigma for the Normal Distribution, but the variability of P-Values continues whether or not I specify my mu/sigma when I create and/or fit the data.
Thank you for your time.
DistributionFitTest
chooses that statistic for your data (as the"AutomaticTest"
property of theHypothesisTestData
reveals). Try running this code:Do[ data=RandomVariate[NormalDistribution[],10^5]; fit=CramerVonMisesTest[data]; Print@fit ,{10}]
and you'll see similar results. $\endgroup$