0
$\begingroup$

I have a list of data: data

I want to find the best fit distribution curve and then fit curve. After finding the fitting distribution, I want to compute the Pearson chi-square to test the fitting goodness.

I wrote the following code:

data = Flatten[
   Table[Import["IMGareacell_" <> ToString[i] <> ".csv", "Data"], {i, 
     5}]];
edist = EstimatedDistribution[data, 
  ParetoDistribution[k, \[Alpha], \[Gamma], \[Mu]]]
\[ScriptCapitalH] = 
      DistributionFitTest[data, edist, "HypothesisTestData"]
\[ScriptCapitalH]["TestDataTable", All]
Show[Histogram[data, {2, 20, 0.4}, "PDF"], 
 Plot[PDF[TruncatedDistribution[{2, 20}, edist], x], {x, 2, 20}, 
  PlotStyle -> Thick, PlotRange -> {0, 2}]]

The fitting curve seems very close to the histogram, however the p value in the testtable is extramely small. I am new to mathematica and really want to know why and slove the problem.

Thank you

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

There are two things going on. First, you have nearly 37,000 data points. With that much data (I assume from the real world) it is extremely unlikely that the data generation process has exactly a Pareto distribution. So you're going to get a small P-value.

Second (and maybe more importantly), it is the choice of bin width that hides the lack of fit. Here is the fit and the histogram with the bin width that you used:

Original histogram

Because of the large amount of data, one needs to use smaller bin widths for a adequate description of the data. Below is a histogram with a smaller bin width that shows where the data departs from a Pareto distribution:

Show[Histogram[data, {2, 10, 0.1}, "PDF"],
 Plot[PDF[edist, x], {x, 2, 10}, PlotStyle -> Thick]]

Histogram and fit

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you, Jim! Yes, I see the problem, so this data cannot find an appropriate distribution curve? $\endgroup$
    – Shuoqi Li
    Jun 1, 2017 at 12:46
  • $\begingroup$ It is simply that the standard distribution curves don't fit the data well. Nothing wrong with the data. If you use SmoothKernelDistribution or EmpiricalDistribution, you can get (within Mathematica) everything that's needed: means, standard deviations, proportion less than 7.6, skewness, etc. If you really need to transport the description outside of Mathematica, then either evaluating the probability density or cumulative distribution function at many points is one possibility. $\endgroup$
    – JimB
    Jun 1, 2017 at 14:10
  • $\begingroup$ If you need a more compact description that is transportable, then maybe using a MixtureDistribution might provide a better fit. $\endgroup$
    – JimB
    Jun 1, 2017 at 14:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.