0
$\begingroup$

Here is a sample of data which corresponds to the following histogram

data = Flatten[Import["hist.dat", "Table"], 1];

P00 = Histogram[data, Automatic, "Probability", ChartStyle -> Gray, 
      ChartBaseStyle -> EdgeForm[None], Frame -> True, 
      FrameLabel -> {"N", "P"}, RotateLabel -> False, 
      LabelStyle -> Directive[FontFamily -> "Helvetica", 20], 
      Epilog -> {Red, Thick, Dashed, 
      Line[{{#, 0}, {#, 1}} &@Last@Commonest[data]]}, PlotRange -> All, 
      PlotRangePadding -> 0.001, AspectRatio -> 1, ImageSize -> 550]

enter image description here

Now I want to find the best possible probability distribution function (PDF). So let's try some of the most well-known PDFs:

1. Laplace distribution:

fit1 = FindDistributionParameters[Flatten@data, LaplaceDistribution[m, b]]
m0 = m /. fit1[[1]];
b0 = b /. fit1[[2]]; 
t1 = Plot[PDF[LaplaceDistribution[m0, b0], x], {x, 0, 30}, 
     PlotStyle -> {Magenta, Thick}, PlotRange -> All];
Show[{P00, t1}, Axes -> False]

enter image description here

As we can see, the right-hand side of the distribution is acceptable but the left-hand side is not.

2. Maxwell distribution:

fit2 = FindDistributionParameters[Flatten@data, MaxwellDistribution[m]]
m1 = m /. fit2[[1]];
t2 = Plot[PDF[MaxwellDistribution[m1], x], {x, 0, 30}, PlotStyle -> {Magenta, Thick}];
Show[{P00, t2}, Axes -> False]

enter image description here

3. Reyleigh distribution:

fit3 = FindDistributionParameters[Flatten@data, RayleighDistribution[m]]
m2 = m /. fit3[[1]];
t3 = Plot[PDF[RayleighDistribution[m2], x], {x, 0, 30}, PlotStyle -> {Magenta, Thick}];
Show[{P00, t3}, Axes -> False]

enter image description here

We observe that the last two types of distribution do not fit well the data. In particular, they do not go high enough.

My question: Is there a way to fix/modify/improve these types of PDFs in order to achieve the best possible fit in my data? Moreover, is there any other type of PDF which might work better in this case?

Many thanks in advance!

Improve this question
$\endgroup$
6
  • $\begingroup$ "Why?" Because that would be an even worse fit, the distributions tie height with width. You can't just guess at the distributions, surely you have an idea which one would fit before you run the experiment? $\endgroup$
    – Feyre
    Commented Oct 28, 2016 at 8:39
  • $\begingroup$ Have you tried FindDistribution? $\endgroup$
    – corey979
    Commented Oct 28, 2016 at 9:39
  • 1
    $\begingroup$ If your integer data consists of counts, then attempting to fit continuous distributions will never get you the "correct" distribution (which is taken directly from your question). If your integer data is simply rounded, then treating your data as if it were continuous is not the best way to estimate the parameters in an underlying continuous distribution. So...what is the generating process of your data? As has been suggested by others in your previous question, are there not candidate distributions suggested or implied by your experiment? $\endgroup$
    – JimB
    Commented Oct 28, 2016 at 15:19
  • 1
    $\begingroup$ Your data is clearly discrete (iintegers going from 4 to 29) and so fitting a continuous distribution is not likely a great approach. If you could mention something about the data generation process, that would help. For instance, are these numbers counts? Is there just a lot of round-off that results in integer values? And why you think you need to estimate a probability density or probability mass function, would also be helpful. In other words, what would having a function with a few parameters benefit you over just showing the histogram/frequency table? $\endgroup$
    – JimB
    Commented Dec 8, 2018 at 17:46
  • 1
    $\begingroup$ I don't know why but I felt if I repeated myself from 2 years ago, it might have an effect. And note that whatever you end up with, it won't be "correct" (as requested in the title). It will be (hopefully) a reasonably parsimonious description of the data (no more, no less). $\endgroup$
    – JimB
    Commented Dec 8, 2018 at 17:50

1 Answer 1

Reset to default
3
$\begingroup$

From a statistical point of view, the question is not well-posed. If you insist on finding some distribution, then use FindDistribution:

data = Import["hist.dat", "List"];
th = Last @ Commonest[data];
data = Select[data, # >= th &]; (*because in a previous question you were interested only in these data *)

dist = FindDistribution[data]

MixtureDistribution[{0.859154, 0.140846}, {PascalDistribution[6, 0.834241], PoissonDistribution[12.6456]}]

t1 = Plot[PDF[dist, x], {x, th, 30}, PlotStyle -> {Black, Thick}, 
   PlotRange -> All];
Show[P00, t1, Axes -> False]

enter image description here

Improve this answer
$\endgroup$
4
  • $\begingroup$ FindDistribution is not recognized in version 9 :( Is there a way to obtain the analytical expression of dist? $\endgroup$
    – Vaggelis_Z
    Commented Oct 28, 2016 at 12:56
  • $\begingroup$ You can take the dist from the answer (can take also general parameters of the distributions) and PDF[dist, x] will give an analytical formula. $\endgroup$
    – corey979
    Commented Oct 28, 2016 at 13:01
  • 1
    $\begingroup$ Are there any references as to how exactly FindDistribution works? The entry in Wolfram Reference does not have much information besides the fact there is a Bayesian approach behind the scenes. $\endgroup$
    – Skumin
    Commented Sep 21, 2017 at 8:51
  • $\begingroup$ The OP specifies that he is looking for a pdf (which stands for probability density function - not prob distribution function). By standard statistical nomenclature, this means s/he is looking for a continuous distribution, whereas that provided above is discrete. Perhaps the OP could clarify what s/he wants ... but if the OP is after a discrete distribution, the correct expression is pmf (not pdf). $\endgroup$
    – wolfies
    Commented Dec 8, 2018 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.