4
$\begingroup$

Here is my scenario: got some data that I have to fit to a specific parametric family of probability distributions (cannot be changed).

Data consist of several bunches of integers between, say, 0 and 50 and I have to do fitting for each bunch separately.

The main problem is that systematically in each case amount of values in the lower half (roughly between 0 and 25 say) is qualitatively less than in the upper half. As a result, in the bunches with fewer data I simply do not have any values below 25, but I know for sure that probability of getting them is not negligible. On top of that, it is quite likely that the part of the population which might produce such values has been underrepresented (as a result of sampling imperfections).

In the documentation, what I thought might be relevant is

  • to use TruncatedDistribution[{(lowest existing value), ∞}, neededDistribution[parameters]]
  • to use in place of the lowest existing value a variable value as an additional parameter
  • or to use Censored in place of Truncated
  • or to use EventData in some way but I am not sure how

What would be the proper way to do it?

$\endgroup$
5
$\begingroup$

Here are some toy examples:

Approaches usingTruncatedDistribution assuming or deciding on a cut-point:

Testing beta distribution

bd = BetaDistribution[4.2, 2.1]
test = With[{r = RandomVariate[bd, 100000]},
   Pick[r, # > 0.25 & /@ r]];
EstimatedDistribution[test, 
 TruncatedDistribution[{0.25, 1}, BetaDistribution[a, b]]]
FindDistributionParameters[test, 
 TruncatedDistribution[{0.25, 1}, BetaDistribution[a, b]]]

yields

TruncatedDistribution[{0.25, 1}, BetaDistribution[4.16804, 2.08636]]

{a -> 4.16804, b -> 2.08636}

Binomial Distribution

test2 = With[{rv = 
     RandomVariate[BinomialDistribution[50, 0.3], 100000]}, 
   Pick[rv, # > 25 & /@ rv]];
par = FindDistributionParameters[test2, 
  TruncatedDistribution[{25, Infinity}, BinomialDistribution[a, b]]]
Show[Histogram[test2, Automatic, "PDF"], 
 DiscretePlot[
  PDF[TruncatedDistribution[{25, Infinity}, 
        BinomialDistribution[##]] & @@ {Rationalize@#1, #2} & @@ (({a,
         b} /. par)), x], {x, 25, 30}, 
  PlotStyle -> {Thickness[0.04], Red}]]

yields:

{a -> 45., b -> 0.337797}

enter image description here

Poisson Distribution

test3 = With[{r = RandomVariate[PoissonDistribution[25], 10000]}, 
   Pick[r, # > 25 & /@ r]];
pois = FindDistributionParameters[test3, 
  TruncatedDistribution[{25, Infinity}, PoissonDistribution[a]]]
Show[Histogram[test3, Automatic, "PDF"], 
 DiscretePlot[
  PDF[TruncatedDistribution[{25, Infinity}, 
     PoissonDistribution[a]] /. pois, x], {x, 25, 50}, 
  PlotStyle -> {Thickness[0.04], Red}]]

yields:

{a -> 24.9577}

enter image description here

Using All data and MixtureDistribution

md = MixtureDistribution[{0.1, 
    0.9}, {TruncatedDistribution[{0, 25}, PoissonDistribution[25]], 
    TruncatedDistribution[{25, Infinity}, PoissonDistribution[25]]}];
Histogram[test4 = RandomVariate[md, 1000]]
FindDistributionParameters[test4, 
 MixtureDistribution[{a, 
   b}, {TruncatedDistribution[{0, 25}, PoissonDistribution[c]], 
   TruncatedDistribution[{25, Infinity}, PoissonDistribution[c]]}]]

The mixture distribution looks like this:

enter image description here

with estimate mixture and parameter:

{a -> 0.097, b -> 0.903, c -> 24.6988}

I just post this as toy examples. Proper way seems complex to me and also depends on nature of data, reasons missing, ultimate aim but is also beyond me, so I eagerly await expert answers. These toy examples of nice well behaved distributions almost certainly give a false sense cf very complex distributions.

$\endgroup$
4
  • $\begingroup$ So your overall vote is for the first option listed in the question (truncated with fixed cut value)? $\endgroup$ – მამუკა ჯიბლაძე Oct 25 '15 at 10:49
  • 1
    $\begingroup$ @მამუკაჯიბლაძე you know what you are aiming for. There is nothing to prevent looking estimation in multiple ways testing robustness of estimates. I am hopeful that experts on the site will chime in. The order was from simplest to slightly more complex with "nice" distributions..."toys" not an order of preference and not exhaustive. Really a post to illustrate some MMA functionality. Perhaps, after you do some preliminary processing (and I am not implying you have not done that) you will resolve this, Dr Jibladze. $\endgroup$ – ubpdqn Oct 25 '15 at 10:57
  • $\begingroup$ Fair enough. But basically I was just asking whether the fact that you did not consider other options implies that you think censored distributions should be excluded in any case and/or that making the cut threshold an additional variable parameter is not a good idea. $\endgroup$ – მამუკა ჯიბლაძე Oct 25 '15 at 11:45
  • $\begingroup$ @მამუკაჯიბლაძე CensoredDistribution may well be appropriate...it is poverty of time (and expertise) that was limitation. You know the data, its limitations etc. Late in my timezone and on call. Am hopeful that experts will illuminate and edify :-) $\endgroup$ – ubpdqn Oct 25 '15 at 11:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.