Proper command to fit a distribution with data incomplete in a known way

Question

Here is my scenario: got some data that I have to fit to a specific parametric family of probability distributions (cannot be changed).

Data consist of several bunches of integers between, say, 0 and 50 and I have to do fitting for each bunch separately.

The main problem is that systematically in each case amount of values in the lower half (roughly between 0 and 25 say) is qualitatively less than in the upper half. As a result, in the bunches with fewer data I simply do not have any values below 25, but I know for sure that probability of getting them is not negligible. On top of that, it is quite likely that the part of the population which might produce such values has been underrepresented (as a result of sampling imperfections).

In the documentation, what I thought might be relevant is

• to use TruncatedDistribution[{(lowest existing value), ∞}, neededDistribution[parameters]]
• to use in place of the lowest existing value a variable value as an additional parameter
• or to use Censored in place of Truncated
• or to use EventData in some way but I am not sure how

What would be the proper way to do it?

Answer 1

Here are some toy examples:

Approaches usingTruncatedDistribution assuming or deciding on a cut-point:

Testing beta distribution

bd = BetaDistribution[4.2, 2.1]
test = With[{r = RandomVariate[bd, 100000]},
   Pick[r, # > 0.25 & /@ r]];
EstimatedDistribution[test, 
 TruncatedDistribution[{0.25, 1}, BetaDistribution[a, b]]]
FindDistributionParameters[test, 
 TruncatedDistribution[{0.25, 1}, BetaDistribution[a, b]]]

yields

TruncatedDistribution[{0.25, 1}, BetaDistribution[4.16804, 2.08636]]

{a -> 4.16804, b -> 2.08636}

Binomial Distribution

test2 = With[{rv = 
     RandomVariate[BinomialDistribution[50, 0.3], 100000]}, 
   Pick[rv, # > 25 & /@ rv]];
par = FindDistributionParameters[test2, 
  TruncatedDistribution[{25, Infinity}, BinomialDistribution[a, b]]]
Show[Histogram[test2, Automatic, "PDF"], 
 DiscretePlot[
  PDF[TruncatedDistribution[{25, Infinity}, 
        BinomialDistribution[##]] & @@ {Rationalize@#1, #2} & @@ (({a,
         b} /. par)), x], {x, 25, 30}, 
  PlotStyle -> {Thickness[0.04], Red}]]

yields:

{a -> 45., b -> 0.337797}

Poisson Distribution

test3 = With[{r = RandomVariate[PoissonDistribution[25], 10000]}, 
   Pick[r, # > 25 & /@ r]];
pois = FindDistributionParameters[test3, 
  TruncatedDistribution[{25, Infinity}, PoissonDistribution[a]]]
Show[Histogram[test3, Automatic, "PDF"], 
 DiscretePlot[
  PDF[TruncatedDistribution[{25, Infinity}, 
     PoissonDistribution[a]] /. pois, x], {x, 25, 50}, 
  PlotStyle -> {Thickness[0.04], Red}]]

yields:

{a -> 24.9577}

Using All data and MixtureDistribution

md = MixtureDistribution[{0.1, 
    0.9}, {TruncatedDistribution[{0, 25}, PoissonDistribution[25]], 
    TruncatedDistribution[{25, Infinity}, PoissonDistribution[25]]}];
Histogram[test4 = RandomVariate[md, 1000]]
FindDistributionParameters[test4, 
 MixtureDistribution[{a, 
   b}, {TruncatedDistribution[{0, 25}, PoissonDistribution[c]], 
   TruncatedDistribution[{25, Infinity}, PoissonDistribution[c]]}]]

The mixture distribution looks like this:

with estimate mixture and parameter:

{a -> 0.097, b -> 0.903, c -> 24.6988}

I just post this as toy examples. Proper way seems complex to me and also depends on nature of data, reasons missing, ultimate aim but is also beyond me, so I eagerly await expert answers. These toy examples of nice well behaved distributions almost certainly give a false sense cf very complex distributions.