Second Order Probability

The literature has considerable discussion of higher order probabilities and in particular second-order probability, which is the probability that the true probability of something has the value P. For example, if the probability of an event is estimated at 0.8, and the probability of that estimate (i.e. the certainty in that estimate) is say 0.7, then the true or underlying probability excluding uncertainty is something like 0.65 (i.e. approximately 0.8 less half the uncertainty of (1-0.7)/2).

There are a number of philosophical approaches to true probability. One of these approaches is to use conventional statistics. My question is why the analytical Distributions approach described here seems to produce the incorrect result of 0.77, compared to the presumably approximately correct Monte Carlo simulation result of 0.66?

An independent variable is needed to work with Beta distributions, which I have chosen to be the Standard Deviation of 0.05. Another aspect of the analysis will be to investigate sensitivity to the Standard Deviation assumption.

The analytical approach is:

params =
Minimize[(0.7 -
Probability[p <= 0.8,
p \[Distributed]
0.05^2)^2, {\[Alpha], \[Beta]}]
BetaDistribution @@ params[[2, All, 2]] // Mean


which produces the BetaDistribution[53.6059,15.8539] and Mean of 0.7717.

The Monte Carlo approach is:

fn[firstOrderProb_, secondOrderProb_, monteCarloIterations_] :=
Module[{\[Alpha], \[Beta], \[ScriptCapitalD]secondOrderProb,
secondOrderRandomVariable},
\[ScriptCapitalD]secondOrderProb =
Solve[{Mean@# == secondOrderProb, StandardDeviation@# == 0.05} &@
secondOrderRandomVariable = (#/Max@#) &@
RandomVariate[\[ScriptCapitalD]secondOrderProb,
Sqrt@monteCarloIterations];
RandomVariate[BernoulliDistribution[firstOrderProb #],
Sqrt@monteCarloIterations] & /@ secondOrderRandomVariable //
Flatten // Mean // N];
fn[0.8, 0.7, 10^6]
Table[fn[0.8, 0.7, 10^6], 10^2];
Histogram@%
Mean@%%


The simulation histogram is:

Although the simulation histogram is quite volatile with Table size (here 10^2), the CLT mean is relatively stable at about 0.65+/-0.02. The analytical result isn't within the range of the simulation.

A second case of an event probability of say 0.3 and a second order probability of the same 0.7 as above might be considered as well. In this case the analytical and simulation results are 0.275 and 0.249 respectively. Again, the analytical result isn't within the range of the simulation.

Thoughts about the difference between the analytical approach and the Monte Carlo result would be much appreciated.

• I assume what you're describing is analogous (or even equivalent) to a Bayesian having hyper-parameters on a prior distribution. If so, giving the model structure (say, typeset in LaTeX) rather than just the Mathematica code would be helpful.
– JimB
Jul 16, 2023 at 15:51
• JimB ... thanks for your observations, however this context is not Bayesian ... I've reached the conclusion that my question is in the wrong direction ... I've found an appropriate analysis of second order probability in SUNDGREN, David y KARLSSON, Alexander. Uncertainty Levels of Second-Order Probability. Polibits [online]. 2013, n.48, pp.5-11. ISSN 1870-9044.(scielo.org.mx/…) Jul 19, 2023 at 1:13
• There is no such thing as a probability-of-a-probability. There are only statistical models and probability distributions. All missing information is cast in the probability distribution, after the adoption of a model. You may benefit from reading David J. Blower's books. Jul 20, 2023 at 16:46
• Romke ... many thanks for your thoughts ... your view of probability-of-probability relates to one of the philosophical discussions I referred to at the start of my question ... this debate goes back at least to Haim Gaifman in 1986 ... the paper I referenced to JimB addressees issues of entropy similarly to Blower ... despite both the conventional statistics and Subjective Logic camps being entirely Bayesian nowadays, there is still a fairly wide difference in interpretations of things like second order uncertainty ... I will have an interesting journey with Blower thank you Jul 22, 2023 at 0:07
• @RomkeBontekoe ... your recommended reading of some 2,109 pages all up would be better as an appendix for a summary book ... appreciated the discussion of Jayne's point, about which many are oblivious, regarding a failure of maximum entropy prediction (see Volume II, Section 17.6, p15) ... I can't support your recommendation of this material ... I would instead refer readers to other excellent and integrated entropy material, say the 22 pages of Chapter 8 in Koller & Friedman Probabilistic Graphical Models Jul 31, 2023 at 8:57

In answer to my own question:

1. My short-form analytical approach is incorrect for the problem ... it tries to do too much in the one Probability function ... this results in the exactly opposite of what is required ... the mean of the implied probability distribution rises as the probability of this mean being less than a certain amount declines ... which is of course intuitively correct ... Sundgren and Karlsson's 2013 paper (referenced in my comment to JimB) has a much better approach ... second-order statistical uncertainty is derived from Kullback-Leibler divergence of the distribution to the mean (see Note 1 (below) for Mathematica code with some of Sundgren and Karlsson's test cases) ... however the second-order uncertainty for BetaDistributions turns out to be quite small at about 0.02 (or about 10% of the first-order order uncertainty) ... and of course this varies with the assumed standard deviation of the first-order distribution.
2. The Monte Carlo simulation approach in my question remains a point illustration for the background of my question, which is the domain of second-order uncertainty in Subjective Logic ... however this is not really a systematic Subjective Logic approach to adjusting first-order probability for second-order uncertainty ... distributing the first-order mean with random DiracDeltas from the second-order distribution (i.e. the DiracDeltas shift the first-order distribution) in the way I have shown is approximately a convolution of the first-order distribution (although obviously not in the usual sense of the convolution of two distributions where the mean of the resultant distribution is the sum of the means) ... and without normalising the samples, the mean of the resultant probability distribution in the example would just be the first order times the second order probability (i.e. 0.56 = 0.8 x 0.7, in the normal way that the joint probability of independent events is just the product of the probabilities) ... the Subjective Logic paradigm is really incompatible with any conventional statistic analysis in the sense that I have been tentatively investigating in my question ... in the Subjective Logic paradigm, second-order uncertainty is highly significant, usually far more-so than the first-order uncertainty that is inherent in any beta distribution (Subjective logic second order uncertainty is not a standard deviation but rather a constant probability density shared by the states, say yes and no) ... in fact in Subjective Logic we really don't care about first-order uncertainty (standard deviation) because we are using the means as handles of the distributions ... first order standard deviation is always present and really such doesn't matter because we are more concerned with our models being completely wrong-footed (i.e. computational irreducibility where our models can't cover the range of real-world outcomes, for example that we are so busy in the forest that we miss what is happening outside of the forest) ... than with the minor spread in some engineering measurement or the range of an individual expert's opinion (which it is worth noting here that we combine with the opinions of other experts in either a linear opinion pool or in a Bayesian/Markov manner for an independent opinion pool) ... in contrast, in conventional statistical analysis it is completely the other way around and first-order uncertainty is the most important thing as we see in Sundgren and Karlsson (Point 1 above) ... and so important that we can make statements like all the uncertainty/entropy is contained within the first order distribution ... this is unambiguously not the case in Subjective Logic where second-order uncertainty is an estimate of the "unknown unknowns" and "unknown knowns" ... one can think of this as a pie chart (with say 60% confident yes, 20% confident no and 20% "computational irreducibility" or "don't know uncertainty") ... at present the techniques for estimating Subjective Logic second order uncertainty remain more of an art than a science ... however, rarely would the second order uncertainty be less than 0.15 and far higher than this in many practical cases.

Note 1: Mathematica code for Sundgren and Karlsson's 2013 KL method:

fnWd[case_, xi_, px_] := Module[{hx, wh, hx2, ui, wd, di},
hx = # Log[2, #] & /@ xi // -Total /@ # & // N;
wh = px hx // Total // N;
hx2 = # Log[2, #] & /@ px // -Total@# & // N;
ui = px xi // Total@# &;
wd = # Log[2, #/ui] & /@ xi // Total /@ # & // # px & // Total //
N;
di = (Max /@ xi - Min /@ xi) // Total // #/Length@xi & // N;
{case, xi // N, px // N, hx, wh, hx2, ui // N, wd, wh + wd, di}]
{fnWd["B", {{32, 29, 29}, {29, 32, 29}, {29, 29, 32}}/90, {1/3, 1/3,
1/3}],
fnWd["F", {{1, 1, 4}/6, {1, 4, 1}/6}, {3, 1}/4],
fnWd["G", {{4, 1, 1}/6, {1, 1, 1}/3, {1, 1, 2}/4}, {1, 1, 1}/3],
fnWd["J", {{1, 1, 4}/6, {1, 4, 1}/6, {7, 1, 2}/10}, {3, 2, 1}/6]} //
TableForm[#,
TableHeadings -> {None, {"Case", "xi", "px", "hx", "wh", "hx2",
"ui", "wd", "wh+wd", "di"}}] &