# Entropy of a probability distribution (symbolically)

How can I compute the entropy of a probability distribution in Mathematica?

For example,

Entropy[NormalDistribution[m, s]]


does not work, because Entropy does not compute the entropy of a probability distribution.

On the other hand,

Expectation[Log@PDF[NormalDistribution[m, s], x], Distributed[x, NormalDistribution[m, s]]]


takes forever to run, and I don't think it will give an answer. So it would seem Mathematica does not have symbolic rules in place to compute entropies of probability distributions.

Is there a better way to get the formula of the entropy of probability distributions in Mathematica?

• Thank you for your effort: We now have a rather concise definition for forever which appears to be around 160 secs. :)
– gwr
Sep 17, 2017 at 15:55

Timing can be improved with simplification with assumptions:

exp = Expand[
FullSimplify[-Log[PDF[NormalDistribution[m, s], x]],
Assumptions -> {{m, x, s} ∈ Reals, s > 0}]];
Integrate[
exp PDF[NormalDistribution[m, s], x], {x, -∞, ∞},
Assumptions -> {{m, x, s} ∈ Reals, s > 0}] // Timing


yields:

{2.09375, 1/2 (1 + Log[2 π s^2])}


See comment by @gwr below.

Expectation


yields better performance on simplified expression (as 'expected')...too bad I didn't think to use it :(

• Using Expectation here I do get an even more impressive timing, e.g. Expectation[ exp, x \[Distributed] NormalDistribution[m, s], Assumptions -> {{m, x, s} \[Element] Reals, s > 0}] // RepeatedTiming will give 0.0018 secs on my machine.
– gwr
Sep 17, 2017 at 15:46
• @gwr yes I should have tried. It is 'expected' (pardon the pun) given exp is just a constant + scaled mean and scaled second moment, i.e.easy to calculate. Thank you for pointing it out:) Sep 17, 2017 at 20:32

Let $X \sim N(\mu, \sigma^2)$ with pdf $f(x)$: Then, the way we solve this in Chapter 1 of Mathematical Statistics with Mathematica (free download of book available here) is to find $E[-log(f)]$: ... which uses the mathStatica Expect function, and takes about 6 seconds to evaluate on my Mac in 11.2.

I was surprised at the OP's suggestion that:

Expectation[Log@PDF[NormalDistribution[m, s], x],
Distributed[x, NormalDistribution[m, s]]] // AbsoluteTiming


... does not evaluate. When I tried it, it does evaluate, but it takes about 160 seconds to return the equivalent:

{161.93, 1/2 (-1 - Log[2 [Pi] s^2])}

... on the same computer in v11.1.1 and about 80 seconds under 11.2.

• I let it run some more time and it evaluated here too, so my statement was inaccurate. However, it takes a very long time.
– a06e
Sep 16, 2017 at 14:49

I was just recently dealing with this, so I thought I should leave this line here for anyone who winds up finding this question too:

ShannonEntropy[dist_?DistributionParameterQ] :=
-Expectation[LogLikelihood[dist,{x}], x \[Distributed] dist]

AbsoluteTiming[ShannonEntropy[NormalDistribution[]]]
(* {0.122426, 1/2 (1 + Log + Log[\[Pi]])} *)


Additionally, a slightly more thorough version is available here.