6
$\begingroup$

How can I compute the entropy of a probability distribution in Mathematica?

For example,

Entropy[NormalDistribution[m, s]]

does not work, because Entropy does not compute the entropy of a probability distribution.

On the other hand,

Expectation[Log@PDF[NormalDistribution[m, s], x], Distributed[x, NormalDistribution[m, s]]]

takes forever to run, and I don't think it will give an answer. So it would seem Mathematica does not have symbolic rules in place to compute entropies of probability distributions.

Is there a better way to get the formula of the entropy of probability distributions in Mathematica?

$\endgroup$
  • 2
    $\begingroup$ Thank you for your effort: We now have a rather concise definition for forever which appears to be around 160 secs. :) $\endgroup$ – gwr Sep 17 '17 at 15:55
6
$\begingroup$

Timing can be improved with simplification with assumptions:

exp = Expand[
   FullSimplify[-Log[PDF[NormalDistribution[m, s], x]], 
    Assumptions -> {{m, x, s} ∈ Reals, s > 0}]];
Integrate[
  exp PDF[NormalDistribution[m, s], x], {x, -∞, ∞}, 
  Assumptions -> {{m, x, s} ∈ Reals, s > 0}] // Timing

yields:

{2.09375, 1/2 (1 + Log[2 π s^2])}

See comment by @gwr below.

Expectation 

yields better performance on simplified expression (as 'expected')...too bad I didn't think to use it :(

| improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Using Expectation here I do get an even more impressive timing, e.g. Expectation[ exp, x \[Distributed] NormalDistribution[m, s], Assumptions -> {{m, x, s} \[Element] Reals, s > 0}] // RepeatedTiming will give 0.0018 secs on my machine. $\endgroup$ – gwr Sep 17 '17 at 15:46
  • $\begingroup$ @gwr yes I should have tried. It is 'expected' (pardon the pun) given exp is just a constant + scaled mean and scaled second moment, i.e.easy to calculate. Thank you for pointing it out:) $\endgroup$ – ubpdqn Sep 17 '17 at 20:32
5
$\begingroup$

Let $X \sim N(\mu, \sigma^2)$ with pdf $f(x)$:

enter image description here

Then, the way we solve this in Chapter 1 of Mathematical Statistics with Mathematica (free download of book available here) is to find $E[-log(f)]$:

enter image description here

... which uses the mathStatica Expect function, and takes about 6 seconds to evaluate on my Mac in 11.2.

I was surprised at the OP's suggestion that:

Expectation[Log@PDF[NormalDistribution[m, s], x], 
Distributed[x, NormalDistribution[m, s]]] // AbsoluteTiming

... does not evaluate. When I tried it, it does evaluate, but it takes about 160 seconds to return the equivalent:

{161.93, 1/2 (-1 - Log[2 [Pi] s^2])}

... on the same computer in v11.1.1 and about 80 seconds under 11.2.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I let it run some more time and it evaluated here too, so my statement was inaccurate. However, it takes a very long time. $\endgroup$ – becko Sep 16 '17 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.