8
$\begingroup$

How can I compute the entropy of a probability distribution in Mathematica?

For example, 

Entropy[NormalDistribution[m, s]]

does not work, because Entropy does not compute the entropy of a probability distribution.

On the other hand,

Expectation[Log@PDF[NormalDistribution[m, s], x], Distributed[x, NormalDistribution[m, s]]]

takes forever to run, and I don't think it will give an answer. So it would seem Mathematica does not have symbolic rules in place to compute entropies of probability distributions.

Is there a better way to get the formula of the entropy of probability distributions in Mathematica?

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Thank you for your effort: We now have a rather concise definition for forever which appears to be around 160 secs. :) $\endgroup$
    – gwr
    Sep 17, 2017 at 15:55

3 Answers 3

Reset to default
7
$\begingroup$

Timing can be improved with simplification with assumptions:

exp = Expand[
   FullSimplify[-Log[PDF[NormalDistribution[m, s], x]], 
    Assumptions -> {{m, x, s} ∈ Reals, s > 0}]];
Integrate[
  exp PDF[NormalDistribution[m, s], x], {x, -∞, ∞}, 
  Assumptions -> {{m, x, s} ∈ Reals, s > 0}] // Timing

yields:

{2.09375, 1/2 (1 + Log[2 π s^2])}

See comment by @gwr below.

Expectation

yields better performance on simplified expression (as 'expected')...too bad I didn't think to use it :(

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Using Expectation here I do get an even more impressive timing, e.g. Expectation[ exp, x \[Distributed] NormalDistribution[m, s], Assumptions -> {{m, x, s} \[Element] Reals, s > 0}] // RepeatedTiming will give 0.0018 secs on my machine. $\endgroup$
    – gwr
    Sep 17, 2017 at 15:46
  • $\begingroup$ @gwr yes I should have tried. It is 'expected' (pardon the pun) given exp is just a constant + scaled mean and scaled second moment, i.e.easy to calculate. Thank you for pointing it out:) $\endgroup$
    – ubpdqn
    Sep 17, 2017 at 20:32
6
$\begingroup$

Let $X \sim N(\mu, \sigma^2)$ with pdf $f(x)$:

enter image description here

Then, the way we solve this in Chapter 1 of Mathematical Statistics with Mathematica (free download of book available here) is to find $E[-log(f)]$:

enter image description here

... which uses the mathStatica Expect function, and takes about 6 seconds to evaluate on my Mac in 11.2.

I was surprised at the OP's suggestion that:

Expectation[Log@PDF[NormalDistribution[m, s], x], 
Distributed[x, NormalDistribution[m, s]]] // AbsoluteTiming

... does not evaluate. When I tried it, it does evaluate, but it takes about 160 seconds to return the equivalent:

{161.93, 1/2 (-1 - Log[2 [Pi] s^2])}

... on the same computer in v11.1.1 and about 80 seconds under 11.2.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I let it run some more time and it evaluated here too, so my statement was inaccurate. However, it takes a very long time. $\endgroup$
    – a06e
    Sep 16, 2017 at 14:49
5
$\begingroup$

I was just recently dealing with this, so I thought I should leave this line here for anyone who winds up finding this question too:

ShannonEntropy[dist_?DistributionParameterQ] :=
  -Expectation[LogLikelihood[dist,{x}], x \[Distributed] dist]

AbsoluteTiming[ShannonEntropy[NormalDistribution[]]]
(* {0.122426, 1/2 (1 + Log[2] + Log[\[Pi]])} *)

Additionally, a slightly more thorough version is available here.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.