I have some data as:


I want to numerically calculate the integral: $S = - \int dx p(x) \ln p(x)$, where $p(x)$ is the probability density function. This integral is also called the entropy of distribution.

To do so, we define

f = HistogramDistribution[data]


p = PDF[f, x]

Now, we can calculate the integral as

NIntegrate[ -p  Log[p], {x,-\[Infinity], \[Infinity]}]

But, why does Mahematica return errors?


It seems that most of the probability mass of $p$ is between $15$ and $40$.


NIntegrate[-(p) Log[p], {x, 15, 40}]

(*ans = 2.95531*)

Your getting errors because Mathematica is blindly trying to numberically integrate extremely small real numbers outside this region. Check out the piecewise representation of $p$

enter image description here

enter image description here

You can see that the PDF is basically defined from $15$ to $40$

enter image description here

A better way to compute the Entropy of a distribution is:

-Expectation[Log[p], x \[Distributed] f]

(*ans = 2.95531*)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy