5
$\begingroup$

I want to evaluate the differential entropy according to here

$$h(X) = - \int f(x) \log{f(x)} dx$$ where $f$ is the probability density function.

Lets create some test data (normal distribution):

data1D = RandomVariate[NormalDistribution[], 50000];

Create histograms from the data using 2 different methods:

smoothDistribution1D = SmoothKernelDistribution[data1D];
distribution1D = HistogramDistribution[data1D];

Test if the histograms are normalised:

Integrate[
  PDF[smoothDistribution1D, x],
  {x, -∞, ∞}
]
Integrate[
  PDF[distribution1D, x],
  {x, -∞, ∞}
]

The above integrals do indeed return the value of 1.

Now lets try to evaluate the differential entropy:

Integrate[
  PDF[smoothDistribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -∞, ∞}
]
Integrate[
  PDF[distribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -∞, ∞}
]

It seems like both Integrate and NIntegrate cannot evaluate.

For such distribution, it is possible to calculate the differential entropy analytically.

normalDistribution[x_, σ_, μ_] := 
1/(σ Sqrt[2 π]) Exp[-((x - μ)^2/(2 σ^2))]

Integrate[
  normalDistribution[x, σ, μ]*
  Log[normalDistribution[x, σ, μ]],
  {x, -∞, ∞},
  Assumptions -> Element[{σ, μ}, Reals] && σ > 0
]

$$-\frac{1}{2} \left( 1 + \log{(2 \pi \sigma^2)} \right)$$

$\endgroup$
6
$\begingroup$

The error I get is that the integrals do not converge. I expect that this is because the empirical PDFs are zero far from the data, and Mathematica does not automatically take $0\log 0$ to be $0$. You can do it explicitly as follows:

NIntegrate[With[{f = PDF[smoothDistribution1D, x]}, If[f > 0, f Log[f], 0]], {x, -∞, ∞}]
(* -1.4238 *)

Compared to the analytical solution, $-\frac12 (\log 2\pi + 1)\approx-1.41894$, it's not too far off.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.