How to evaluate differential entropy from raw data?

Question

I want to evaluate the differential entropy according to here

$$h(X) = - \int f(x) \log{f(x)} dx$$ where $f$ is the probability density function.

Lets create some test data (normal distribution):

data1D = RandomVariate[NormalDistribution[], 50000];

Create histograms from the data using 2 different methods:

smoothDistribution1D = SmoothKernelDistribution[data1D];
distribution1D = HistogramDistribution[data1D];

Test if the histograms are normalised:

Integrate[
  PDF[smoothDistribution1D, x],
  {x, -∞, ∞}
]
Integrate[
  PDF[distribution1D, x],
  {x, -∞, ∞}
]

The above integrals do indeed return the value of 1.

Now lets try to evaluate the differential entropy:

Integrate[
  PDF[smoothDistribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -∞, ∞}
]
Integrate[
  PDF[distribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -∞, ∞}
]

It seems like both Integrate and NIntegrate cannot evaluate.

For such distribution, it is possible to calculate the differential entropy analytically.

normalDistribution[x_, σ_, μ_] := 
1/(σ Sqrt[2 π]) Exp[-((x - μ)^2/(2 σ^2))]

Integrate[
  normalDistribution[x, σ, μ]*
  Log[normalDistribution[x, σ, μ]],
  {x, -∞, ∞},
  Assumptions -> Element[{σ, μ}, Reals] && σ > 0
]

$$-\frac{1}{2} \left( 1 + \log{(2 \pi \sigma^2)} \right)$$

Answer 1

The error I get is that the integrals do not converge. I expect that this is because the empirical PDFs are zero far from the data, and Mathematica does not automatically take $0\log 0$ to be $0$. You can do it explicitly as follows:

NIntegrate[With[{f = PDF[smoothDistribution1D, x]}, If[f > 0, f Log[f], 0]], {x, -∞, ∞}]
(* -1.4238 *)

Compared to the analytical solution, $-\frac12 (\log 2\pi + 1)\approx-1.41894$, it's not too far off.