Skip to main content
Mathematica

Return to Question

added 2 characters in body
Source Link
user29165
user29165
  • 565
  • 3
  • 10

I want to evaluate the differential entropy according to here

$$h(X) = \int f(x) \log{f(x)} dx$$$$h(X) = - \int f(x) \log{f(x)} dx$$ where $f$ is the probability density function.

Lets create some test data (normal distribution):

data1D = RandomVariate[NormalDistribution[], 50000];

Create histograms from the data using 2 different methods:

smoothDistribution1D = SmoothKernelDistribution[data1D];
distribution1D = HistogramDistribution[data1D];

Test if the histograms are normalised:

Integrate[
  PDF[smoothDistribution1D, x],
  {x, -∞, ∞}
]
Integrate[
  PDF[distribution1D, x],
  {x, -∞, ∞}
]

The above integrals do indeed return the value of 1.

Now lets try to evaluate the differential entropy:

Integrate[
  PDF[smoothDistribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -∞, ∞}
]
Integrate[
  PDF[distribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -∞, ∞}
]

It seems like both Integrate and NIntegrate cannot evaluate.

For such distribution, it is possible to calculate the differential entropy analytically.

normalDistribution[x_, σ_, μ_] := 
1/(σ Sqrt[2 π]) Exp[-((x - μ)^2/(2 σ^2))]

Integrate[
  normalDistribution[x, σ, μ]*
  Log[normalDistribution[x, σ, μ]],
  {x, -∞, ∞},
  Assumptions -> Element[{σ, μ}, Reals] && σ > 0
]

$$-\frac{1}{2} \left( 1 + \log{(2 \pi \sigma^2)} \right)$$

I want to evaluate the differential entropy according to here

$$h(X) = \int f(x) \log{f(x)} dx$$ where $f$ is the probability density function.

Lets create some test data (normal distribution):

data1D = RandomVariate[NormalDistribution[], 50000];

Create histograms from the data using 2 different methods:

smoothDistribution1D = SmoothKernelDistribution[data1D];
distribution1D = HistogramDistribution[data1D];

Test if the histograms are normalised:

Integrate[
  PDF[smoothDistribution1D, x],
  {x, -∞, ∞}
]
Integrate[
  PDF[distribution1D, x],
  {x, -∞, ∞}
]

The above integrals do indeed return the value of 1.

Now lets try to evaluate the differential entropy:

Integrate[
  PDF[smoothDistribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -∞, ∞}
]
Integrate[
  PDF[distribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -∞, ∞}
]

It seems like both Integrate and NIntegrate cannot evaluate.

For such distribution, it is possible to calculate the differential entropy analytically.

normalDistribution[x_, σ_, μ_] := 
1/(σ Sqrt[2 π]) Exp[-((x - μ)^2/(2 σ^2))]

Integrate[
  normalDistribution[x, σ, μ]*
  Log[normalDistribution[x, σ, μ]],
  {x, -∞, ∞},
  Assumptions -> Element[{σ, μ}, Reals] && σ > 0
]

$$-\frac{1}{2} \left( 1 + \log{(2 \pi \sigma^2)} \right)$$

I want to evaluate the differential entropy according to here

$$h(X) = - \int f(x) \log{f(x)} dx$$ where $f$ is the probability density function.

Lets create some test data (normal distribution):

data1D = RandomVariate[NormalDistribution[], 50000];

Create histograms from the data using 2 different methods:

smoothDistribution1D = SmoothKernelDistribution[data1D];
distribution1D = HistogramDistribution[data1D];

Test if the histograms are normalised:

Integrate[
  PDF[smoothDistribution1D, x],
  {x, -∞, ∞}
]
Integrate[
  PDF[distribution1D, x],
  {x, -∞, ∞}
]

The above integrals do indeed return the value of 1.

Now lets try to evaluate the differential entropy:

Integrate[
  PDF[smoothDistribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -∞, ∞}
]
Integrate[
  PDF[distribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -∞, ∞}
]

It seems like both Integrate and NIntegrate cannot evaluate.

For such distribution, it is possible to calculate the differential entropy analytically.

normalDistribution[x_, σ_, μ_] := 
1/(σ Sqrt[2 π]) Exp[-((x - μ)^2/(2 σ^2))]

Integrate[
  normalDistribution[x, σ, μ]*
  Log[normalDistribution[x, σ, μ]],
  {x, -∞, ∞},
  Assumptions -> Element[{σ, μ}, Reals] && σ > 0
]

$$-\frac{1}{2} \left( 1 + \log{(2 \pi \sigma^2)} \right)$$

Greek and infinity only
Source Link
Öskå
Öskå
  • 8.6k
  • 4
  • 32
  • 50

I want to evaluate the differential entropy according to here

$$h(X) = \int f(x) \log{f(x)} dx$$ where $f$ is the probability density function.

Lets create some test data (normal distribution):

data1D = RandomVariate[NormalDistribution[], 50000];

Create histograms from the data using 2 different methods:

smoothDistribution1D = SmoothKernelDistribution[data1D];
distribution1D = HistogramDistribution[data1D];

Test if the histograms are normalised:

Integrate[
  PDF[smoothDistribution1D, x],
  {x, -\[Infinity]∞, \[Infinity]∞}
]
Integrate[
  PDF[distribution1D, x],
  {x, -\[Infinity]∞, \[Infinity]∞}
]

The above integrals do indeed return the value of 1.

Now lets try to evaluate the differential entropy:

Integrate[
  PDF[smoothDistribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -\[Infinity]∞, \[Infinity]∞}
]
Integrate[
  PDF[distribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -\[Infinity]∞, \[Infinity]∞}
]

It seems like both Integrate and NIntegrate cannot evaluate.

For such distribution, it is possible to calculate the differential entropy analytically.

normalDistribution[x_, \[Sigma]_σ_, \[Mu]_]μ_] := 
1/(\[Sigma]σ Sqrt[2 \[Pi]]π]) Exp[-((x - \[Mu]μ)^2/(2 \[Sigma]^2σ^2))]

Integrate[
  normalDistribution[x, \[Sigma]σ, \[Mu]]*μ]*
  Log[normalDistribution[x, \[Sigma]σ, \[Mu]]]μ]],
  {x, -\[Infinity]∞, \[Infinity]∞},
  Assumptions -> Element[{\[Sigma]σ, \[Mu]μ}, Reals] && \[Sigma]σ > 0
]

$$-\frac{1}{2} \left( 1 + \log{(2 \pi \sigma^2)} \right)$$

I want to evaluate the differential entropy according to here

$$h(X) = \int f(x) \log{f(x)} dx$$ where $f$ is the probability density function.

Lets create some test data (normal distribution):

data1D = RandomVariate[NormalDistribution[], 50000];

Create histograms from the data using 2 different methods:

smoothDistribution1D = SmoothKernelDistribution[data1D];
distribution1D = HistogramDistribution[data1D];

Test if the histograms are normalised:

Integrate[
  PDF[smoothDistribution1D, x],
  {x, -\[Infinity], \[Infinity]}
]
Integrate[
  PDF[distribution1D, x],
  {x, -\[Infinity], \[Infinity]}
]

The above integrals do indeed return the value of 1.

Now lets try to evaluate the differential entropy:

Integrate[
  PDF[smoothDistribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -\[Infinity], \[Infinity]}
]
Integrate[
  PDF[distribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -\[Infinity], \[Infinity]}
]

It seems like both Integrate and NIntegrate cannot evaluate.

For such distribution, it is possible to calculate the differential entropy analytically.

normalDistribution[x_, \[Sigma]_, \[Mu]_] := 
1/(\[Sigma] Sqrt[2 \[Pi]]) Exp[-((x - \[Mu])^2/(2 \[Sigma]^2))]

Integrate[
  normalDistribution[x, \[Sigma], \[Mu]]*
  Log[normalDistribution[x, \[Sigma], \[Mu]]],
  {x, -\[Infinity], \[Infinity]},
  Assumptions -> Element[{\[Sigma], \[Mu]}, Reals] && \[Sigma] > 0
]

$$-\frac{1}{2} \left( 1 + \log{(2 \pi \sigma^2)} \right)$$

I want to evaluate the differential entropy according to here

$$h(X) = \int f(x) \log{f(x)} dx$$ where $f$ is the probability density function.

Lets create some test data (normal distribution):

data1D = RandomVariate[NormalDistribution[], 50000];

Create histograms from the data using 2 different methods:

smoothDistribution1D = SmoothKernelDistribution[data1D];
distribution1D = HistogramDistribution[data1D];

Test if the histograms are normalised:

Integrate[
  PDF[smoothDistribution1D, x],
  {x, -∞, ∞}
]
Integrate[
  PDF[distribution1D, x],
  {x, -∞, ∞}
]

The above integrals do indeed return the value of 1.

Now lets try to evaluate the differential entropy:

Integrate[
  PDF[smoothDistribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -∞, ∞}
]
Integrate[
  PDF[distribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -∞, ∞}
]

It seems like both Integrate and NIntegrate cannot evaluate.

For such distribution, it is possible to calculate the differential entropy analytically.

normalDistribution[x_, σ_, μ_] := 
1/(σ Sqrt[2 π]) Exp[-((x - μ)^2/(2 σ^2))]

Integrate[
  normalDistribution[x, σ, μ]*
  Log[normalDistribution[x, σ, μ]],
  {x, -∞, ∞},
  Assumptions -> Element[{σ, μ}, Reals] && σ > 0
]

$$-\frac{1}{2} \left( 1 + \log{(2 \pi \sigma^2)} \right)$$

Source Link
user29165
user29165
  • 565
  • 3
  • 10

How to evaluate differential entropy from raw data?

I want to evaluate the differential entropy according to here

$$h(X) = \int f(x) \log{f(x)} dx$$ where $f$ is the probability density function.

Lets create some test data (normal distribution):

data1D = RandomVariate[NormalDistribution[], 50000];

Create histograms from the data using 2 different methods:

smoothDistribution1D = SmoothKernelDistribution[data1D];
distribution1D = HistogramDistribution[data1D];

Test if the histograms are normalised:

Integrate[
  PDF[smoothDistribution1D, x],
  {x, -\[Infinity], \[Infinity]}
]
Integrate[
  PDF[distribution1D, x],
  {x, -\[Infinity], \[Infinity]}
]

The above integrals do indeed return the value of 1.

Now lets try to evaluate the differential entropy:

Integrate[
  PDF[smoothDistribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -\[Infinity], \[Infinity]}
]
Integrate[
  PDF[distribution1D, x] Log[PDF[smoothDistribution1D, x]],
  {x, -\[Infinity], \[Infinity]}
]

It seems like both Integrate and NIntegrate cannot evaluate.

For such distribution, it is possible to calculate the differential entropy analytically.

normalDistribution[x_, \[Sigma]_, \[Mu]_] := 
1/(\[Sigma] Sqrt[2 \[Pi]]) Exp[-((x - \[Mu])^2/(2 \[Sigma]^2))]

Integrate[
  normalDistribution[x, \[Sigma], \[Mu]]*
  Log[normalDistribution[x, \[Sigma], \[Mu]]],
  {x, -\[Infinity], \[Infinity]},
  Assumptions -> Element[{\[Sigma], \[Mu]}, Reals] && \[Sigma] > 0
]

$$-\frac{1}{2} \left( 1 + \log{(2 \pi \sigma^2)} \right)$$