# Entropy[…] doesn't calculate the entropy of a probability distribution? What does it do?

I know the definition of entropy of a probability distribution:

$$H = - \sum_i p_i \log p_i$$

So for example, in a Bernoulli distribution with $p = 0.2$, $1-p=0.8$, the entropy is $0.5$. However, in Mathematica

 Entropy[{0.2,0.8}]


returns Log[2]. So either the Mathematica has a bug, or I don't understand what it is that Entropy[...] calculates in Mathematica. Can someone clarify this for me?

-
try N@Entropy@RandomVariate[BernoulliDistribution[.2], 100000] – Dr. belisarius Feb 2 at 15:48
I have discuss this problem on community I think like you that is not the Shannon entropy, but the Ashby entropy A. Dauphiné – Andre Feb 2 at 15:49
@Dr.belisarius Okay, that gives the number I want. Care to explain why? – becko Feb 2 at 15:53
@becko, it's just what the function does, estimate the entropy from samples. Like Mean or Variance. However, those 2 also do the proper symbolic thing when passed a distribution and Entropy doesn't. But you can do it by hand, e.g, entropy[dist_] := Expectation[-Log[PDF[dist, \[FormalX]]], \[FormalX] \[Distributed] dist] ? – Rojo Feb 2 at 16:15
Entropy[list] is the same thing as Total[(-# Log[#] &) /@ (Values@Counts[list]/Length[list])]. Does this answer your question? – Szabolcs Feb 2 at 19:13

It seems Mathematica's Entropy is equivalent to the following code (at least for lists of symbols and strings):

entropy[list_List] :=
With[{p = Tally[list][[All, 2]]/Length[list]},
-p.Log[p]
]

entropy[str_String] :=
With[{p = Tally[Characters@str][[All, 2]]/StringLength[str]},
-p.Log[p]
]


You can try this on the examples on the Entropy help page to see the result is the same:

entropy[{0, 1, 1, 4, 1, 1}] == Entropy[{0, 1, 1, 4, 1, 1}]
(* True *)

entropy["A quick brown fox jumps over the lazy dog"] ==
Entropy["A quick brown fox jumps over the lazy dog"]
(* True *)


This means that Mathematica calculates entropy using Log base e, which is called nat entropy. With a choice of 2 for the base of the Log you get the Shannon entropy and with 10 as base you end up with the Hartley entropy.

-

Borrowing from Sjoerd C. de Vries,(noticed this also matches rojolalalalalalalalalalalalala's comment), you don't need to generate a list of random number in order to calculate the entropy of a distribution, but you do need to if you want to use Entropy.

Expectation[-Log[PDF[BernoulliDistribution[.2], q]],
q \[Distributed] BernoulliDistribution[.2]]
(* 0.500402 *)


This matches the formula for the entropy of the Bernoulli distribution,

-.2 Log[.2] - .8 Log[.8]
(* 0.500402 *)

-
And what does Entropy do, exactly? That was my original question. – becko Feb 2 at 16:49
Based on the comment above by Belisarius, I assume it creates a distribution function from the input list and calculates the entropy from it, but as you can see, the documentation is minimal – Jason is no longer a postdoc Feb 2 at 16:59

The Entropy function takes a list of numbers and gets the proportion of values for each unique number and applies the entropy formula you show using those proportions ($p_i$).

For a binomial distribution:

(* Sample size *)
n = 97

(* Take random sample *)
x = RandomVariate[BinomialDistribution[1, 0.5], n]
(* {0,0,1,0,1,1,1,0,0,1,0,0,0,1,1,0,0,0,1,0,0,0,1,1,1,1,0,0,1,1,0,0,0,
0,1,1,0,0,0,1,1,1,0,1,0,1,1,1,0,0,0,0,1,1,0,0,1,0,0,0,0,1,0,0,1,1,1,1,
0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,1,1,1,0,0,0,1,1,0,1,0,0,1,1} *)

(* Calculate entropy *)
Entropy[x]


(* Totals for each unique value *)
x1 = Total[x]
(* 41 *)
x0 = n - Total[x]
(* 56 *)


For a random sample from a normal distribution where all values are unique:

n = 97
x = RandomVariate[NormalDistribution[0, 1], n]
Entropy[x]
(* Log[97] *)

-
Great! So OP would get Log[2] as an answer from any list with 2 unique values. – Jason is no longer a postdoc Feb 2 at 19:31
@JasonB If the list had a length of 2 and the two numbers were not equal to each other, then, yes, the OP would get Log[2]. But otherwise the number obtained would be dependent on the frequencies of the two unique numbers. – Jim Baldwin Feb 2 at 20:20

The entropy of a normalized list of probabilities is returned by

entropy[prob_List]/;Total[prob]==1 := With[{q=prob/.{0->1,0.0->1}}, -q.Log[q] ]


This expression avoids 0*Log[0] = Indeterminate results from probability distributions as e.g. {0.0, 0.2, 0.8}.

-