# Entropy[...] doesn't calculate the entropy of a probability distribution? What does it do?

I know the definition of entropy of a probability distribution:

$$H = - \sum_i p_i \log p_i$$

So for example, in a Bernoulli distribution with $p = 0.2$, $1-p=0.8$, the entropy is $0.5$. However, in Mathematica

 Entropy[{0.2,0.8}]


returns Log. So either the Mathematica has a bug, or I don't understand what it is that Entropy[...] calculates in Mathematica. Can someone clarify this for me?

• try N@Entropy@RandomVariate[BernoulliDistribution[.2], 100000] Feb 2 '16 at 15:48
• I have discuss this problem on community I think like you that is not the Shannon entropy, but the Ashby entropy A. Dauphiné Feb 2 '16 at 15:49
• @Dr.belisarius Okay, that gives the number I want. Care to explain why? Feb 2 '16 at 15:53
• @becko, it's just what the function does, estimate the entropy from samples. Like Mean or Variance. However, those 2 also do the proper symbolic thing when passed a distribution and Entropy doesn't. But you can do it by hand, e.g, entropy[dist_] := Expectation[-Log[PDF[dist, \[FormalX]]], \[FormalX] \[Distributed] dist] ?
– Rojo
Feb 2 '16 at 16:15
• Entropy[list] is the same thing as Total[(-# Log[#] &) /@ (Values@Counts[list]/Length[list])]. Does this answer your question? Feb 2 '16 at 19:13

It seems Mathematica's Entropy is equivalent to the following code (at least for lists of symbols and strings):

entropy[list_List] :=
With[{p = Tally[list][[All, 2]]/Length[list]},
-p.Log[p]
]

entropy[str_String] :=
With[{p = Tally[Characters@str][[All, 2]]/StringLength[str]},
-p.Log[p]
]


You can try this on the examples on the Entropy help page to see the result is the same:

entropy[{0, 1, 1, 4, 1, 1}] == Entropy[{0, 1, 1, 4, 1, 1}]
(* True *)

entropy["A quick brown fox jumps over the lazy dog"] ==
Entropy["A quick brown fox jumps over the lazy dog"]
(* True *)


This means that Mathematica calculates entropy using Log base e, which is called nat entropy. With a choice of 2 for the base of the Log you get the Shannon entropy and with 10 as base you end up with the Hartley entropy.

• Isn't the Hartley entropy the max-entropy rather than a base 10 entropy? Jul 29 at 2:52
• Just follow the link in the answer and you’ll see that the Hartley entropy is just the nat entropy up to a constant factor. Jul 30 at 8:29
• Have a look at Hartley or max-entropy. Apparently base 10 refers to the hartley logarithmic unit. IMHO using the same name for both is very prone to confusion. Jul 30 at 14:01

Borrowing from Sjoerd C. de Vries,(noticed this also matches rojolalalalalalalalalalalalala's comment), you don't need to generate a list of random number in order to calculate the entropy of a distribution, but you do need to if you want to use Entropy.

Expectation[-Log[PDF[BernoulliDistribution[.2], q]],
q \[Distributed] BernoulliDistribution[.2]]
(* 0.500402 *)


This matches the formula for the entropy of the Bernoulli distribution, -.2 Log[.2] - .8 Log[.8]
(* 0.500402 *)

• And what does Entropy do, exactly? That was my original question. Feb 2 '16 at 16:49
• Based on the comment above by Belisarius, I assume it creates a distribution function from the input list and calculates the entropy from it, but as you can see, the documentation is minimal Feb 2 '16 at 16:59

The Entropy function takes a list of numbers and gets the proportion of values for each unique number and applies the entropy formula you show using those proportions ($p_i$).

For a binomial distribution:

(* Sample size *)
n = 97

(* Take random sample *)
x = RandomVariate[BinomialDistribution[1, 0.5], n]
(* {0,0,1,0,1,1,1,0,0,1,0,0,0,1,1,0,0,0,1,0,0,0,1,1,1,1,0,0,1,1,0,0,0,
0,1,1,0,0,0,1,1,1,0,1,0,1,1,1,0,0,0,0,1,1,0,0,1,0,0,0,0,1,0,0,1,1,1,1,
0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,1,1,1,0,0,0,1,1,0,1,0,0,1,1} *)

(* Calculate entropy *)
Entropy[x] (* Totals for each unique value *)
x1 = Total[x]
(* 41 *)
x0 = n - Total[x]
(* 56 *)


For a random sample from a normal distribution where all values are unique:

n = 97
x = RandomVariate[NormalDistribution[0, 1], n]
Entropy[x]
(* Log *)

• Great! So OP would get Log as an answer from any list with 2 unique values. Feb 2 '16 at 19:31
• @JasonB If the list had a length of 2 and the two numbers were not equal to each other, then, yes, the OP would get Log. But otherwise the number obtained would be dependent on the frequencies of the two unique numbers.
– JimB
Feb 2 '16 at 20:20

The entropy of a normalized list of probabilities is returned by

entropy[prob_List]/;Total[prob]==1 := With[{q=prob/.{0->1,0.0->1}}, -q.Log[q] ]


This expression avoids 0*Log = Indeterminate results from probability distributions as e.g. {0.0, 0.2, 0.8}.