10
$\begingroup$

I know the definition of entropy of a probability distribution:

$$H = - \sum_i p_i \log p_i $$

So for example, in a Bernoulli distribution with $p = 0.2$, $1-p=0.8$, the entropy is $0.5$. However, in Mathematica

 Entropy[{0.2,0.8}]

returns Log[2]. So either the Mathematica has a bug, or I don't understand what it is that Entropy[...] calculates in Mathematica. Can someone clarify this for me?

Improve this question
$\endgroup$
7
  • $\begingroup$ try N@Entropy@RandomVariate[BernoulliDistribution[.2], 100000] $\endgroup$
    – Dr. belisarius
    Feb 2, 2016 at 15:48
  • 1
    $\begingroup$ I have discuss this problem on community I think like you that is not the Shannon entropy, but the Ashby entropy A. Dauphiné $\endgroup$
    – Andre
    Feb 2, 2016 at 15:49
  • $\begingroup$ @Dr.belisarius Okay, that gives the number I want. Care to explain why? $\endgroup$
    – becko
    Feb 2, 2016 at 15:53
  • 3
    $\begingroup$ @becko, it's just what the function does, estimate the entropy from samples. Like Mean or Variance. However, those 2 also do the proper symbolic thing when passed a distribution and Entropy doesn't. But you can do it by hand, e.g, entropy[dist_] := Expectation[-Log[PDF[dist, \[FormalX]]], \[FormalX] \[Distributed] dist] ? $\endgroup$
    – Rojo
    Feb 2, 2016 at 16:15
  • 3
    $\begingroup$ Entropy[list] is the same thing as Total[(-# Log[#] &) /@ (Values@Counts[list]/Length[list])]. Does this answer your question? $\endgroup$
    – Szabolcs
    Feb 2, 2016 at 19:13

4 Answers 4

Reset to default
8
$\begingroup$

It seems Mathematica's Entropy is equivalent to the following code (at least for lists of symbols and strings):

entropy[list_List] :=
 With[{p = Tally[list][[All, 2]]/Length[list]},
  -p.Log[p]
  ]

entropy[str_String] :=
 With[{p = Tally[Characters@str][[All, 2]]/StringLength[str]},
  -p.Log[p]
  ]

You can try this on the examples on the Entropy help page to see the result is the same:

entropy[{0, 1, 1, 4, 1, 1}] == Entropy[{0, 1, 1, 4, 1, 1}]
(* True *)

entropy["A quick brown fox jumps over the lazy dog"] == 
 Entropy["A quick brown fox jumps over the lazy dog"]
(* True *)

This means that Mathematica calculates entropy using Log base e, which is called nat entropy. With a choice of 2 for the base of the Log you get the Shannon entropy and with 10 as base you end up with the Hartley entropy.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Isn't the Hartley entropy the max-entropy rather than a base 10 entropy? $\endgroup$
    – DurandA
    Jul 29, 2021 at 2:52
  • $\begingroup$ Just follow the link in the answer and you’ll see that the Hartley entropy is just the nat entropy up to a constant factor. $\endgroup$
    – Sjoerd C. de Vries
    Jul 30, 2021 at 8:29
  • $\begingroup$ Have a look at Hartley or max-entropy. Apparently base 10 refers to the hartley logarithmic unit. IMHO using the same name for both is very prone to confusion. $\endgroup$
    – DurandA
    Jul 30, 2021 at 14:01
7
$\begingroup$

Borrowing from Sjoerd C. de Vries,(noticed this also matches rojolalalalalalalalalalalalala's comment), you don't need to generate a list of random number in order to calculate the entropy of a distribution, but you do need to if you want to use Entropy.

Expectation[-Log[PDF[BernoulliDistribution[.2], q]], 
 q \[Distributed] BernoulliDistribution[.2]]
(* 0.500402 *)

This matches the formula for the entropy of the Bernoulli distribution,

enter image description here

-.2 Log[.2] - .8 Log[.8]
(* 0.500402 *)
Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ And what does Entropy do, exactly? That was my original question. $\endgroup$
    – becko
    Feb 2, 2016 at 16:49
  • 2
    $\begingroup$ Based on the comment above by Belisarius, I assume it creates a distribution function from the input list and calculates the entropy from it, but as you can see, the documentation is minimal $\endgroup$
    – Jason B.
    Feb 2, 2016 at 16:59
5
$\begingroup$

The Entropy function takes a list of numbers and gets the proportion of values for each unique number and applies the entropy formula you show using those proportions ($p_i$).

For a binomial distribution:

(* Sample size *)
n = 97 

(* Take random sample *)
x = RandomVariate[BinomialDistribution[1, 0.5], n]
(* {0,0,1,0,1,1,1,0,0,1,0,0,0,1,1,0,0,0,1,0,0,0,1,1,1,1,0,0,1,1,0,0,0,
0,1,1,0,0,0,1,1,1,0,1,0,1,1,1,0,0,0,0,1,1,0,0,1,0,0,0,0,1,0,0,1,1,1,1,
0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,1,1,1,0,0,0,1,1,0,1,0,0,1,1} *)

(* Calculate entropy *)
Entropy[x]

Entropy result

(* Totals for each unique value *)
x1 = Total[x]
(* 41 *)
x0 = n - Total[x]
(* 56 *)

For a random sample from a normal distribution where all values are unique:

n = 97
x = RandomVariate[NormalDistribution[0, 1], n]
Entropy[x]
(* Log[97] *)
Improve this answer
$\endgroup$
2
  • $\begingroup$ Great! So OP would get Log[2] as an answer from any list with 2 unique values. $\endgroup$
    – Jason B.
    Feb 2, 2016 at 19:31
  • $\begingroup$ @JasonB If the list had a length of 2 and the two numbers were not equal to each other, then, yes, the OP would get Log[2]. But otherwise the number obtained would be dependent on the frequencies of the two unique numbers. $\endgroup$
    – JimB
    Feb 2, 2016 at 20:20
1
$\begingroup$

The entropy of a normalized list of probabilities is returned by

entropy[prob_List]/;Total[prob]==1 := With[{q=prob/.{0->1,0.0->1}}, -q.Log[q] ]

This expression avoids 0*Log[0] = Indeterminate results from probability distributions as e.g. {0.0, 0.2, 0.8}.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.