10
$\begingroup$

I know the definition of entropy of a probability distribution:

$$H = - \sum_i p_i \log p_i $$

So for example, in a Bernoulli distribution with $p = 0.2$, $1-p=0.8$, the entropy is $0.5$. However, in Mathematica

 Entropy[{0.2,0.8}]

returns Log[2]. So either the Mathematica has a bug, or I don't understand what it is that Entropy[...] calculates in Mathematica. Can someone clarify this for me?

$\endgroup$
7
  • $\begingroup$ try N@Entropy@RandomVariate[BernoulliDistribution[.2], 100000] $\endgroup$ – Dr. belisarius Feb 2 '16 at 15:48
  • 1
    $\begingroup$ I have discuss this problem on community I think like you that is not the Shannon entropy, but the Ashby entropy A. Dauphiné $\endgroup$ – Andre Feb 2 '16 at 15:49
  • $\begingroup$ @Dr.belisarius Okay, that gives the number I want. Care to explain why? $\endgroup$ – becko Feb 2 '16 at 15:53
  • 3
    $\begingroup$ @becko, it's just what the function does, estimate the entropy from samples. Like Mean or Variance. However, those 2 also do the proper symbolic thing when passed a distribution and Entropy doesn't. But you can do it by hand, e.g, entropy[dist_] := Expectation[-Log[PDF[dist, \[FormalX]]], \[FormalX] \[Distributed] dist] ? $\endgroup$ – Rojo Feb 2 '16 at 16:15
  • 2
    $\begingroup$ Entropy[list] is the same thing as Total[(-# Log[#] &) /@ (Values@Counts[list]/Length[list])]. Does this answer your question? $\endgroup$ – Szabolcs Feb 2 '16 at 19:13
8
$\begingroup$

It seems Mathematica's Entropy is equivalent to the following code (at least for lists of symbols and strings):

entropy[list_List] :=
 With[{p = Tally[list][[All, 2]]/Length[list]},
  -p.Log[p]
  ]

entropy[str_String] :=
 With[{p = Tally[Characters@str][[All, 2]]/StringLength[str]},
  -p.Log[p]
  ]

You can try this on the examples on the Entropy help page to see the result is the same:

entropy[{0, 1, 1, 4, 1, 1}] == Entropy[{0, 1, 1, 4, 1, 1}]
(* True *)

entropy["A quick brown fox jumps over the lazy dog"] == 
 Entropy["A quick brown fox jumps over the lazy dog"]
(* True *)

This means that Mathematica calculates entropy using Log base e, which is called nat entropy. With a choice of 2 for the base of the Log you get the Shannon entropy and with 10 as base you end up with the Hartley entropy.

$\endgroup$
3
  • $\begingroup$ Isn't the Hartley entropy the max-entropy rather than a base 10 entropy? $\endgroup$ – DurandA Jul 29 at 2:52
  • $\begingroup$ Just follow the link in the answer and you’ll see that the Hartley entropy is just the nat entropy up to a constant factor. $\endgroup$ – Sjoerd C. de Vries 2 days ago
  • $\begingroup$ Have a look at Hartley or max-entropy. Apparently base 10 refers to the hartley logarithmic unit. IMHO using the same name for both is very prone to confusion. $\endgroup$ – DurandA 2 days ago
7
$\begingroup$

Borrowing from Sjoerd C. de Vries,(noticed this also matches rojolalalalalalalalalalalalala's comment), you don't need to generate a list of random number in order to calculate the entropy of a distribution, but you do need to if you want to use Entropy.

Expectation[-Log[PDF[BernoulliDistribution[.2], q]], 
 q \[Distributed] BernoulliDistribution[.2]]
(* 0.500402 *)

This matches the formula for the entropy of the Bernoulli distribution,

enter image description here

-.2 Log[.2] - .8 Log[.8]
(* 0.500402 *)
$\endgroup$
2
  • 1
    $\begingroup$ And what does Entropy do, exactly? That was my original question. $\endgroup$ – becko Feb 2 '16 at 16:49
  • 2
    $\begingroup$ Based on the comment above by Belisarius, I assume it creates a distribution function from the input list and calculates the entropy from it, but as you can see, the documentation is minimal $\endgroup$ – Jason B. Feb 2 '16 at 16:59
5
$\begingroup$

The Entropy function takes a list of numbers and gets the proportion of values for each unique number and applies the entropy formula you show using those proportions ($p_i$).

For a binomial distribution:

(* Sample size *)
n = 97 

(* Take random sample *)
x = RandomVariate[BinomialDistribution[1, 0.5], n]
(* {0,0,1,0,1,1,1,0,0,1,0,0,0,1,1,0,0,0,1,0,0,0,1,1,1,1,0,0,1,1,0,0,0,
0,1,1,0,0,0,1,1,1,0,1,0,1,1,1,0,0,0,0,1,1,0,0,1,0,0,0,0,1,0,0,1,1,1,1,
0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,1,1,1,0,0,0,1,1,0,1,0,0,1,1} *)

(* Calculate entropy *)
Entropy[x]

Entropy result

(* Totals for each unique value *)
x1 = Total[x]
(* 41 *)
x0 = n - Total[x]
(* 56 *)

For a random sample from a normal distribution where all values are unique:

n = 97
x = RandomVariate[NormalDistribution[0, 1], n]
Entropy[x]
(* Log[97] *)
$\endgroup$
2
  • $\begingroup$ Great! So OP would get Log[2] as an answer from any list with 2 unique values. $\endgroup$ – Jason B. Feb 2 '16 at 19:31
  • $\begingroup$ @JasonB If the list had a length of 2 and the two numbers were not equal to each other, then, yes, the OP would get Log[2]. But otherwise the number obtained would be dependent on the frequencies of the two unique numbers. $\endgroup$ – JimB Feb 2 '16 at 20:20
1
$\begingroup$

The entropy of a normalized list of probabilities is returned by

entropy[prob_List]/;Total[prob]==1 := With[{q=prob/.{0->1,0.0->1}}, -q.Log[q] ] 

This expression avoids 0*Log[0] = Indeterminate results from probability distributions as e.g. {0.0, 0.2, 0.8}.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.