Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I know the definition of entropy of a probability distribution:

$$H = - \sum_i p_i \log p_i $$

So for example, in a Bernoulli distribution with $p = 0.2$, $1-p=0.8$, the entropy is $0.5$. However, in Mathematica

 Entropy[{0.2,0.8}]

returns Log[2]. So either the Mathematica has a bug, or I don't understand what it is that Entropy[...] calculates in Mathematica. Can someone clarify this for me?

share|improve this question
    
try N@Entropy@RandomVariate[BernoulliDistribution[.2], 100000] – Dr. belisarius Feb 2 at 15:48
    
I have discuss this problem on community I think like you that is not the Shannon entropy, but the Ashby entropy A. Dauphiné – Andre Feb 2 at 15:49
    
@Dr.belisarius Okay, that gives the number I want. Care to explain why? – becko Feb 2 at 15:53
2  
@becko, it's just what the function does, estimate the entropy from samples. Like Mean or Variance. However, those 2 also do the proper symbolic thing when passed a distribution and Entropy doesn't. But you can do it by hand, e.g, entropy[dist_] := Expectation[-Log[PDF[dist, \[FormalX]]], \[FormalX] \[Distributed] dist] ? – Rojo Feb 2 at 16:15
1  
Entropy[list] is the same thing as Total[(-# Log[#] &) /@ (Values@Counts[list]/Length[list])]. Does this answer your question? – Szabolcs Feb 2 at 19:13
up vote 7 down vote accepted

It seems Mathematica's Entropy is equivalent to the following code (at least for lists of symbols and strings):

entropy[list_List] :=
 With[{p = Tally[list][[All, 2]]/Length[list]},
  -p.Log[p]
  ]

entropy[str_String] :=
 With[{p = Tally[Characters@str][[All, 2]]/StringLength[str]},
  -p.Log[p]
  ]

You can try this on the examples on the Entropy help page to see the result is the same:

entropy[{0, 1, 1, 4, 1, 1}] == Entropy[{0, 1, 1, 4, 1, 1}]
(* True *)

entropy["A quick brown fox jumps over the lazy dog"] == 
 Entropy["A quick brown fox jumps over the lazy dog"]
(* True *)

This means that Mathematica calculates entropy using Log base e, which is called nat entropy. With a choice of 2 for the base of the Log you get the Shannon entropy and with 10 as base you end up with the Hartley entropy.

share|improve this answer

Borrowing from Sjoerd C. de Vries,(noticed this also matches rojolalalalalalalalalalalalala's comment), you don't need to generate a list of random number in order to calculate the entropy of a distribution, but you do need to if you want to use Entropy.

Expectation[-Log[PDF[BernoulliDistribution[.2], q]], 
 q \[Distributed] BernoulliDistribution[.2]]
(* 0.500402 *)

This matches the formula for the entropy of the Bernoulli distribution,

enter image description here

-.2 Log[.2] - .8 Log[.8]
(* 0.500402 *)
share|improve this answer
1  
And what does Entropy do, exactly? That was my original question. – becko Feb 2 at 16:49
1  
Based on the comment above by Belisarius, I assume it creates a distribution function from the input list and calculates the entropy from it, but as you can see, the documentation is minimal – JasonB Feb 2 at 16:59

The Entropy function takes a list of numbers and gets the proportion of values for each unique number and applies the entropy formula you show using those proportions ($p_i$).

For a binomial distribution:

(* Sample size *)
n = 97 

(* Take random sample *)
x = RandomVariate[BinomialDistribution[1, 0.5], n]
(* {0,0,1,0,1,1,1,0,0,1,0,0,0,1,1,0,0,0,1,0,0,0,1,1,1,1,0,0,1,1,0,0,0,
0,1,1,0,0,0,1,1,1,0,1,0,1,1,1,0,0,0,0,1,1,0,0,1,0,0,0,0,1,0,0,1,1,1,1,
0,0,0,0,0,0,0,0,0,1,0,1,0,0,0,1,1,1,0,0,0,1,1,0,1,0,0,1,1} *)

(* Calculate entropy *)
Entropy[x]

Entropy result

(* Totals for each unique value *)
x1 = Total[x]
(* 41 *)
x0 = n - Total[x]
(* 56 *)

For a random sample from a normal distribution where all values are unique:

n = 97
x = RandomVariate[NormalDistribution[0, 1], n]
Entropy[x]
(* Log[97] *)
share|improve this answer
    
Great! So OP would get Log[2] as an answer from any list with 2 unique values. – JasonB Feb 2 at 19:31
    
@JasonB If the list had a length of 2 and the two numbers were not equal to each other, then, yes, the OP would get Log[2]. But otherwise the number obtained would be dependent on the frequencies of the two unique numbers. – Jim Baldwin Feb 2 at 20:20

The entropy of a normalized list of probabilities is returned by

entropy[prob_List]/;Total[prob]==1 := With[{q=prob/.{0->1,0.0->1}}, -q.Log[q] ] 

This expression avoids 0*Log[0] = Indeterminate results from probability distributions as e.g. {0.0, 0.2, 0.8}.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.