Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have defined a function to compute the Shannon entropy of a probability distribution:

ent[p_] := Expectation[-Log[PDF[p, x]], x \[Distributed] p]

This works fine for univarite distributions:

nd = NormalDistribution[]

Out[9]= 1/2 (1 + Log[2 \[Pi]])

I would have expected ent to work with multivariate distributions as well, since x \[Distributed] p in ent should cause x to be bound to vectors of the appropriate length to make PDF[p, x] work, but I get

nd2 = ProductDistribution[nd, nd]
ent[nd2] // FullSimplify

Out[10]= Expectation[-Log[
   PDF[ProductDistribution[NormalDistribution[0, 1], 
     NormalDistribution[0, 1]], x]], 
 x \[Distributed] 
  ProductDistribution[NormalDistribution[0, 1], 
   NormalDistribution[0, 1]]]

I can write a specific function for each arity:

ent2[p_] := Expectation[-Log[PDF[p, {x, y}]], {x, y} \[Distributed] p]

Out[11]= 1 + Log[2 \[Pi]]

Is there a way to write ent to be generic wrt to the dimention of the distribution? I would also like to better understand why ent does not work for multivariate distributions as is. The definition seems mathematically correct for any dimention.

share|improve this question
You'll need to parse out the distribution, arriving at the number of vars needed for say the PDF, then generate these (say using With and Unique), and inject these. Your expectation this will happen "automagically" with x~p is incorrect. – ciao Apr 17 '14 at 8:55
up vote 1 down vote accepted

Per my comment (this is not a fleshed out answer, just an example ):

entX[p_] := With[{vars = Unique[] & /@ Range@Length@p},
  Expectation[-Log[PDF[p, vars]], vars \[Distributed] p]]


(* 1+Log[2 π] *)

Note, you'll want to use more sophistacated means to detrmine needed number, Length works here for your example, and is probably OK for some generic combination distributions, but will fail in many cases (e.g., if you feed it just nd from your example, it sees the default values for NormalDistribution and poops two variables instead of one.)

share|improve this answer
Combined with the pointer to to StatisticsLibraryDistributionDimensionality give in the answer to… I think this provides a complete solution. I would still like to better understand why the original ent does not work though – Daniel Mahler Apr 17 '14 at 20:49
@DanielMahler: Ooh! Thanks for that link! Even though undocumented, that's going to be useful. – ciao Apr 17 '14 at 21:14

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.