4
$\begingroup$

There is a function available from the Statistics`Library context called NConditionalEntropy that appears to compute ConditionalEntropy. Thus ...

Statistics`Library`NConditionalEntropy[{1, 1, 1, 0, 1, 1, 0, 0}, {1, 
0, 0, 1, 0, 1, 0, 0}]

outputs 0.954434

When I look at the definition of Conditional Entropy in Wikipedia (http://en.wikipedia.org/wiki/Conditional_entropy), it suggests that it is the expectation of the base 2 log of the PDF of a marginal distribution of p[x,y] that it calls p[x] divided by the PDF of the distribution p[x,y], where the results are weighted according to this same distribution p[x,y]. So, this gave me hope that I could recreate ConditionalEntropy as an expectation and see if I really understood what was going on.

Thus, I write the following code:

xv = {1, 0, 0, 1, 0, 1, 0, 0, 1};(* just some test data*)
yv = {1, 1, 1, 0, 1, 1, 0, 0, 1};(* just some test data*)
ed = EmpiricalDistribution[
Flatten[Outer[
List, xv,yv], 1];
Expectation[
Log[2, PDF[MarginalDistribution[ed, 1], x]/
PDF[ed, {x, y}]], {x, y} \[Distributed] ed]//N

And I get 0.918296

But when I write ...

Statistics`Library`NConditionalEntropy[xv, yv]

I get 0.899985

In case I've got the order of arguments wrong, I've also tried 

Statistics`Library`NConditionalEntropy[yv, xv]

But this yields 0.972765, which still does not match up.

Several theories for the discrepancy:

1) I do not understand the concept of Conditional Entropy well enough 2) My code for implementing conditional entropy is missing something 3) Other

Help appreciated.

Improve this question
$\endgroup$
4
  • $\begingroup$ I haven't reviewed the wikipedia entry, but I recently looked at the one on Scholarpeia: scholarpedia.org/article/Entropy. It might prove helpful. $\endgroup$
    – Jagra
    Commented Mar 5, 2013 at 14:29
  • $\begingroup$ In v9, when I run your: Statistics...NConditionalEntropy[{1, 1, 1, 0, 1, 1, 0, 0}, {1, 0, 0, 1, 0, 1, 0, 0}] I get: `0.951205. What version and OS do you run? $\endgroup$
    – Jagra
    Commented Mar 5, 2013 at 14:42
  • $\begingroup$ Sorry about the error in my original post. I too get 0.951205. $\endgroup$
    – Seth Chandler
    Commented Mar 5, 2013 at 16:50
  • $\begingroup$ Based on my answer below it doesn't look possible to come up with a clean distribution-based way to do this. Notice the structure of the data coming from Pick in my answer. They aren't strictly the same length so EmpiricalDistribution is out. Also, any attempt at padding will change the value of Entropy since it takes the number of elements into account. $\endgroup$
    – Andy Ross
    Commented Mar 5, 2013 at 19:10

1 Answer 1

Reset to default
10
$\begingroup$

Conditional entropy H(y|x) is defined as the average entropy of y for each value in x. Using the base 2 entropy this would give.

x = {1, 1, 1, 0, 1, 1, 0, 0};
y = {1, 0, 0, 1, 0, 1, 0, 0};

Mean[Entropy[2, Pick[y, x, #]] & /@ x] // N

(* 0.951205 *)

Which is what we get for NConditionalEntropy.

Statistics`Library`NConditionalEntropy[x, y]

(* 0.951205 *)
Improve this answer
$\endgroup$
2
  • $\begingroup$ ... and the order of the vectors shouldn't matter. ...NConditionalEntropy[a, b] == NConditionalEntropy[b,a] @Andy, How does Seth get: 0.954434? $\endgroup$
    – Jagra
    Commented Mar 5, 2013 at 14:55
  • $\begingroup$ @Jagra I'm not sure. I've tried it in M9 and M8 with the same result .951205. $\endgroup$
    – Andy Ross
    Commented Mar 5, 2013 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.