Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

There is a function available from the Statistics`Library context called NConditionalEntropy that appears to compute ConditionalEntropy. Thus ...

Statistics`Library`NConditionalEntropy[{1, 1, 1, 0, 1, 1, 0, 0}, {1, 
0, 0, 1, 0, 1, 0, 0}]

outputs 0.954434

When I look at the definition of Conditional Entropy in Wikipedia (http://en.wikipedia.org/wiki/Conditional_entropy), it suggests that it is the expectation of the base 2 log of the PDF of a marginal distribution of p[x,y] that it calls p[x] divided by the PDF of the distribution p[x,y], where the results are weighted according to this same distribution p[x,y]. So, this gave me hope that I could recreate ConditionalEntropy as an expectation and see if I really understood what was going on.

Thus, I write the following code:

xv = {1, 0, 0, 1, 0, 1, 0, 0, 1};(* just some test data*)
yv = {1, 1, 1, 0, 1, 1, 0, 0, 1};(* just some test data*)
ed = EmpiricalDistribution[
Flatten[Outer[
List, xv,yv], 1];
Expectation[
Log[2, PDF[MarginalDistribution[ed, 1], x]/
PDF[ed, {x, y}]], {x, y} \[Distributed] ed]//N

And I get 0.918296

But when I write ...

Statistics`Library`NConditionalEntropy[xv, yv]

I get 0.899985

In case I've got the order of arguments wrong, I've also tried

Statistics`Library`NConditionalEntropy[yv, xv]

But this yields 0.972765, which still does not match up.

Several theories for the discrepancy:

1) I do not understand the concept of Conditional Entropy well enough 2) My code for implementing conditional entropy is missing something 3) Other

Help appreciated.

share|improve this question
    
I haven't reviewed the wikipedia entry, but I recently looked at the one on Scholarpeia: scholarpedia.org/article/Entropy. It might prove helpful. –  Jagra Mar 5 '13 at 14:29
    
In v9, when I run your: Statistics...NConditionalEntropy[{1, 1, 1, 0, 1, 1, 0, 0}, {1, 0, 0, 1, 0, 1, 0, 0}] I get: `0.951205. What version and OS do you run? –  Jagra Mar 5 '13 at 14:42
    
Sorry about the error in my original post. I too get 0.951205. –  Seth Chandler Mar 5 '13 at 16:50
    
Based on my answer below it doesn't look possible to come up with a clean distribution-based way to do this. Notice the structure of the data coming from Pick in my answer. They aren't strictly the same length so EmpiricalDistribution is out. Also, any attempt at padding will change the value of Entropy since it takes the number of elements into account. –  Andy Ross Mar 5 '13 at 19:10
add comment

1 Answer 1

Conditional entropy H(y|x) is defined as the average entropy of y for each value in x. Using the base 2 entropy this would give.

x = {1, 1, 1, 0, 1, 1, 0, 0};
y = {1, 0, 0, 1, 0, 1, 0, 0};

Mean[Entropy[2, Pick[y, x, #]] & /@ x] // N

(* 0.951205 *)

Which is what we get for NConditionalEntropy.

Statistics`Library`NConditionalEntropy[x, y]

(* 0.951205 *)
share|improve this answer
    
... and the order of the vectors shouldn't matter. ...NConditionalEntropy[a, b] == NConditionalEntropy[b,a] @Andy, How does Seth get: 0.954434? –  Jagra Mar 5 '13 at 14:55
    
@Jagra I'm not sure. I've tried it in M9 and M8 with the same result .951205. –  Andy Ross Mar 5 '13 at 14:58
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.