# ConditionalEntropy and StatisticsLibraryNConditionalEntropy

There is a function available from the StatisticsLibrary context called NConditionalEntropy that appears to compute ConditionalEntropy. Thus ...

StatisticsLibraryNConditionalEntropy[{1, 1, 1, 0, 1, 1, 0, 0}, {1,
0, 0, 1, 0, 1, 0, 0}]


outputs 0.954434

When I look at the definition of Conditional Entropy in Wikipedia (http://en.wikipedia.org/wiki/Conditional_entropy), it suggests that it is the expectation of the base 2 log of the PDF of a marginal distribution of p[x,y] that it calls p[x] divided by the PDF of the distribution p[x,y], where the results are weighted according to this same distribution p[x,y]. So, this gave me hope that I could recreate ConditionalEntropy as an expectation and see if I really understood what was going on.

Thus, I write the following code:

xv = {1, 0, 0, 1, 0, 1, 0, 0, 1};(* just some test data*)
yv = {1, 1, 1, 0, 1, 1, 0, 0, 1};(* just some test data*)
ed = EmpiricalDistribution[
Flatten[Outer[
List, xv,yv], 1];
Expectation[
Log[2, PDF[MarginalDistribution[ed, 1], x]/
PDF[ed, {x, y}]], {x, y} \[Distributed] ed]//N


And I get 0.918296

But when I write ...

StatisticsLibraryNConditionalEntropy[xv, yv]


I get 0.899985

In case I've got the order of arguments wrong, I've also tried

StatisticsLibraryNConditionalEntropy[yv, xv]


But this yields 0.972765, which still does not match up.

Several theories for the discrepancy:

1) I do not understand the concept of Conditional Entropy well enough 2) My code for implementing conditional entropy is missing something 3) Other

Help appreciated.

-
I haven't reviewed the wikipedia entry, but I recently looked at the one on Scholarpeia: scholarpedia.org/article/Entropy. It might prove helpful. –  Jagra Mar 5 '13 at 14:29
In v9, when I run your: Statistics...NConditionalEntropy[{1, 1, 1, 0, 1, 1, 0, 0}, {1, 0, 0, 1, 0, 1, 0, 0}] I get: 0.951205. What version and OS do you run? –  Jagra Mar 5 '13 at 14:42
Sorry about the error in my original post. I too get 0.951205. –  Seth Chandler Mar 5 '13 at 16:50
Based on my answer below it doesn't look possible to come up with a clean distribution-based way to do this. Notice the structure of the data coming from Pick in my answer. They aren't strictly the same length so EmpiricalDistribution is out. Also, any attempt at padding will change the value of Entropy since it takes the number of elements into account. –  Andy Ross Mar 5 '13 at 19:10

Conditional entropy H(y|x) is defined as the average entropy of y for each value in x. Using the base 2 entropy this would give.

x = {1, 1, 1, 0, 1, 1, 0, 0};
y = {1, 0, 0, 1, 0, 1, 0, 0};

Mean[Entropy[2, Pick[y, x, #]] & /@ x] // N

(* 0.951205 *)


Which is what we get for NConditionalEntropy.

StatisticsLibraryNConditionalEntropy[x, y]

(* 0.951205 *)

-
... and the order of the vectors shouldn't matter. ...NConditionalEntropy[a, b] == NConditionalEntropy[b,a] @Andy, How does Seth get: 0.954434? –  Jagra Mar 5 '13 at 14:55
@Jagra I'm not sure. I've tried it in M9 and M8 with the same result .951205. –  Andy Ross Mar 5 '13 at 14:58