# Mutual Information involving two matrix states

I basically retrieved the following technique of evaluating the mutual information involving two matrices from this site at http://bmia.bmt.tue.nl/People/BRomeny/Courses/8C080/default.htm The original code was written for image-processing, but could perhaps apply to other areas as well.

Mutual Information = H(A)+H(B)-H(AB), where H(X) is the entropy of X

My question: How to do you generalize this to apply to a matrix of arbitrary elements (including decimals less than 1, and negative numbers)? Can this also be extended to matrix of any dimension (i.e. larger than 2?).

The program follows as:

imA = ({
{0, 1, 3, 0},
{0, 0, 2, 5},
{3, 2, 0, 1},
{0, 0, 3, 4}
});

imB = ({
{1, 1, 0, 4},
{3, 0, 4, 5},
{0, 0, 2, 5},
{0, 1, 4, 5}
});

histogramA = BinCounts[Flatten[imA], {0, 6, 1}]
hA = Total[-(histogramA/nA) Log[histogramA/nA]] // N
nA = Total[histogramA]

histogramB = BinCounts[Flatten[imB], {0, 6, 1}];
nB = Total[histogramB];
hB = Total[-(histogramB/nB) Log[histogramB/nB]] // N
imAB = Transpose[{imA, imB}, {3, 1, 2}]
histogramAB = BinCounts[Flatten[imAB, 1], {0, 6, 1}, {0, 6, 1}];
nAB = Total[histogramAB, 2];
fh = Flatten[histogramAB]
fh2 = DeleteCases[fh, 0]
hAB = Total[-(fh2/nAB) Log[fh2/nAB]] // N
MI = hA + hB - hAB

-
Simplest thing might be to normalize the data so it all lies between (say) 0 and 255, quantize to integers, then apply the method you have. – bill s Jul 15 '13 at 21:19
How about normalizing a matrix a by: Round[255*a/(Max[Max[a]] - Min[Min[a]])] – bill s Jul 16 '13 at 0:52
I used imA = Partition[Normalize[Flatten[matA]], 4] imB = Partition[Normalize[Flatten[matB]], 4] but there appears to be some changes needed in BinCounts and so no output from this amendment. – thils Jul 16 '13 at 5:12
This is probably the best way forward: Suppose we have imA = Flatten[({ {0.25, 0, 0, 0}, {0, 0.25, 0, 0}, {0, 0, 0.25, 0}, {0, 0, 0, 0.25} })] imB = Flatten[({ {0.25, -0.01, 0, 0}, {-0.01, 0.25, 0, 0}, {0, 0, 0.25, -0.05}, {0, 0, -0.05, 0.25} })] N[Entropy[imA]] N[Entropy[imB]] Entropy[imA] + Entropy[imB] - StatisticsLibraryNConditionalEntropy[imA, imB] is the required answer. – thils Jul 16 '13 at 12:44