Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I basically retrieved the following technique of evaluating the mutual information involving two matrices from this site at http://bmia.bmt.tue.nl/People/BRomeny/Courses/8C080/default.htm The original code was written for image-processing, but could perhaps apply to other areas as well.

Mutual Information = H(A)+H(B)-H(AB), where H(X) is the entropy of X

My question: How to do you generalize this to apply to a matrix of arbitrary elements (including decimals less than 1, and negative numbers)? Can this also be extended to matrix of any dimension (i.e. larger than 2?).

The program follows as:

imA = ({
   {0, 1, 3, 0},
   {0, 0, 2, 5},
   {3, 2, 0, 1},
   {0, 0, 3, 4}
  }); 

imB = ({
   {1, 1, 0, 4},
   {3, 0, 4, 5},
   {0, 0, 2, 5},
   {0, 1, 4, 5}
  }); 

histogramA = BinCounts[Flatten[imA], {0, 6, 1}]
hA = Total[-(histogramA/nA) Log[histogramA/nA]] // N
nA = Total[histogramA]

histogramB = BinCounts[Flatten[imB], {0, 6, 1}];
nB = Total[histogramB];
hB = Total[-(histogramB/nB) Log[histogramB/nB]] // N
imAB = Transpose[{imA, imB}, {3, 1, 2}]
histogramAB = BinCounts[Flatten[imAB, 1], {0, 6, 1}, {0, 6, 1}];
nAB = Total[histogramAB, 2];
fh = Flatten[histogramAB]
fh2 = DeleteCases[fh, 0]
hAB = Total[-(fh2/nAB) Log[fh2/nAB]] // N
MI = hA + hB - hAB
share|improve this question
    
Simplest thing might be to normalize the data so it all lies between (say) 0 and 255, quantize to integers, then apply the method you have. –  bill s Jul 15 '13 at 21:19
    
How about normalizing a matrix a by: Round[255*a/(Max[Max[a]] - Min[Min[a]])] –  bill s Jul 16 '13 at 0:52
    
I used imA = Partition[Normalize[Flatten[matA]], 4] imB = Partition[Normalize[Flatten[matB]], 4] but there appears to be some changes needed in BinCounts and so no output from this amendment. –  thils Jul 16 '13 at 5:12
    
This is probably the best way forward: Suppose we have imA = Flatten[({ {0.25, 0, 0, 0}, {0, 0.25, 0, 0}, {0, 0, 0.25, 0}, {0, 0, 0, 0.25} })] imB = Flatten[({ {0.25, -0.01, 0, 0}, {-0.01, 0.25, 0, 0}, {0, 0, 0.25, -0.05}, {0, 0, -0.05, 0.25} })] N[Entropy[imA]] N[Entropy[imB]] Entropy[imA] + Entropy[imB] - Statistics`Library`NConditionalEntropy[imA, imB] is the required answer. –  thils Jul 16 '13 at 12:44
add comment

1 Answer 1

Mutual Info= H(A) - NConditionalEntropy(A,B) as described above will yield the answer.

share|improve this answer
    
imA = Flatten[({ {0.25, 0, 0, 0}, {0, 0.25, 0, 0}, {0, 0, 0.25, 0}, {0, 0, 0, 0.25} })]; N[Entropy[imA] + Entropy[imB] - Entropy[Flatten[TensorProduct[imA, imB]]]] (*negative MI avoided using this approach *) –  thils Jul 16 '13 at 13:38
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.