8
$\begingroup$

I basically retrieved the following technique of evaluating the mutual information involving two matrices from this site at http://bmia.bmt.tue.nl/People/BRomeny/Courses/8C080/default.htm The original code was written for image-processing, but could perhaps apply to other areas as well.

Mutual Information = H(A)+H(B)-H(AB), where H(X) is the entropy of X

My question: How to do you generalize this to apply to a matrix of arbitrary elements (including decimals less than 1, and negative numbers)? Can this also be extended to matrix of any dimension (i.e. larger than 2?).

The program follows as:

imA = ({
   {0, 1, 3, 0},
   {0, 0, 2, 5},
   {3, 2, 0, 1},
   {0, 0, 3, 4}
  }); 

imB = ({
   {1, 1, 0, 4},
   {3, 0, 4, 5},
   {0, 0, 2, 5},
   {0, 1, 4, 5}
  }); 

histogramA = BinCounts[Flatten[imA], {0, 6, 1}]
hA = Total[-(histogramA/nA) Log[histogramA/nA]] // N
nA = Total[histogramA]

histogramB = BinCounts[Flatten[imB], {0, 6, 1}];
nB = Total[histogramB];
hB = Total[-(histogramB/nB) Log[histogramB/nB]] // N
imAB = Transpose[{imA, imB}, {3, 1, 2}]
histogramAB = BinCounts[Flatten[imAB, 1], {0, 6, 1}, {0, 6, 1}];
nAB = Total[histogramAB, 2];
fh = Flatten[histogramAB]
fh2 = DeleteCases[fh, 0]
hAB = Total[-(fh2/nAB) Log[fh2/nAB]] // N
MI = hA + hB - hAB
Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Simplest thing might be to normalize the data so it all lies between (say) 0 and 255, quantize to integers, then apply the method you have. $\endgroup$
    – bill s
    Jul 15, 2013 at 21:19
  • $\begingroup$ How about normalizing a matrix a by: Round[255*a/(Max[Max[a]] - Min[Min[a]])] $\endgroup$
    – bill s
    Jul 16, 2013 at 0:52
  • $\begingroup$ I used imA = Partition[Normalize[Flatten[matA]], 4] imB = Partition[Normalize[Flatten[matB]], 4] but there appears to be some changes needed in BinCounts and so no output from this amendment. $\endgroup$
    – thils
    Jul 16, 2013 at 5:12
  • 1
    $\begingroup$ This is probably the best way forward: Suppose we have imA = Flatten[({ {0.25, 0, 0, 0}, {0, 0.25, 0, 0}, {0, 0, 0.25, 0}, {0, 0, 0, 0.25} })] imB = Flatten[({ {0.25, -0.01, 0, 0}, {-0.01, 0.25, 0, 0}, {0, 0, 0.25, -0.05}, {0, 0, -0.05, 0.25} })] N[Entropy[imA]] N[Entropy[imB]] Entropy[imA] + Entropy[imB] - StatisticsLibraryNConditionalEntropy[imA, imB] is the required answer. $\endgroup$
    – thils
    Jul 16, 2013 at 12:44

1 Answer 1

Reset to default
4
$\begingroup$

Mutual Info= H(A) - NConditionalEntropy(A,B) as described above will yield the answer.

Improve this answer
$\endgroup$
3
  • $\begingroup$ imA = Flatten[({ {0.25, 0, 0, 0}, {0, 0.25, 0, 0}, {0, 0, 0.25, 0}, {0, 0, 0, 0.25} })]; N[Entropy[imA] + Entropy[imB] - Entropy[Flatten[TensorProduct[imA, imB]]]] (*negative MI avoided using this approach *) $\endgroup$
    – thils
    Jul 16, 2013 at 13:38
  • $\begingroup$ thils = beautiful code. Compact and elegant. $\endgroup$
    – user20296
    Oct 9, 2014 at 12:06
  • $\begingroup$ Could you please give the MMA code for the MI you calculated? I cannot find NConditionalEntropy in Mathematica. I need to calculate MI across various input-output tables in Economics. $\endgroup$
    – Tugrul Temel
    Jan 5, 2021 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.