I'm trying to implement a function to calculate Kullback-Liebler Divergence but I am running into a problem with complex infinities in my intermediate result. Here is how I have implemented it thus far:


Function[{p, q}, p*Log[p] /Log[q]][pmfA,pmfB]

For some "nice" sample data (containing no zeros), we get the nice result:


However, if we use some "not so nice" sample data (containing zeros for some components of the PMF), it won't work, because of a negative infinity for Log[0].

eg. Try it out with...


For the purpose of computing the total, I think it would be acceptable to treat the instances as "0" but I am unsure of how to implement that.

  • 1
    $\begingroup$ p Log[p]/Log[q] /. -\[Infinity] -> 0 or DeleteCases[p Log[p]/Log[q], -\[Infinity]] are two possible implementations. $\endgroup$
    – C. E.
    Jan 20, 2016 at 21:07
  • 2
    $\begingroup$ Your definition is incorrect. The ratio p/q should be inside the logarithm. @Pickett You would also have to guard against Indeterminate when machine-precision numbers are used, I think. But anyway, I think the point is moot - there's only one logarithm. $\endgroup$
    – Jens
    Jan 20, 2016 at 23:30
  • $\begingroup$ You may use EmpiricalDistribution and simply drop the zero mass entries from it. Then you can use Expectation to obtain the result given by @Jens as I have demonstrated here. $\endgroup$
    – gwr
    Feb 17, 2017 at 18:11

1 Answer 1


My interpretation of the question is that, after fixing the error in the definition of the divergence, the mathematical problem that is being asked about arises when the numerator under the Log goes to zero - not (as is written in the question) the denominator. Here is how you can then get the correct result in this case:

pmfA = {1/6, 1/6, 1/6, 1/6, 1/6, 1/6};
pmfB = {1/4, 1/4, 1/2, 0, 0, 0};

Function[{p, q}, 
  Limit[p*Log[(p + ϵ)/(q + ϵ)], ϵ -> 
    0]][pmfB, pmfA]

(* ==> {1/4 Log[3/2], 1/4 Log[3/2], Log[3]/2, 0, 0, 0} *)

(* ==> 1/2 Log[3/2] + Log[3]/2 *)

So I switched the roles of pmfB, pmfA to get to the mathematically tractable case where the logarithm goes to negative infinity while being multiplied by 0. This is calculated by introducing a formal Limit.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.