2
$\begingroup$

I need to calculate the entropy of a distribution from raw data. I found two approaches here and here.

But it seems they don't work for my case.

My data is:

d={2.33301,2.32517,2.14544,2.10534,1.89189,1.88113,1.8695,1.82432,1.80658,1.73438,1.73086,1.6627,1.66117,1.6443,1.63424,1.62412,1.59592,1.59035,1.58624,1.58396,1.58171,1.58171,1.57658,1.57658,1.57593,1.57593,1.57158,1.55735,1.5529,1.55128,1.54554,1.54554,1.54505,1.54374,1.54078,1.53186,1.5317,1.5317,1.52573,1.51839,1.51839,1.5132,1.5132,1.51085,1.51085,1.50917,1.50008,1.4974,1.49736,1.4955,1.49262,1.48803,1.48516,1.48188,1.48119,1.48067,1.48067,1.47758,1.47501,1.47088,1.47088,1.4707,1.46867,1.46828,1.46828,1.46677,1.46655,1.46655,1.46413,1.4558,1.45544,1.45544,1.45544,1.45544,1.45305,1.45113,1.45074,1.44443,1.44443,1.44324,1.4432,1.44215,1.44155,1.4365,1.43474,1.43371,1.43085,1.42623,1.42623,1.42534,1.42053,1.41892,1.41892,1.4182,1.4182,1.41176,1.41085,1.41085,1.41085,1.40726,1.40527,1.40473,1.40473,1.4008,1.3993,1.39639,1.39221,1.3913,1.3913,1.39003,1.39003,1.39003,1.38784,1.38563,1.38289,1.37221,1.36999,1.36813,1.36689,1.36354,1.36126,1.36126,1.3607,1.3607,1.3607,1.3607,1.3607,1.35809,1.35809,1.35809,1.35509,1.35211,1.35173,1.35173,1.35135,1.35135,1.35135,1.35135,1.35135,1.35135,1.35135,1.35135,1.35102,1.35102,1.35102,1.34627,1.345,1.34118,1.34099,1.33862,1.33815,1.33815,1.33815,1.33815,1.33815,1.33815,1.33777,1.33777,1.33649,1.33649,1.33264,1.33264,1.33188,1.33093,1.33093,1.33093,1.32964,1.32959,1.32959,1.32902,1.32902,1.32902,1.32902,1.32902,1.32902,1.32902,1.32749,1.32481,1.31983,1.31983,1.31983,1.31983,1.31983,1.31216,1.31192,1.31192,1.31057,1.3065,1.3065,1.3065,1.3065,1.30631,1.30592,1.30592,1.30592,1.30592,1.30407,1.30125,1.30125,1.29934,1.29465,1.29245,1.29245,1.29088,1.29088,1.29088,1.29009,1.29009,1.29009,1.29009,1.29009,1.28989,1.28989,1.28778,1.2824,1.282,1.282,1.282,1.28062,1.28042,1.2771,1.27486,1.27486,1.27486,1.27486,1.27406,1.26993,1.26948,1.26948,1.26948,1.26848,1.26808,1.26206,1.26126,1.26126,1.25905,1.25546,1.25359,1.25177,1.2504,1.25,1.24914,1.2469,1.24588,1.24588,1.24588,1.2443,1.24385,1.24385,1.24201,1.23956,1.23894,1.23464,1.23217,1.23217,1.22949,1.22722,1.22474,1.21913,1.21913,1.21913,1.21809,1.21809,1.21809,1.21622,1.21413,1.21293,1.20868,1.20868,1.20727,1.20406,1.20301,1.19963,1.19899,1.19899,1.19899,1.19899,1.19899,1.1973,1.19709,1.19391,1.19013,1.18195,1.17851,1.17851,1.17382,1.17382,1.17204,1.17139,1.17117,1.1634,1.1634,1.15832,1.15832,1.1581,1.15657,1.15657,1.15459,1.15459,1.14914,1.14865,1.13515,1.1342,1.13314,1.13107,1.12972,1.12882,1.12882,1.12882,1.12612,1.12612,1.12567,1.11481,1.11435,1.09992,1.099,1.0983,1.09042,1.08716,1.08716,1.08483,1.08225,1.07502,1.06834,1.06792,1.06507,1.06023,1.0579,1.04796,1.04796,1.03604,1.03137,1.03137,1.02372,0.99099,0.95608,0.945945,0.895888,0.86938,0.840907,0.749238};

And I have:

f1 = HistogramDistribution[d];
p1 = PDF[f1, x];

The entropy becomes:

-Expectation[Log[p1], x \[Distributed] f1]

which turns out to be negative: $-0.360647$. Entropy cannot be negative! What has gone wrong here?

Improve this question
$\endgroup$
6
  • 7
    $\begingroup$ From the Wikipedia: "One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1. For example, the uniform distribution $\mathcal{U}(0,1/2)$ has negative differential entropy $\int_0^{\frac12} -2\log(2)\,dx=-\log(2)$. Thus, differential entropy does not share all properties of discrete entropy." In other words: entropy can indeed be negative. $\endgroup$
    – Roman
    Commented Feb 13, 2021 at 12:52
  • $\begingroup$ The recept from the second link works: NIntegrate[ With[{f = PDF[f1, x]}, If[f > 0, f Log[f], 0]], {x, -\[Infinity], \[Infinity]}] results in 0.360647. $\endgroup$
    – user64494
    Commented Feb 13, 2021 at 12:59
  • 1
    $\begingroup$ @user64494 Thanks! However, in that formula the minus sign is not included. $\endgroup$
    – user77316
    Commented Feb 13, 2021 at 13:00
  • 2
    $\begingroup$ Also keep in mind that for small datasets your entropy will depend strongly on the histogram bin size. I don't think it's a good measure of entropy, for this reason. $\endgroup$
    – Roman
    Commented Feb 13, 2021 at 13:05
  • 1
    $\begingroup$ Take a look at KullbackLeiblerDivergence for an entropy measure that's always positive (against a reference distribution, that is). $\endgroup$
    – Sjoerd Smit
    Commented Feb 17, 2021 at 16:27

1 Answer 1

Reset to default
1
$\begingroup$

I believe the proposition that entropy cannot be negative applies only to categorical distributions, which are discrete distributions for which the categories have no intrinsic numerical values. For example, the outcome of a (stylized) coin flip can be characterized by two categories: Heads and Tails. One can associate the numbers 1 and 0 with those categories, but they are for convenience only. For example, with those numerical values the expectation of the random variable equals the probability of Heads. It is, however, the probability of Heads that is fundamental, not the expectation.

In the example given in the question, the distribution is continuous. Nevertheless, we can use the bin probabilities themselves --- purely as an example. We can obtain a categorical distribution by treating the bins as categories for which the bin centers (for example) are the category labels, which have no numerical significance. In this case, the entropy is given by \begin{equation} -\sum_i p_i\,\log(p_i) . \end{equation}

In Mathematica, one can do the following:

probs = Cases[p1, x_ * Boole[_] :> x/10];
-probs.Log[probs]

which produces 1.94194.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I don't think this method makes sense. The result is dominated by the (arbitrary) bin size. What is the statistical meaning of the quantity you are calculating? $\endgroup$
    – Roman
    Commented Feb 17, 2021 at 18:43
  • $\begingroup$ @Roman. I think you are correct in thinking that bin size matters. But the example I use this is not meant as a serious treatment of the histogram distribution, which as I mentioned is continuous. The example is only meant to illustrate how to use the probabilities from a categorical distribution to compute entropy. I hoped this illustration would help the OP see the connection between (1) the collection of probabilities that happened to show up in histogram distribution with the chosen bin-widths and (2) the implicit formula that I suspect the OP had in mind. I've made an edit. Does it help? $\endgroup$
    – mef
    Commented Feb 17, 2021 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.