# Probability density histogram with unequal bin widths

I am confused by output of Histogram and HistogramList for probability density ("PDF") when bin widths are not equal.

According to this and this pages and other sources, density histograms are computed by dividing counts by bar widths and total number of observations. But Histogram obviously uses another algorithm as one can see from the following example:

SeedRandom[1];
data = RandomVariate[NormalDistribution[0, 1], 15];
HistogramList[data, {{-2, 0, 1}}, "PDF"][[2]]
BinCounts[data, {{-2, 0, 1}}]/((Length[data] - 3) Differences[{-2, 0, 1}])


{1/6, 2/3}

{1/6, 2/3}

The outputs are identical when bin counts are divided by total number of observations minus 3 in this case. Why is it? What algorithm Histogram uses for determining this difference (in other cases I got other numbers)?

Rod has answered the original question but there is another issue: if one gives upper bin boundary equal to the upper datapoint value then this value will be excluded from the histogram. It does not contradict the documentation where stated that {{b1,b2,...}} will use the bins [b1,b2),[b2,b3),.... Here is an illustration:

HistogramList[data, {{-2, 0, Max[data] - 10^-13}},
"PDF"][[2]] - HistogramList[data, {{-2, 0, Max[data]}}, "PDF"][[2]]
HistogramList[data, {{-2, 0, Max[data] + 10^-13}}, "PDF"][[2]] -
HistogramList[data, {{-2, 0, Max[data]}}, "PDF"][[2]]


{0, 2.99205*10^-14}

{-(1/105), 0.0123342}

One can see that subtraction of 10^-13 does not alter the result significantly as expected but addition of 10^-13 changes it considerably because now the point Max[data] is included in the histogram. One can check this directly:

HistogramList[data, {{-2, 0, Max[data]}}, "PDF"][[2]]
BinCounts[data, {{-2, 0, Max[data]}}]/
Differences[{-2, 0, Max[data]}]/(Length[data] - 1)

HistogramList[data, {{-2, 0, Max[data] + 10^-13}}, "PDF"][[2]]
BinCounts[data, {{-2, 0, Max[data] + 10^-13}}]/
Differences[{-2, 0, Max[data]}]/Length[data]


{1/7, 0.462531}

{1/7, 0.462531}

{2/15, 0.474865}

{2/15, 0.474865}

-
Using SeedRandom[1] you get 3 observations higher than 1. When you use Histogram[data,{{-2,0,1}}] you're excluding those 3 observations... –  Rod Apr 30 '13 at 10:37
@Rod I just checked, yes. But does it matter? –  Alexey Popkov Apr 30 '13 at 10:40
It does matter when you're computing PDF's. If you exclude those 3 observations, now your probability (i.e., "PDF") should be based on 12 observations, and not 15... –  Rod Apr 30 '13 at 10:49
@Rod Thank you for clarification. You may post it as an answer. –  Alexey Popkov Apr 30 '13 at 11:00

Using SeedRandom[1] you get 3 observations higher than 1. When you use Histogram[data,{{-2,0,1}}] you're excluding those 3 observations...

If you exclude those 3 observations, now your probability (i.e., "PDF") should be based on 12 observations, and not 15...

-

Alexey,

Histogram probability is based on Area and BinCounts, and not on BinCounts alone...

If you use Max[data] you'll get

1.5443

So the area of your histogram will be computed untill 1.5443. (Try yourself: compare Histogram[data, {{-2, 0, Max[data]}},"PDF"] and Histogram[data, {{-2, 0, Max[data]}}]).

If you use Length[data] you'll get

15

So the area of your histogram will be computed untill 15. (Try yourself: compare Histogram[data, {{-2, 0, Length[data]}},"PDF"] and Histogram[data, {{-2, 0, Length[data]}}]).

That's why you should set the max value for you Histogram...

-
I think you misunderstand my second question. The point is the difference between HistogramList[data, {{-2, 0, Max[data]}}, "PDF"] and HistogramList[data, {{-2, 0, Max[data] + 10^-13}}, "PDF"]. Obviously adding 10^-13 cannot alter the result significantly (one can verify this by subtracting 10^-13). The reason for the difference is that in the second case the value Max[data] is included in the histogram while in the first case it is not. I do not understand why it is not included in the first case. –  Alexey Popkov Apr 30 '13 at 12:38
When you add 10^-13 to one block of the histogram, you are, at the same time, "removing" slightly the probability of the other blocks... That's why you get the difference... Anyway I think, too, that the difference is considerable... –  Rod Apr 30 '13 at 12:48
Compare HistogramList[data, {{-2, 0, Max[data] - 10^-13}}, "PDF"][[2]] - HistogramList[data, {{-2, 0, Max[data]}}, "PDF"][[2]] with HistogramList[data, {{-2, 0, Max[data] + 10^-13}}, "PDF"][[2]] - HistogramList[data, {{-2, 0, Max[data]}}, "PDF"][[2]]. The huge difference in the second case is due to included point Max[data] while in other cases it is discarded. –  Alexey Popkov Apr 30 '13 at 14:14