I am using HistogramList
to bin the data and get frequency counts to generate probability distributions. I've noticed something surprising and terrible about Mathematica 10.4's binning method: it leaves out any points on the highest bin border.
TestData={1,1,1,1,2,2,2,3,3,3,3,4,4,4,5,5,5,6,6,7,7,8,8,9,9};
NumberOfBins=10;
TheMinMax=MinMax[TestData];
HistogramBinWidth=N@(TheMinMax[[2]]-TheMinMax[[1]])/NumberOfBins;
HistogramData=HistogramList[TestData,{TheMinMax[[1]],TheMinMax[[2]],HistogramBinWidth}]
The results of this is:
{{1.,1.8,2.6,3.4,4.2,5.,5.8,6.6,7.4,8.2,9.},{4,3,4,3,0,3,2,2,2,0}}
You see that the counts for the value 9 are missing from the final bin. If you check:
Length[TestData]
Total[HistogramData[[2]]]
We see that there are 25
items in the list, but only 23
counts in the HistogramList
output. This is clearly a mistake.
The reason is clearly that Mathematica uses a use bin boundaries like this: [a,b),[b,c),...which works up until the last bin. That needs a special handler to include those points too, and Wolfram forgot about it.
So the question is, what is the best way to compensate for this error/bug?
I've padded the min and max by a tiny amount to get it to work, but that introduces error (it's a kludge rather than a fix). One option would be to test the list for points that match the upper bound and manually add them to the counts for that bin. Another would be to bypass the HistogramList
function completely and do the correct binning manually. But which of these is the fastest? What's the best way to get accurate bin counts?
bins = ConstantArray[0, 9]; Scan[bins[[#]]++ &, TestData]
. $\endgroup$