# Histograms with pre-counted data

I've got some large data sets which have been counted but not binned already - essentially, a list of pairs of values (not bins) and counts.* (Or, equivalently, it's been binned into too-small bins.) I want to plot histograms for them. I remember the deprecated version of Histogram from a separate package had a FrequencyData option, but that seems to have disappeared. Is there any built-in way to accomplish this now? (I'd like to still have all the fancy features of Histogram, i.e. I don't want to just rebin the data myself and plot it directly. Notably I'd like to still be able to use Histogram's automatic bin specification, or something like it.)

*That is, my data is represented as {{1, 6}, {2, 4}, {3, 2}, ...} instead of {1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, ...}. (And before anyone suggests just expanding the data to the latter form to pass to Histogram: there are over 100K values, and the total count is over 100M.)

Edit: okay, let me be really explicit. The perfect thing would be to be able to take the first representation of the data ({{1,6}, ...}), and get exactly what Histogram would have produced had I given it the second version ({1,1,1,1,...}), without having to actually expand it to that form. (This includes being able to specify various options and extra arguments to Histogram.) I do not want a bar chart with 100K bars. I do not want to have to decide how many bins to make every time I do this, because I may do it many times with many varieties of data.

-
So, your example data could be of this form: Tally[RandomInteger[1000, 1000000]]? – FJRA Apr 6 '12 at 22:55
@FJRA: More like Tally[RandomInteger[10^5, 10^8]]. – Jefromi Apr 6 '12 at 23:00
sure, I just didn't want to crash my kernel :P. – FJRA Apr 6 '12 at 23:02
From the help : Histogram[data] by default plots a histogram with equal bin widths chosen to approximate an assumed underlying smooth distribution of the values Subscript[x, i]. So, no nice intelligence lost by using BinCounts[] instead – belisarius has settled Apr 7 '12 at 0:14
@belisarius: Am I missing something? BinCounts[] doesn't operate on frequency data either. – Jefromi Apr 7 '12 at 0:32

Histogram doesn't have any built-in support for weighted data, although it's an interesting idea, and most of the binning algorithms should be amenable to working with it.

That being said, here's a WeightedHistogram function, with some feedback from Andy Ross. It accepts

• weighted values (in the same format as RandomChoice and EmpiricalDistribution)
• binning specifications
• Histogram options.

It doesn't support the height functions, since they'd have to be manually implemented. (This isn't hard, just a bit tedious since there are several of them.)

The implementation creates a representative sample of the data to compute the bins from. This is combined with the list of actual values to make sure we cover the extremes, which might have low weights and otherwise not show up in the sample.

Options[WeightedHistogram] =
Append[Options[Histogram], "SampleSize" -> 1000];

WeightedHistogram[weights_ -> values_, o : OptionsPattern[]] :=
WeightedHistogram[weights -> values, Automatic, o]

WeightedHistogram[weights_ -> values_, bins_, o : OptionsPattern[]] :=
Block[{sample, newbins, valuelists, partitions},
sample = Join[
RandomChoice[weights -> values, OptionValue["SampleSize"]],
values];
newbins = First[HistogramList[sample, bins]];
partitions = Partition[newbins, 2, 1];
valuelists =
Total[Pick[weights, Thread[# <= values < #2]]] & @@@ partitions;
Histogram[values, {newbins}, valuelists &,
FilterRules[Flatten[{o}], Options[Histogram]]]
]


Now let's try it out with some data that is easily weighted:

data = RandomVariate[PoissonDistribution[30], 10^5];
{values, weights} = Transpose[Tally[data]];


Here's the Histogram applied to the original data:

Histogram[data]


Here's the weighted data, in vanilla and rainbow flavors:

Row[{
WeightedHistogram[weights -> values],
WeightedHistogram[weights -> values, {1}, ChartStyle -> "Rainbow"]
}]


-
Unless I misunderstand the question, I think this is about as close as they can get without writing their own function given the current functionality. +1 – Andy Ross Apr 7 '12 at 4:55
I'd have been okay with a simple "no, you have to rebin it yourself" - this is awesome. – Jefromi Apr 7 '12 at 5:14
I had to write a simple weighted histogram for something but yours is much better! – s0rce Apr 8 '12 at 2:53
@Brett I would very much like to use your function in a regime where the weights can be negative. Would you know how to achieve this? – chris Aug 29 '12 at 8:30
@Brett It seems when the weights are all equal to one, your histogram does not return the same. I.e. Histogram[ {1, 2, 2, 3, 3, 3, 4, 4, 5}, {1, 6, 1}] differs from WeightedHistogram[ Array[1 # &, {9}] -> {1, 2, 2, 3, 3, 3, 4, 4, 5}, {1, 6, 1}] – chris Aug 29 '12 at 8:32

Now version 10 Histogram accepts weighted data!

Let's simulate some data using RandomVariate:

data = Floor[RandomVariate[NormalDistribution[0, 10], 10000], 1];


Now we can use simulate a weighted data:

dataW = WeightedData@@Transpose@Tally[data];


Let's compare both histograms:

h1=Histogram[dataW, {1}];
h2=Histogram[data, {1}];
GraphicsRow@{h1, h2}


h1==h2


True

-

This is an extension of @BrettChampion 's answer, in case you have 2D information already pre-counted or weighted. Haven't yet added all the bells and whistles to propagate the options yet, but it works. It should give the same output as HistogramList. You have to specify the bins yourself at the moment, but the input is the same as in the previous answers.

WeightedHistrogram2D[weights_ -> values_, {binsx__, binsy__}] :=
Module[{mypartitions1},
mypartitions1 =Partition[#, 2, 1]&/@First@HistogramList[{{0, 0}}, {binsx, binsy}];
Return[{mypartitions1,
Partition[
Plus@@@Map[Pick[weights,
Map[Function[{x}, ((#[[1, 1]] <=x[[1]] < #[[1, 2]]) &&
(#[[2, 1]] <=x[[2]] < #[[2, 2]])) ], values]] &,
Reverse /@ Tuples[Reverse@mypartitions1]],
Length@Last@mypartitions1]}];
];

-