Bin a large matrix

I have a large dataset and I'm trying to plot its scatterplot as well as an overlayed set of binned "summary" data points (in a different color of course). My intended effect is similar to this one

So far, I know how to sort my data's x-y matrix by the x-variable. How can I make it so that I cut this huge matrix into bins of equal width in the x-direction, and then find the average for all the y values within each bin? I feel like this task will be trivial once I can automatically slice my dataset into bins of arbitrary width in the x-direction.

• Is this question related to Mathematica ? – Sektor Jun 18 '15 at 15:17
• Histogram and HistogramList – BlacKow Jun 18 '15 at 15:26
• How will Histogram work if my goal is to bin it as a scatterplot? – joetheschmoe4000 Jun 18 '15 at 15:47
• This is Mathematica related because I want to understand matrix manipulation. So far, I have a list of ordered pairs that looks something like: {{1,2} , {3,4}, {5,6} ....... } (with different values of course). I want to cut this list into bins of any given length, and then average the y-values within each bin. – joetheschmoe4000 Jun 18 '15 at 15:52
• You might want to look at BinLists. – image_doctor Jun 18 '15 at 15:54

Let's generate some random data to play with:

data = Table[{x, 0.18 x + 1 + RandomReal[RandomReal]}, {x, 0, 25, 0.01}];
ListPlot[data, PlotRange -> All] Let us then define a helper function that will calculate the bin-scatter data, given the data and how many bins we want:

Clear[binscatter]
binscatter[list_?ListQ, numbins_Integer] := Module[
{sorted, partitioned},
sorted = SortBy[First][list];
partitioned = Partition[sorted, Floor[Length[sorted]/numbins]];
Mean /@ partitioned
]

We can try this out on the data generated above, by generating e.g. 10 bins:

binscatter[data, 10]

(* Out:
{{1.245, 10.869}, {3.745, 11.9407}, {6.245, 11.3557}, {8.745, 12.4129}, {11.245, 12.8107},
{13.745, 14.8622}, {16.245, 14.0835}, {18.745, 15.305}, {21.245, 13.7489},
{23.745, 15.7441}}
*)

We can then plot the bin-scatter data, e.g. on top of the original data, for comparison:

ListPlot[
{data, binscatter[data, 10]},
PlotStyle -> {Black, Directive[Red, PointSize[0.02]],}
] 