Make a density list plot/histogram from large, pre-binned data set?

Question

I have a large data set consisting of $\mathcal{O}(10^9)$ two-dimensional points. In order to save memory and time I have pre-binned these into a uniform grid of $500 \times 500$ bins using Fortran. When imported into Mathematica 8.0 as a table the resulting data look like:

data = {{0.388348, 0.388349, 9},{0.388348, 0.776699, 23},...},

where the first two items of each entry correspond to the $x$-$y$-coordinates of the upper-right-hand corner of the bin and the third is the count.

Edit:

• For a sample of the raw data, raw=RandomReal[1,{1000000000,2}] is a good approximation. This is obviously unworkable.

• For the binned data: binned=Table[{.01*Ceiling[raw[[i,1]]/.01],.01*Ceiling[raw[[i,2]]/.01],RandomInteger[1000]},{i,1,250000}].

I would like to plot this pre-binned data set in the form of a DensityHistogram, but my data format doesn't fit into what this function is expecting. I have reviewed a similar question for one-dimensional histograms at Histograms with pre-counted data, however I'm at a loss as to how to apply this to 2-D. I have also looked at doing

Image[Rescale[data]]

on the raw data. However, this crashes immediately with a SIGSEGV error that has the Wolfram Support team puzzled. Consequently, I haven't gone very far down this road.

Edit:

• I have also tried ListDensityPlot[data,InterpolationOrder->0]. For the full data set, Mathematica hangs for over 10 minutes, at which point it runs out of memory and the kernel shuts down. For a subset of the data, I get something more reasonable, but I would need some way to scale this up to $500^2$ data points.

Making these plots seem to be something that is fairly easily done in Matplotlib, but I have already made some other plots in Mathematica and don't want to mess with different styles. I'm fairly new to Mathematica and don't have a good knowledge of all the functionality, unfortunately.

So, how can I make a DensityHistogram when the bins and counts have already been calculated?

Answer 1

You may be able to use the new WeightedData in version 9 with HistogramDistribution to create a weighted histogram. I've reduced the number of points for speed but it should hopefully scale to your actual problem.

raw = RandomReal[1, {10000000, 2}];

binned = Table[{.01*Ceiling[raw[[i, 1]]/.01], .01*
     Ceiling[raw[[i, 2]]/.01], RandomInteger[1000]}, {i, 1, 25000}];

Now I create the WeightedData using your bin counts and fit a HistogramDistribution to them. Note that you can set a different binning if you choose but I'm using the automatic binning.

wd = WeightedData[binned[[All, 1 ;; 2]], binned[[All, 3]]];

hd = HistogramDistribution[wd];

Now to use DensityPlot to visualize the PDF.

DensityPlot[PDF[hd, {x, y}], {x, 0.01, 1}, {y, 0.01, 1}, 
 PlotRange -> All, Exclusions -> None, PlotPoints -> 50, 
 PlotLegends -> Automatic]