The problem
In a dense data set plotted with opaque points, data points can pile on top of each other. This can cause serious problems with the interpretation of the plots. I'll use as an example a scatterplot of normally distributed points.
theData = RandomReal[NormalDistribution[], {100000, 2}];
Unless one is careful, these plots end up showing a solid disk around the origin, which obscures the distribution since point densities above a certain level appear clipped. This also gives inappropriate emphasis to outliers, by plotting them with the same intensity as what might be a pile of a million points close together at the origin.
Here is the same plot shown at different sizes.
Row[ListPlot[
theData,
AspectRatio -> Automatic, ImageSize -> #, PlotStyle -> Directive[Black]
] & /@ {90, 180, 360}]
An artificial black disk appears at the origin and its apparent size depends on the plot size, even though the data are identical.
Here is the same plot with different settings for PointSize.
Row[ListPlot[
theData,
AspectRatio -> Automatic, ImageSize -> Medium, PlotStyle -> Directive[Black, PointSize[#]]
] & /@ {0.01, 0.005, 0.0025}]
The disk appears a different size depending on point size, even though the data are identical.
Here are plots of successively smaller subsets of the data.
Row[ListPlot[
theData[[;; #]],
AspectRatio -> Automatic, ImageSize -> Medium, PlotStyle -> Directive[Black]
] & /@ {100000, 50000, 25000}]
The distributions are obviously all the same, but they look different in the plot because of the pileup of points.
Solving pileup by changing opacity
For a given point size, one solution is to choose the minimum opacity where the darkest pixel in the plot is black. Here is an ugly, slow, procedural, brute-force way of choosing the correct opacity:
Block[
{darkestPixel},
darkestPixel=1;
idealOpacity=0;
While[
darkestPixel>0,
idealOpacity=idealOpacity+0.001;
darkestPixel=Min@Flatten@ImageData@ColorConvert[
Rasterize@ListPlot[
theData,
AspectRatio->Automatic,ImageSize->Medium,
PlotStyle->Directive[Black,PointSize[0.01],Opacity[idealOpacity]],Axes->False
],
"Grayscale"
]
]
]
Here are the resulting plots for different subsets of the data:
Row[ListPlot[
theData[[;; #[[1]]]],
AspectRatio -> Automatic, ImageSize -> Medium,
PlotStyle -> Directive[Black, PointSize[0.01], Opacity[#[[2]]]]
] & /@ {{100000, 0.018}, {50000, 0.042}, {25000, 0.1}}]
Solving pileup by changing point size
Another way is to choose a fixed opacity and increase the size of the points until the darkest pixel is black.
Block[
{darkestPixel},
darkestPixel = 1;
idealPointSize = 0;
While[
darkestPixel > 0,
idealPointSize = idealPointSize + 0.001;
darkestPixel = Min@Flatten@ImageData@ColorConvert[
Rasterize@
ListPlot[theData,
AspectRatio->Automatic, ImageSize->Medium,
PlotStyle->Directive[Black,PointSize[idealPointSize],Opacity[0.05]],Axes->False
],
"Grayscale"
]
]
]
The results for different subsets of the data:
Row[
ListPlot[
theData[[;;#[[1]]]],
AspectRatio->Automatic,ImageSize->Medium,
PlotStyle->Directive[Black,PointSize[#[[2]]],Opacity[0.05]]
]&/@{{100000,0.003},{50000,0.01},{25000,0.015}}
]
The question
What's the best way to quickly and automatically choose the correct opacity, point size or combination of the two so that there is no "clipping" in a dense plot? Is it possible to do this inside of PlotStyle
instead of as a pre-processing step? For bonus points, explain the best way to do this for multiple data sets on the same plot.
Opacity
/PointSize
to fit a plot with all data is nice, but there might be limitations. I myself encountered one when dealing with a similar data visualization problem, where the data is so dense, I used a very smallOpacity
and end with a discontinuity of the color.. $\endgroup$