# SmoothDensityHistogram hiding data structure

I have data which shows very clear structure in a ListPlot.

xf = Import["https://pastebin.com/raw/r5Aiq03E", "List"];
zf = Import["https://pastebin.com/raw/7bvzMuQV", "List"];

a0 = 5.29177*^-11; λ = 800*^-9/a0;
V2aGrid = Partition[Riffle[zf/λ, xf/λ], 2]; v2a = {{33, 37}, {-2, 2}};

ListPlot[V2aGrid, ImageSize -> Large, PlotRange -> v2a]


However, when I try to represent this data with SmoothDensityHistogram, the structure is hidden. Having used:

 SmoothDensityHistogram[V2aGrid, Automatic, "PDF", ColorFunction->


Clearly, the structure has vanished. Is there a way to represent the data as a continuum, yet not lose the critical shape of the plot? Increasing PlotPoints doesn't seem to help.

Edit: Lowing the bandwidth helps a little, but there is still a lot of structure that isn't represented in the density plot. It seems to stop changing at a bandwidth lower than 0.01, so I'll post the image at that value.

I tried all of the pre-defined kernel options along with bw in espec, and none of them seemed to make a difference.

Edit: Minimal working example here, or raw data here and here.

• I think the key word is "Smooth" and with apparent sharp boundaries, you're going to get "smooth". Now...if there's some theory that says you have an overlay of two (or more) areas of say uniform density, then one could fit those two types separately and then add the two together which should get you sharper borders. But if you just have the data with no information on source or where the boundaries might be, then you're going to get something smoothed. – JimB Feb 23 '18 at 17:08
• @JimB I don't see why "smooth" and "populated" need to be mutually exclusive. I don't mind diffuse boarders, if that's what you mean by sharpness. I think its pretty easy to look at the scatter plot and envision a smoother representation of the data which still preserves the 3 main features - a "diamond" in the center, two "arms" on the right, and some sort of "wedge" or "wake" on the left. In the SmoothDensityHistogram, only the "diamond" structure is preserved at all from the data, and even that is sketchy. Maybe I'm trying to use the wrong kind of plot? – avikarto Feb 23 '18 at 17:40
• Can you share the data? – Rahul Feb 23 '18 at 17:47
• Your minimal working example is not working for me. – Rahul Feb 23 '18 at 18:22
• I wonder if this might explain the situation. Our eye/brain conglomerate sees borders and shapes but SmoothDensityHistogram looks at shading by density of data which are two different viewpoints in this case. The borders we see are not at a constant density but rather the density of points varies from high to much lower as one traverses a border. So contours of constant density will not follow the borders. Just like looking at a mountain top, we can "see" the ridge line but a contour/density plot won't have a "line" were we clearly see a ridge line. – JimB Feb 23 '18 at 22:48

Use the bandwidth option for SmoothDensityHistogram (where you have Automatic).

First generate some data (and I've literally stolen this from @ybeltukov in MathematicaStackExchange: Generate random points in a region...):

n = 10000000;
r2 = 0.1;
choise = RandomChoice[{\[Pi] r2, 1} -> {0, 1}, n];
box = RandomReal[1, {n, 2}];
circle = Transpose@{0.5 + # Cos@#2, 0.5 + # Sin@#2} &[
Sqrt@RandomReal[r2, n], RandomReal[2 \[Pi], n]];
pts = box choise + circle (1 - choise);


Plot data and SmoothDensityHistogram at various bandwidths:

Grid[{{ListPlot[pts[[;; ;; 300]], AspectRatio -> Automatic,
PlotLabel -> "Data"],
SmoothDensityHistogram[pts[[;; ;; 300]], Automatic,
PlotRange -> {{0, 1}, {0, 1}},
PlotLabel -> "Bandwidth = Automatic"]},
{SmoothDensityHistogram[pts[[;; ;; 300]], .1,
PlotRange -> {{0, 1}, {0, 1}},
PlotLabel -> "Bandwidth = 0.1"],
SmoothDensityHistogram[pts[[;; ;; 300]], .02,
PlotRange -> {{0, 1}, {0, 1}},
PlotLabel -> "Bandwidth=0.02"]}}]


Update:

One of the reasons for a poor representation is that what is shown above is just a small portion of the complete data set. Having a single bandwidth with the complexity of the data (multiple peaks, multiple voids, etc.) is not likely to reproduce the pattern seen in the points. Here is a figure showing all of the data and the red box delineates the subset shown above:

ListPlot[{V2aGrid, {{33, -2}, {33, 2}, {37, 2}, {37, -2}, {33, -2}}},
ImageSize -> Medium, PlotRange -> All, PlotRangePadding -> None, Joined -> {False, True},
AspectRatio -> 1, PlotStyle -> {Darker[Blue], {Thickness[0.01], Red}},
PlotRangeClipping -> False]


An alternative approach is to use an "adaptive" kernel which increases in size in low density areas and decreases in size in high density areas. SmoothDensityHistogram offers this but with the number of points for this example, it did not complete after 20 minutes.

But if one just looks at the subset of data shown above, then a better match with the data points is achieved. And here I use SmoothKernelDistribution followed by a ContourPlot to have a bit more control (i.e., convenient manipulation):

V2aGrid = Select[V2aGrid, -2 < #[[2]] < 2 &];
ListPlot[V2aGrid, ImageSize -> Medium, PlotRangePadding -> None]
skd = SmoothKernelDistribution[V2aGrid, 0.075];
ContourPlot[(1 - PDF[skd, {z, x}])^10, {z, 33, 37}, {x, -2, 2},
PlotRange -> All,
PlotPoints -> 100, ColorFunction -> "AvocadoColors", Contours -> 30,
AspectRatio -> 0.65, ContourShading -> True, ContourStyle -> None]


I used (1 - PDF[skd, {z, x}])^10 rather than PDF[skd, {z, x}] to separate the colors to better match the figure with the data points.

2nd update:

Rather than messing with the PDF it's probably better to do a bit of rescaling of the "AvocadoColors" (again using a subset of the complete dataset):

data = Select[V2aGrid, -2 < #[[2]] < 2 &];
SmoothDensityHistogram[data, 0.05, "PDF",
PlotRangePadding -> None, AspectRatio -> 0.65,
PlotRange -> All]


And here's the whole dataset:

SmoothDensityHistogram[V2aGrid, Automatic, "PDF",
InterpolationPoints -> 200,