I have data which shows very clear structure in a ListPlot.

xf = Import["https://pastebin.com/raw/r5Aiq03E", "List"];
zf = Import["https://pastebin.com/raw/7bvzMuQV", "List"];

a0 = 5.29177*^-11; λ = 800*^-9/a0;
V2aGrid = Partition[Riffle[zf/λ, xf/λ], 2]; v2a = {{33, 37}, {-2, 2}};

ListPlot[V2aGrid, ImageSize -> Large, PlotRange -> v2a]


However, when I try to represent this data with SmoothDensityHistogram, the structure is hidden. Having used:

 SmoothDensityHistogram[V2aGrid, Automatic, "PDF", ColorFunction->
 "AvocadoColors", PlotRangePadding->None, PlotPoints->200]


Clearly, the structure has vanished. Is there a way to represent the data as a continuum, yet not lose the critical shape of the plot? Increasing PlotPoints doesn't seem to help.

Edit: Lowing the bandwidth helps a little, but there is still a lot of structure that isn't represented in the density plot. It seems to stop changing at a bandwidth lower than 0.01, so I'll post the image at that value.

enter image description here

I tried all of the pre-defined kernel options along with bw in espec, and none of them seemed to make a difference.

Edit: Minimal working example here, or raw data here and here.

  • $\begingroup$ I think the key word is "Smooth" and with apparent sharp boundaries, you're going to get "smooth". Now...if there's some theory that says you have an overlay of two (or more) areas of say uniform density, then one could fit those two types separately and then add the two together which should get you sharper borders. But if you just have the data with no information on source or where the boundaries might be, then you're going to get something smoothed. $\endgroup$
    – JimB
    Feb 23 '18 at 17:08
  • $\begingroup$ @JimB I don't see why "smooth" and "populated" need to be mutually exclusive. I don't mind diffuse boarders, if that's what you mean by sharpness. I think its pretty easy to look at the scatter plot and envision a smoother representation of the data which still preserves the 3 main features - a "diamond" in the center, two "arms" on the right, and some sort of "wedge" or "wake" on the left. In the SmoothDensityHistogram, only the "diamond" structure is preserved at all from the data, and even that is sketchy. Maybe I'm trying to use the wrong kind of plot? $\endgroup$
    – avikarto
    Feb 23 '18 at 17:40
  • $\begingroup$ Can you share the data? $\endgroup$
    – user484
    Feb 23 '18 at 17:47
  • $\begingroup$ Your minimal working example is not working for me. $\endgroup$
    – user484
    Feb 23 '18 at 18:22
  • 1
    $\begingroup$ I wonder if this might explain the situation. Our eye/brain conglomerate sees borders and shapes but SmoothDensityHistogram looks at shading by density of data which are two different viewpoints in this case. The borders we see are not at a constant density but rather the density of points varies from high to much lower as one traverses a border. So contours of constant density will not follow the borders. Just like looking at a mountain top, we can "see" the ridge line but a contour/density plot won't have a "line" were we clearly see a ridge line. $\endgroup$
    – JimB
    Feb 23 '18 at 22:48

Use the bandwidth option for SmoothDensityHistogram (where you have Automatic).

First generate some data (and I've literally stolen this from @ybeltukov in MathematicaStackExchange: Generate random points in a region...):

n = 10000000;
r2 = 0.1;
choise = RandomChoice[{\[Pi] r2, 1} -> {0, 1}, n];
box = RandomReal[1, {n, 2}];
circle = Transpose@{0.5 + # Cos@#2, 0.5 + # Sin@#2} &[
   Sqrt@RandomReal[r2, n], RandomReal[2 \[Pi], n]];
pts = box choise + circle (1 - choise);

Plot data and SmoothDensityHistogram at various bandwidths:

Grid[{{ListPlot[pts[[;; ;; 300]], AspectRatio -> Automatic, 
    PlotLabel -> "Data"],
   SmoothDensityHistogram[pts[[;; ;; 300]], Automatic, 
    PlotRange -> {{0, 1}, {0, 1}},
    PlotLabel -> "Bandwidth = Automatic"]},
  {SmoothDensityHistogram[pts[[;; ;; 300]], .1, 
    PlotRange -> {{0, 1}, {0, 1}},
    PlotLabel -> "Bandwidth = 0.1"],
   SmoothDensityHistogram[pts[[;; ;; 300]], .02, 
    PlotRange -> {{0, 1}, {0, 1}},
    PlotLabel -> "Bandwidth=0.02"]}}]

Data and various bandwidths


One of the reasons for a poor representation is that what is shown above is just a small portion of the complete data set. Having a single bandwidth with the complexity of the data (multiple peaks, multiple voids, etc.) is not likely to reproduce the pattern seen in the points. Here is a figure showing all of the data and the red box delineates the subset shown above:

ListPlot[{V2aGrid, {{33, -2}, {33, 2}, {37, 2}, {37, -2}, {33, -2}}}, 
 ImageSize -> Medium, PlotRange -> All, PlotRangePadding -> None, Joined -> {False, True}, 
 AspectRatio -> 1, PlotStyle -> {Darker[Blue], {Thickness[0.01], Red}},
 PlotRangeClipping -> False]

Complete dataset

An alternative approach is to use an "adaptive" kernel which increases in size in low density areas and decreases in size in high density areas. SmoothDensityHistogram offers this but with the number of points for this example, it did not complete after 20 minutes.

But if one just looks at the subset of data shown above, then a better match with the data points is achieved. And here I use SmoothKernelDistribution followed by a ContourPlot to have a bit more control (i.e., convenient manipulation):

V2aGrid = Select[V2aGrid, -2 < #[[2]] < 2 &];
ListPlot[V2aGrid, ImageSize -> Medium, PlotRangePadding -> None]
skd = SmoothKernelDistribution[V2aGrid, 0.075];
ContourPlot[(1 - PDF[skd, {z, x}])^10, {z, 33, 37}, {x, -2, 2}, 
 PlotRange -> All,
 PlotPoints -> 100, ColorFunction -> "AvocadoColors", Contours -> 30,
 AspectRatio -> 0.65, ContourShading -> True, ContourStyle -> None]

Data subset

SmoothKernel distribution on data subset

I used (1 - PDF[skd, {z, x}])^10 rather than PDF[skd, {z, x}] to separate the colors to better match the figure with the data points.

2nd update:

Rather than messing with the PDF it's probably better to do a bit of rescaling of the "AvocadoColors" (again using a subset of the complete dataset):

data = Select[V2aGrid, -2 < #[[2]] < 2 &];
SmoothDensityHistogram[data, 0.05, "PDF",
 ColorFunction -> Function[x, ColorData["AvocadoColors"][x^0.5]],
 PlotRangePadding -> None, AspectRatio -> 0.65,
 PlotRange -> All]

SmoothDensityHistogram with color scaling

And here's the whole dataset:

SmoothDensityHistogram[V2aGrid, Automatic, "PDF", 
 InterpolationPoints -> 200,
 ColorFunction -> Function[x, ColorData["AvocadoColors"][x^0.5]],
 PlotRangePadding -> None, AspectRatio -> 0.65,
 PlotRange -> All, PlotPoints -> 200]

Complete dataset figure

So some of the issue seems to be that the color scheme needs a bit of rescaling to show the detail.

  • $\begingroup$ Thanks. I have edited the OP to reflect trying to modify the bw. It helps, but doesn't really recover the structure apparent in the scatter plot. Any additional ideas? $\endgroup$
    – avikarto
    Feb 23 '18 at 16:30
  • $\begingroup$ Seems like you pretty much nailed it. Thanks! I have accepted this answer. $\endgroup$
    – avikarto
    Feb 26 '18 at 14:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.