# Best way to include a “singularity” in a SmoothHistogram?

I have a dataset like this:

data = {0,0,1.1,1.3,0,0,3.4,0,0,2.3,0,0 .....}


You can imagine that data is generated from a probability distribution of the form:

$$p(x)=w_0\delta(x) + (1 - w_0)f(x)$$

where $f(x)$ is a smooth probability density function, $\delta$ is Dirac-delta function, and $0\le w_0 \le 1$.

That is, there is a finite probability that $x=0$.

I want to plot an histogram of data. If I exclude 0, then a SmoothHistogram is fine. But now I want to include also in this plot the frequency of 0. In this case, SmoothHistogram performs poorly, since it tries to draw a smooth peak centered at 0.

Can you suggest a better way to visualize data? Note that I know that the location of the singularity is at 0.

• might data look something like this Join[RandomVariate[NormalDistribution[2, 1], 2000], ConstantArray[0, {100}]] ? – george2079 Mar 24 '16 at 18:26
• @george2079 Try putting the 0's in random positions. Other than that, yes. – becko Mar 24 '16 at 18:29
• First you might consider rewriting the density to $p(x)=w_0 \delta(x)+(1-w_0)f(x)$. That makes $f$ a legitimate probability density function. Second, why not just display a legend that states $X$ takes the value zero with probability $w_0$ otherwise $X$ takes the random value from the distribution $f(x)$ described by the smooth histogram? If you want to display a density, then as stated below by @george2079, there really is no appropriate scaling if a smooth histogram is displayed. Alternatively, you could plot the cumulative distribution function which would have a jump at zero. – JimB Mar 24 '16 at 20:21
• @JimBaldwin The main issue is that I am comparing different datasets (which probably have different $w_0$'s). If I just do a SmoothHistogram of the datasets removing 0, I don't get a plot that reflects that there may be different values of $w_0$ involved, since all the SmoothHistograms will get normalized to 1, instead of to $1-w_0$ (I rewrote the density as you suggested, thanks). – becko Mar 24 '16 at 20:25
• As mentioned below as long as the bar widths are constant and the area of the bar at zero is the estimate of $w_0$, then you'd be imparting information in a consistent fashion. And because it is the density/probability mass function $p$ that you want to describe, the area under $f$ should be $1-w_0$ - so just multiply the resulting density values for $f$ by $1-w_0$. – JimB Mar 24 '16 at 20:36

I guess you just do something like this:

data = RandomSample[Join[RandomVariate[NormalDistribution[2, 1], 2000],
ConstantArray[0, {100}]]];
Histogram[data]


SmoothHistogram[Select[data, # != 0 &],
Epilog -> Line[{{0, 0}, {0, .15}}], AxesOrigin -> {-2, 0}]


I cant see a sensible way to scale that line (It really goes to Infinity).

also the overall smooth plot ought to be scaled down somehow something like this should do:

 SmoothHistogram[...]/.Line[x_?(Length[#] > 3 &)] :>
Line[{1, Count[data, 0]/Length@data} # & /@ x]


In this case the scale change is so small its not worth showing another plot.

Edit:

here is an idea, pick a bin width around your singularity, then you can assign an appropriate height to a bar on the chart:

bin = .1;
inbin = Select[data, Abs[#] < bin &] // Length;
hbin = inbin/Length[data]/(2 bin) // N
SmoothHistogram[Select[data, Abs[#] > bin &],
Epilog -> Rectangle[{-bin, 0}, {bin, hbin}],
AxesOrigin -> {-2, 0}] /.
Line[x_?(Length[#] > 3 &)] :>
Line[{1, 1 - inbin/Length@data} # & /@ x]


note this counts all the data in the bin not just the exact zeros. This last chart looks a little different because its using a new random data set by the way.

• I suspect the SmoothedHistogram will always be constructed such that the density under the curve is 1 (even if we remove the 0 data from the curve). If so, one would need to manually 'scale down' the Smootherhistrogram plot by whatever proportion of the data are excluded (the zeroes) – wolfies Mar 24 '16 at 19:28
• Could you use WeightedData[] as an input to SmoothHistogram[] to weight according to if it is zero or not, then add back in the line? – dr.blochwave Mar 24 '16 at 19:48
• I don't think the line should go to infinity. The height of the line (maybe it is more stylish to use a bar?) should represent the weight $w_0$ of $0$, or the frequency of $0$'s in the dataset. – becko Mar 24 '16 at 20:12
• @becko: To make any sense the area of the bar at zero would have to be $w_0$. And because the width is completely arbitrary (and "technically" should have a width of zero), I don't see that any appropriate knowledge/information would be forthcoming unless...there were multiple figures and an arbitrary bar width would be consistent among the figures. – JimB Mar 24 '16 at 20:25
• You estimate $w_0$ from the data, say by counting the 0's. The width is of course arbitrary, it's just visually more appealing, I think. – becko Mar 24 '16 at 20:27

This is what we get playing with the parameters of SmoothHistogram and SmoothKernelDistribution:

data = RandomSample[Join[RandomVariate[NormalDistribution[2, 1], 2000],
ConstantArray[0, {100}]]];
Histogram[data]


SmoothHistogram[data, {"Adaptive", .1, .001}]


SmoothHistogram[data, {"Adaptive", .05, .01}]


dist = SmoothKernelDistribution[data, {"Adaptive", .1, .001}];

Plot[PDF[dist, x], {x, -4, 4}]