2
$\begingroup$

I have a dataset like this:

data = {0,0,1.1,1.3,0,0,3.4,0,0,2.3,0,0 .....}

You can imagine that data is generated from a probability distribution of the form:

$$p(x)=w_0\delta(x) + (1 - w_0)f(x)$$

where $f(x)$ is a smooth probability density function, $\delta$ is Dirac-delta function, and $0\le w_0 \le 1$.

That is, there is a finite probability that $x=0$.

I want to plot an histogram of data. If I exclude 0, then a SmoothHistogram is fine. But now I want to include also in this plot the frequency of 0. In this case, SmoothHistogram performs poorly, since it tries to draw a smooth peak centered at 0.

Can you suggest a better way to visualize data? Note that I know that the location of the singularity is at 0.

$\endgroup$
  • $\begingroup$ might data look something like this Join[RandomVariate[NormalDistribution[2, 1], 2000], ConstantArray[0, {100}]] ? $\endgroup$ – george2079 Mar 24 '16 at 18:26
  • $\begingroup$ @george2079 Try putting the 0's in random positions. Other than that, yes. $\endgroup$ – becko Mar 24 '16 at 18:29
  • $\begingroup$ First you might consider rewriting the density to $p(x)=w_0 \delta(x)+(1-w_0)f(x)$. That makes $f$ a legitimate probability density function. Second, why not just display a legend that states $X$ takes the value zero with probability $w_0$ otherwise $X$ takes the random value from the distribution $f(x)$ described by the smooth histogram? If you want to display a density, then as stated below by @george2079, there really is no appropriate scaling if a smooth histogram is displayed. Alternatively, you could plot the cumulative distribution function which would have a jump at zero. $\endgroup$ – JimB Mar 24 '16 at 20:21
  • $\begingroup$ @JimBaldwin The main issue is that I am comparing different datasets (which probably have different $w_0$'s). If I just do a SmoothHistogram of the datasets removing 0, I don't get a plot that reflects that there may be different values of $w_0$ involved, since all the SmoothHistograms will get normalized to 1, instead of to $1-w_0$ (I rewrote the density as you suggested, thanks). $\endgroup$ – becko Mar 24 '16 at 20:25
  • $\begingroup$ As mentioned below as long as the bar widths are constant and the area of the bar at zero is the estimate of $w_0$, then you'd be imparting information in a consistent fashion. And because it is the density/probability mass function $p$ that you want to describe, the area under $f$ should be $1-w_0$ - so just multiply the resulting density values for $f$ by $1-w_0$. $\endgroup$ – JimB Mar 24 '16 at 20:36
1
$\begingroup$

I guess you just do something like this:

data = RandomSample[Join[RandomVariate[NormalDistribution[2, 1], 2000],
    ConstantArray[0, {100}]]];
Histogram[data]

enter image description here

SmoothHistogram[Select[data, # != 0 &], 
 Epilog -> Line[{{0, 0}, {0, .15}}], AxesOrigin -> {-2, 0}]

enter image description here

I cant see a sensible way to scale that line (It really goes to Infinity).

also the overall smooth plot ought to be scaled down somehow something like this should do:

 SmoothHistogram[...]/.Line[x_?(Length[#] > 3 &)] :> 
               Line[{1, Count[data, 0]/Length@data} # & /@ x]

In this case the scale change is so small its not worth showing another plot.

Edit:

here is an idea, pick a bin width around your singularity, then you can assign an appropriate height to a bar on the chart:

bin = .1;
inbin = Select[data, Abs[#] < bin &] // Length;
hbin = inbin/Length[data]/(2 bin) // N
SmoothHistogram[Select[data, Abs[#] > bin &], 
  Epilog -> Rectangle[{-bin, 0}, {bin, hbin}], 
  AxesOrigin -> {-2, 0}] /. 
 Line[x_?(Length[#] > 3 &)] :> 
  Line[{1, 1 - inbin/Length@data} # & /@ x]

enter image description here

note this counts all the data in the bin not just the exact zeros. This last chart looks a little different because its using a new random data set by the way.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I suspect the SmoothedHistogram will always be constructed such that the density under the curve is 1 (even if we remove the 0 data from the curve). If so, one would need to manually 'scale down' the Smootherhistrogram plot by whatever proportion of the data are excluded (the zeroes) $\endgroup$ – wolfies Mar 24 '16 at 19:28
  • $\begingroup$ Could you use WeightedData[] as an input to SmoothHistogram[] to weight according to if it is zero or not, then add back in the line? $\endgroup$ – dr.blochwave Mar 24 '16 at 19:48
  • $\begingroup$ I don't think the line should go to infinity. The height of the line (maybe it is more stylish to use a bar?) should represent the weight $w_0$ of $0$, or the frequency of $0$'s in the dataset. $\endgroup$ – becko Mar 24 '16 at 20:12
  • $\begingroup$ @becko: To make any sense the area of the bar at zero would have to be $w_0$. And because the width is completely arbitrary (and "technically" should have a width of zero), I don't see that any appropriate knowledge/information would be forthcoming unless...there were multiple figures and an arbitrary bar width would be consistent among the figures. $\endgroup$ – JimB Mar 24 '16 at 20:25
  • $\begingroup$ You estimate $w_0$ from the data, say by counting the 0's. The width is of course arbitrary, it's just visually more appealing, I think. $\endgroup$ – becko Mar 24 '16 at 20:27
1
$\begingroup$

This is what we get playing with the parameters of SmoothHistogram and SmoothKernelDistribution:

data = RandomSample[Join[RandomVariate[NormalDistribution[2, 1], 2000], 
    ConstantArray[0, {100}]]];
Histogram[data]

Mathematica graphics

SmoothHistogram[data, {"Adaptive", .1, .001}]

Mathematica graphics

SmoothHistogram[data, {"Adaptive", .05, .01}]

Mathematica graphics

dist = SmoothKernelDistribution[data, {"Adaptive", .1, .001}];

Plot[PDF[dist, x], {x, -4, 4}]

Mathematica graphics

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.