# Improving kernel density estimation of statistical PDF from measured CDF

I'm trying to use WeightedData and SmoothKernelDistribution to generate a PDF from measured points along a CDF. I've tried to interpolate the data and differentiate the points are undersampled this doesn't provide a good fit between the data points or a smooth PDF.

I'm following the details given in this question and the answer by @kirma, Empirical Cumulative Distribution Function. I'm not very familiar with non-parametric statistics of Kernel density estimation. I've used the default Silverman rule for estimating the bandwidth and a Gaussian kernel but I'm not really sure the impact of these choices. Regardless, the resulting CDF, is nice and smooth, but it doesn't fit the higher end of the CDF data particularly well and thus the estimated PDF isn't representative of the real PDF. Hopefully someone with a better understanding of the underlying statistical methods might be able to guide me to a way to improve the fidelity of the estimated CDF.

Here is a representative dataset and the code to estimate the CDF and PDF:

data = {{-50.6, 0.0026}, {-48.4, 0.0141}, {-45.8, 0.0431}, {-42.9, 0.0746}, {-39.7, 0.1216}, {-37.1, 0.1581}, {-34.2, 0.2102}, {-31.1, 0.2897}, {-28.1, 0.4225}, {-25., 0.6092}, {-22., 0.8}, {-19., 0.9335}, {-16., 0.9743}, {-13., 0.98}};
weighted = WeightedData[data[[All, 1]], Differences[Prepend[data[[All, 2]], 0]]];
dataplot = ListPlot[data, PlotMarkers -> Automatic];
empirical = EmpiricalDistribution[weighted];
smooth = SmoothKernelDistribution[weighted, "Silverman", "Gaussian"];
km = KernelMixtureDistribution[weighted];

cdfplot =
DiscretePlot[{CDF[smooth, x], CDF[empirical, x]}, {x, data[[1, 1]],
data[[-1, 1]], 0.01}];
Show[cdfplot, dataplot]
Plot[PDF[smooth, x], {x, -60, 0}]  For the curious reader the x-axis is temperature in degrees C and the y-axis is the fraction of water melted at that temperature, the water is in a porous material and the melting point is depressed in the pore. The melting point data can be used to estimate the pore size distribution using the Gibbs-Thompson equation.

• Can you explain what you mean by "measured points along the CDF"? Do you have a set of measurements, which you can assume are random and independent, or are you measuring it in some other way? – mikado Jun 20 '16 at 21:07
• @mikado, I'm not sure my measurements are random and independent. I heat a sample that contains ice and measure the of fraction of ice remaining at each temperature from until the sample is completely melted. Maybe my wording was incorrect. – s0rce Jun 20 '16 at 21:21
• CDF's and PDF's are more involved than just curve fitting. They represent the distribution of independent random samples. How is probabilistic sampling involved with what you are doing? CDF's and PDF's are a great source of curve forms to fit - but are you just curve fitting without any probabilistic interpretation? – JimB Jun 20 '16 at 22:38
• @JimBaldwin, I'm interested in determining the PDF from the measured points. I don't really need to fit a curve. Basically the derivative of the measured CDF would give me the PDF and that would be adequate but the noise in the experimental data makes that a bit difficult so I thought using the kernel density estimation tools might be appropriate here. I could also approximate the data with something like a B-spline but I'm not certain thats anymore appropriate. – s0rce Jun 20 '16 at 23:26
• If you have noise, then you need to characterize that noise. Maybe the data can be described as beta regression (as the dependent variable ranges from 0 to 1) with temperature as the covariate and repeated measures (as you take multiple samples from the same experimental unit). Such a question would be better addressed at CrossValidated and then the implementation addressed here. – JimB Jun 21 '16 at 0:50