1
$\begingroup$

Consider a set of data

dataset = {{x1,y1},{x2,y2},...}

It contains ~200000 rows. I would like to obtain the pdf for x. I can do this, for example, using two ways:

1) To construct the bins:

 binsx1 = 
  Partition[Sort[#], Round[Length[#]/#2]] &[
   dataset[[All, 1]], BinsNumber];
xDistrTable = 
  Table[{(binsx1[[i]][[1]] + binsx1[[i]][[BinsHeight]])/
    2, BinsHeight/(
    binsx1[[i]][[BinsHeight]] - binsx1[[i]][[1]])}, {i, 
    1, BinsNumber, 1}];

2) Using double distribution function obtained using built-in Mathematica tools:

xyDistributionTemp = 
  SmoothKernelDistribution[dataset, 
   MaxExtraBandwidths -> 0];
xyDoubleDistribution[x_,y_] = 
 PDF[xyDistributionTemp, {x,y}]
xDistrTable1 = Table[{x,NIntegrate[xyDoubleDistribution[x,y],{y,ymin,ymax}]},{x,xmin,xmax,Deltax}]

Here, BinsHeight, BinsNumber, xmin, xmax, Deltax, ymin, ymax are some numbers.

The problem is that the distributions obtained by xDistrTable and xDistrTable1 do not match each other (see a picture below). My first guess is that it is not enough data to obtain the double differential distribution, while it is enough statistics to get the single distribution function. However (my second guess), the problem also may be in my way to define the double distribution function.

Can anyone please give me an advise about the second guess?

P.S. Unfortunately I can not provide the dataset as it is too large.

enter image description here

$\endgroup$
  • $\begingroup$ Use the "Bounded" option in SmoothKernelDistribution. Also, if you're interested in the pdf for a single variable, why would you want take the trouble to find the joint distribution and then integrate out the other variable? Just use the single variable. $\endgroup$ – JimB May 5 at 16:23
  • $\begingroup$ @JimB : thank you! Actually I need the double distribution. However, I use the single distribution as a cross-check. $\endgroup$ – John Taylor May 5 at 18:22
  • $\begingroup$ @JimB : I have not found examples of using the "Bounded" option in Mathematica. Could you please recommend how to use it? $\endgroup$ – John Taylor May 5 at 20:16
  • $\begingroup$ Look under Scope/Estimation with fixed domain (at least for Mathematica version 12). $\endgroup$ – JimB May 5 at 21:45
3
$\begingroup$

Here is a univariate example using the "Bounded" option.

x = RandomVariate[BetaDistribution[0.5, 0.5], 200000];
skd = SmoothKernelDistribution[x, 0.015, {"Bounded", {0, 1}, "Gaussian"}];
Plot[{PDF[BetaDistribution[0.5, 0.5], x],
  PDF[skd, x]}, {x, 0, 1}, Frame -> True, 
 FrameLabel -> {"x", "Probability density"},
 PlotLegends -> {"True", "Estimated"}]

Beta distribution and fit

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.