Non linear fit/Find distribution parameters for large gradient data;

Question

searching around I've found that this is a common problem with fitting but I haven't found a work-able solution.

I have a symmetric data set that I've manipulated so that it is a probability distribution, ie. integral sums to 1, but the peak occurs between -0.002 and 0.002. Using FindDistributionParameters returns an error but I can fit it with a normal distribution using:

data = ToExpression@ImportString[Import["https://pastebin.com/raw/cVExSjYq"], "Text"]

{\[Mu], \[Sigma]} = (NonlinearModelFit[
  data, 
  1/Sqrt[2*\[Pi]*\[Sigma]^2]*E^(-(x - \[Mu])^2/(
   2*\[Sigma]^2)), {\[Mu], \[Sigma]}, x][
 "BestFitParameters"][[All, 2]]) /. {x_, y_} -> {x, Abs[y]}

outputs:

{1.24156*10^-20, 0.000250996}

The distribution doesn't closely match, hence I'm trying to use other distributions but fitting it with

FindDistributionParameters[data,StudentTDistribution[0, \[Sigma], \[Nu]], {{\[Sigma], 0.00025}, {\[Nu], 0.5}}]

returns the error:

"One or more data points are not in support of the process or distribution StudentTDistribution[0,\[Sigma],\[Nu]]."

I get the same error using WeibullDistribution too and trying to manually find the NonlinearModelFit just returns {1,1,1} for {alpha,beta,mu}.

I've tried removing zero values using Select[data,#[[2]]!=0&] but that doesn't help.

Plotted data with StudentT[0,0.00025,0.5] in blue and NormalDistribution[0,0.00025] in red is shown here:

Any recommendations?

Answer 1

This, too, is an extended comment (and maybe just an repeat of @aardvark2012's and @AntonAntonov's comments).

The data set you present is a relative frequency distribution (and the actual sample size is lost). The integral doesn't sum to 1 as there is no integral to sum or integrate. What you do have is 0.0001*Total[data[[All,2]]] equaling 1.0.

Using "least squares" to fit a probability distribution is usually not a recommended approach as it usually makes no sense to do so. One of the many reasons is that the error variance is not constant: the tail areas that drop to zero would have to have very small error variances and that is not one of the assumptions in the least squares fitting process you've used.

(Note that for any real values of μ and positive real values of σ the integral Integrate[E^(-(x - μ)^2/(2*σ^2))/Sqrt[2*π*σ^2],{x,-∞,∞} will always equal 1.)

In short, you are mixing up regression and fitting probability distributions.

You have a regression situation if you want to predict the data[[All,2]] values from the data[[All,1]] values and believe that the curve form is the same as a probability density function. This, however, does not impart any probabilistic properties to the fit.

If the raw data consists of a random sample from some probability distribution, then you are better off to use that raw data either with FindDistributionParameters if you really know the form of the probability density function. If you don't know the form and have a large enough sample size, then using SmoothKernelDistribution should be considered.

Answer 2

Not an answer, but an extended comment.

The first point I'd make is that the data slot in FindDistributionParameters is not the same as in NonlinearModelFit -- it's just a list of outcomes, not 2D points (which explains the error message).

Second, there are a lot of data points lying on the x-axis. And all of them carry the same weight as the few that give the distribution its shape. This may distort the results returned by NonlinearModelFit.

Third, your data is very spikey, and I don't think either of the models you've tried are, by themselves, spikey enough to get a good fit.

Here's a slightly modified version of the StudentTDistribution:

model[x_, \[Alpha]_, \[Sigma]_, \[Nu]_] := \[Alpha] (\[Nu]/(\[Nu] + 
    x^2/\[Sigma]^2))^((1 + \[Nu])/2)/(
  Sqrt[\[Nu]] \[Sigma] Beta[\[Nu]/2, 1/2])

Manipulate[
 Show[ListPlot[data], 
  Plot[model[x, \[Alpha], \[Sigma], \[Nu]], {x, -0.003, 0.003}, PlotRange -> All], 
  PlotRange -> {{-0.003, 0.003}, {0, 2000}}], 
  {{\[Alpha], 1.36201}, 10^-5, 3}, {{\[Sigma], 0.000149}, 10^-5, 0.0005},   
  {{\[Nu], 0.284}, 10^-5, 1}
]

Unfortunately, FindFit and NonlinearModelFit both seem pretty unhelpful using this model, even with these initial values for the parameters.

I'll try something else if I get a chance.