I'm using NonlinearModelFit to fit 5 parameters to some data points and the "ParameterConfidenceIntervalTable" gives me the estimates with the CIs. Now I want to know if the estimated values of the parameters are statistically different.

Obviously this is the case if the CIs don't overlap, but unfortunately this can also be the case if there is a slight overlap (see here). So the question is:

What is the proper way to test if the parameter estimates are significantly different?


The advice from "statisticsbyjim.com" (not at all associated with me) only applies if you have independent estimators. But if the estimators are not independent (which is the case most of the time with a regression - linear or nonlinear), then you'll need to consider the lack of independence.

If the estimators NonlinearModelFit have approximately a normal distribution, then you can use the estimate of the covariance matrix to perform testing of the equivalence of parameters.

Taking an example from the NonlinearModelFit documentation:

data = BlockRandom[SeedRandom[12345];
   Table[{x, Exp[-2.3 x/(11 + .4 x + x^2)] + RandomReal[{-.5, .5}]}, {x, RandomReal[{1, 15}, 20]}]];
nlm = NonlinearModelFit[data, Exp[a x/(b + c x)], {a, b, c}, x];

Now grab the parameter estimates and the covariance matrix:

estimates = {a, b, c} /. nlm["BestFitParameters"]
cov = nlm["CovarianceMatrix"]

Construct the "z" statistics for each of the 3 possible comparisons:

zab = (estimates[[1]] - estimates[[2]])/Sqrt[cov[[1, 1]] + cov[[2, 2]] - 2 cov[[1, 2]]]
(* -28.276 *)
zac = (estimates[[1]] - estimates[[3]])/Sqrt[cov[[1, 1]] + cov[[3, 3]] - 2 cov[[1, 3]]]
(* -0.422041 *)
zbc = (estimates[[2]] - estimates[[3]])/Sqrt[cov[[2, 2]] + cov[[3, 3]] - 2 cov[[2, 3]]]
(* 1.13192 *)

If one ignores any adjustment for multiple comparisons, then one would reject the hypothesis of equality any time the absolute value of the resulting z-statistic is greater than 1.96 (that is, InverseCDF[NormalDistribution[], (1 + 0.95)/2]). If one still ignores an adjustment for multiple comparisons but wants to be more conservative, then using the following $t$-value rather than 1.96 is appropriate:

(* Error degrees of freedom *)
df = nlm["ANOVATableDegreesOfFreedom"][[2]];

(* t-value *)
tValue = InverseCDF[StudentTDistribution[df], 0.975]
(* 2.10982 *)

An alternative is to perform a bootstrap and compute confidence intervals for the differences or ratios of the parameters.

  • $\begingroup$ Excellent, exactly what I was looking for :-) Btw., how would I be doing the bootstrap since every call to NonlinearModelFit gives the same estimates ? $\endgroup$ – Axel Oct 30 '20 at 12:33
  • $\begingroup$ Here is an example of the bootstrap with NonlinearModelFit: mathematica.stackexchange.com/questions/185024/…. $\endgroup$ – JimB Oct 30 '20 at 16:23
  • 1
    $\begingroup$ I need to put on my Indignant Hat: Just because you have data and great software doesn't mean what you think you should do to your data will be a good thing. If you're an ear, nose, and throat doctor, that doesn't mean you should do brain surgery. So consider consulting a statistician more often. And, yes, resampling the residuals is the preferred approach (by statisticians) rather than resampling the raw data points. Sorry if it appears "home made" to you. OK. End of indignation sermon. $\endgroup$ – JimB Nov 1 '20 at 16:37
  • 1
    $\begingroup$ I think it's a missed opportunity that the "statisticsbyjim.com" domain isn't yours... ;) (+1 of course.) $\endgroup$ – J. M.'s ennui Dec 31 '20 at 3:01
  • 1
    $\begingroup$ @J.M. Maybe. My blood pressure would probability go sky high taking care of such a site. But despite COVID, I love my retirement. I can pick and choose what I want to do and drop things when I want. Glad to see you're still alive and kicking. $\endgroup$ – JimB Dec 31 '20 at 3:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.