I am trying to fit some complicated function that depends on 18 parameters. The problematic is that I have a large correlated dataset (336 points).
When fitting, for simplicity, taking only the diagonal errors with NonlinearModelFit
, the fit is extremely quick (0.15s). If, instead, I construct the chi-square function myself and use FindMinimum
, it takes 8s and get the same result. If using NMinimize
with good boundaries for the parameters, I get the same result in 28s.
Now, the problem is when I try to fit with the full covariance matrix. Now, I construct the chi squared function myself and try to use FindMinimum
or NMinimize
. The problem is that I get the following message:
No more memory available. Mathematica kernel has shut down. Try quitting other applications and then retry
and the fit of course stops and the kernel is empty.
Actually, if I try just to create the chi-square function itself, the same problem happens. I guess the problem is a sum of 336^2 terms with a complicated function resulting in a too long expression.
My naive workaround was to modify the function to be fitted f[x_,a_,b_,...]
as f[x_?NumericQ,a_?NumericQ,b_?NumericQ,...]
to prevent symbolic evaluation and to get ran out of memory.
This wasn't actually enough, see here. Further, even for simple fits, setting the inputs as ?NumericQ
will slow down the minimization considerably. I used then in FindMinimum
the Hold[]
attribute as in the previous thread, but then the fit is far from ideal (even if considering diagonal errors only) and gives the message
Encountered a gradient that is effectively zero. The result returned may not be a minimum; it may be a maximum or a saddle point.
Any ideas on how to handle this?
LogLikelihood
function would seem to be what you want to use. $\endgroup$