I am trying to fit some complicated function that depends on 18 parameters. The problematic is that I have a large correlated dataset (336 points).
When fitting, for simplicity, taking only the diagonal errors with
NonlinearModelFit, the fit is extremely quick (0.15s). If, instead, I construct the chi-square function myself and use
FindMinimum, it takes 8s and get the same result. If using
NMinimize with good boundaries for the parameters, I get the same result in 28s.
Now, the problem is when I try to fit with the full covariance matrix. Now, I construct the chi squared function myself and try to use
NMinimize. The problem is that I get the following message:
No more memory available. Mathematica kernel has shut down. Try quitting other applications and then retry
and the fit of course stops and the kernel is empty.
Actually, if I try just to create the chi-square function itself, the same problem happens. I guess the problem is a sum of 336^2 terms with a complicated function resulting in a too long expression.
My naive workaround was to modify the function to be fitted
f[x_?NumericQ,a_?NumericQ,b_?NumericQ,...] to prevent symbolic evaluation and to get ran out of memory.
This wasn't actually enough, see here. Further, even for simple fits, setting the inputs as
?NumericQ will slow down the minimization considerably. I used then in
Hold attribute as in the previous thread, but then the fit is far from ideal (even if considering diagonal errors only) and gives the message
Encountered a gradient that is effectively zero. The result returned may not be a minimum; it may be a maximum or a saddle point.
Any ideas on how to handle this?