# How to speed up multiple calls of NonlinearModelFit?

I'm trying to do an insane amount of non-linear fits to large amounts of data sets. However, I found that the NonlinearModelFit is horribly slow.

Here's the setup:

The dataset looks like so:

out={
{
{1, R1, val, err},
{2, R1, val, err},
{3, R1, val, err}
},
{
{2, R2, val, err},
{3, R2, val, err},
{5, R2, val, err}
},
...}


, hence each element of the total data list is a list of same r values in ascending order (Rn < R(n+1)), with different indizes(these can have values 0,1,2,4,5,8) and different function values and errors.

I'm fitting the datasets as follows:

TableMV1combined={}; EstVarMV1combined={}; M0 = 4;
Do[If[Length[tab[[2]]] > 0,
jck = tab[[1]];
FitV1combined[beg_, end_] := FitF3combined[beg, end, tab[[2]]];
FitTableV1combined =
FitF3Tablecombined[FitV1combined, tab[[2]]];
input = DeleteCases[FitTableV1combined, {___, Missing[], ___}];
For[i = 1, i <= Length[input], i++,
{AppendTo[
TableMV1combined, {input[[i]][[1]], input[[i]][[4]],
input[[i]][[5]], input[[i]][[12]], input[[i]][[19]][[7]]}];
AppendTo[
EstVarMV1combined, {input[[i]][[1]], input[[i]][[4]],
input[[i]][[5]], input[[i]][[20]]}];}];
ClearAll[jck, input];];, {tab, {
{0, out0}, {1, out1}, {2, out2}, {3, out3}, {4, out4},
{5, out5}, {6, out6}, {7, out7}, {8, out8}, {9, out9},
{10, out10}, {11, out11}, {12, out12}, {13, out13},
{14, out14}, {15, out15}, {16, out16}, {17, out17},
{18, out18}, {19, out19}, {20, out20}, {21, out21},
{22, out22}, {23, out23}, {24, out24}, {25, out25},
{26, out26}, {27, out27}, {28, out28}, {29, out29},
{30, out30}
}}];
ClearAll[M0];


Here I'm using the following ParallelTable function to utilize parallelization:

FitF3Tablecombined[FitF3combined_, formatedDataTrimmed_] :=
Flatten[ParallelTable[{jck, beg, end,
formatedDataTrimmed[[beg + 1]][[1]][[2]],
formatedDataTrimmed[[Length[formatedDataTrimmed] - end]][[1]][[2]],
A0 /. FitF3combined[beg, end]["BestFitParameters"],
A1 /. FitF3combined[beg, end]["BestFitParameters"],
A2 /. FitF3combined[beg, end]["BestFitParameters"],
A3 /. FitF3combined[beg, end]["BestFitParameters"],
A5 /. FitF3combined[beg, end]["BestFitParameters"],
A8 /. FitF3combined[beg, end]["BestFitParameters"],
M /. FitF3combined[beg, end]["BestFitParameters"],
c0 /. FitF3combined[beg, end]["BestFitParameters"],
c1 /. FitF3combined[beg, end]["BestFitParameters"],
c2 /. FitF3combined[beg, end]["BestFitParameters"],
c3 /. FitF3combined[beg, end]["BestFitParameters"],
c5 /. FitF3combined[beg, end]["BestFitParameters"],
c8 /. FitF3combined[beg, end]["BestFitParameters"],
FitF3combined[beg, end]["ParameterErrors"],
FitF3combined[beg, end]["EstimatedVariance"]},
{beg, 1, Length[formatedDataTrimmed] - 8, 1},
{end, 0,(*Length[formatedDataTrimmed]-beg-4*)4, 2}], 1];


The actual non-linear model is:

FitF3combined[beg_, end_, formatedDataTrimmed_] :=
NonlinearModelFit[{#1, #2, #3} & @@@
Flatten[Table[formatedDataTrimmed[[i]],
{i, beg + 1, Length[formatedDataTrimmed] - end}], 1],
{F3[A0, M, R, c0]*KroneckerDelta[sme, 0] +
F3[A1, M, R, c1]*KroneckerDelta[sme, 1]*UnitStep[R - (sme/Nτ)] +
F3[A2, M, R, c2]*KroneckerDelta[sme, 2]*UnitStep[R - (sme/Nτ)] +
F3[A3, M, R, c3]*KroneckerDelta[sme, 3]*UnitStep[R - (sme/Nτ)] +
F3[A5, M, R, c5]*KroneckerDelta[sme, 5]*UnitStep[R - (sme/Nτ)] +
F3[A8, M, R, c8]*KroneckerDelta[sme, 8]*UnitStep[R - (sme/Nτ)],
M > 0 && M < 15},
{A0, A1, A2, A3, A5, A8, {M, M0}, {c0, 0}, {c1, 0},
{c2, 0}, {c3, 0}, {c5, 0}, {c8, 0}}, {sme, R},
Weights ->
1 / #4^2 & @@@ Flatten[Table[formatedDataTrimmed[[i]],
{i, beg + 1, Length[formatedDataTrimmed] - end}], 1]];


And finally the actual fit function:

F3[A_, M_, R_, c_] := A*Exp[-M*R]/R + c;


The parameter Nτ is known at runtime.

To wrap it up: I'm trying to fit the amplitudes Ai, asymptotic constants ci, and the common exponent M while varying the maximum and minimum values of R. For each R value I eventually have different data points separated via the index sme. That index in combination with the external parameter Nτ and the UnitStep function determines if the individual data point should contribute. However, I find that the fit of one set of data points (one ensemble out which results in ~100 possible fit-sets (aka distinct combinations of the minimal and maximal value of R)) takes between 10 and 20 minutes. Thus the fit of all 31 sets takes several hours which is unacceptable.

Any hints on how I could improve the procedure? Would it be sensible to eventually compile the linear combination of F3-calls?

EDIT: As requested by Henrik, here's an example of a dataset

out0={{{1, 0.414225, 1.40345, 0.17096}, {2, 0.414225, 1.65493,
0.0215624}, {3, 0.414225, 1.85901, 0.0129845}}, {{2, 0.434475,
1.43753, 0.0484699}, {3, 0.434475, 1.63697, 0.0259172}}, {{1,
0.450238, 1.09145, 0.0738484}, {2, 0.450238, 1.24257,
0.0243813}, {3, 0.450238, 1.41758, 0.0153114}}, {{2, 0.467925,
1.05682, 0.0439015}, {3, 0.467925, 1.22577, 0.0272181}}, {{2,
0.498813, 0.730337, 0.0536427}, {3, 0.498813, 0.87497,
0.0313678}}, {{1, 0.514863, 0.495473, 0.150721}, {2, 0.514863,
0.621701, 0.0420051}, {3, 0.514863, 0.754838, 0.0281385}}, {{2,
0.52975, 0.587331, 0.0466759}, {3, 0.52975, 0.665299,
0.0319068}}, {{2, 0.544813, 0.526371, 0.0926066}, {3, 0.544813,
0.591536, 0.0552175}}, {{2, 0.5583, 0.418403, 0.0475843}, {3,
0.5583, 0.484318, 0.0334907}}, {{2, 0.572325, 0.371991,
0.0623067}, {3, 0.572325, 0.410476, 0.0457513}}, {{3, 0.586425,
0.32778, 0.136238}}, {{2, 0.612038, 0.207978, 0.0797852}, {3,
0.612038, 0.239743, 0.0636066}}, {{2, 0.624175, 0.248989,
0.05412}, {3, 0.624175, 0.241667, 0.039516}}, {{1, 0.637375,
0.250798, 0.087002}, {2, 0.637375, 0.204584, 0.0455255}, {3,
0.637375, 0.201414, 0.0371171}, {5, 0.637375, 0.289974,
0.0277755}}, {{2, 0.649525, 0.0612087, 0.0821179}, {3, 0.649525,
0.108233, 0.0470898}, {5, 0.649525, 0.223332, 0.0323102}}, {{1,
0.67315, -0.00623123, 0.151202}, {2, 0.67315, 0.0699974,
0.0552972}, {3, 0.67315, 0.106196, 0.0450462}, {5, 0.67315,
0.18297, 0.0345499}}, {{2, 0.68465, 0.0617554, 0.0744936}, {3,
0.68465, 0.0679903, 0.0510932}, {5, 0.68465, 0.138326,
0.0381582}}, {{3, 0.707113, 0.136617, 0.0786026}, {5, 0.707113,
0.152304, 0.0450428}}, {{2, 0.718075, 0.116889, 0.0638092}, {3,
0.718075, 0.0857912, 0.0509252}, {5, 0.718075, 0.110423,
0.0418042}}, {{2, 0.728875, 0.0119743, 0.0857345}, {3, 0.728875,
0.0373805, 0.058675}, {5, 0.728875, 0.0884664, 0.0413072}}, {{2,
0.739513, 0.072656, 0.0726864}, {3, 0.739513, 0.0543086,
0.0511731}, {5, 0.739513, 0.0810656, 0.0409819}}, {{2, 0.75,
0.0468357, 0.071699}, {3, 0.75, 0.044545, 0.0515765}, {5, 0.75,
0.0735967, 0.0356424}}, {{3, 0.76035, -0.00634004, 0.0592361}, {5,
0.76035, 0.0437272, 0.0396548}}, {{3, 0.77055, -0.0118407,
0.0426499}, {5, 0.77055, 0.0343026, 0.0385705}}, {{3,
0.790575, -0.0101993, 0.0533663}, {5, 0.790575, 0.0283652,
0.0383988}}, {{3, 0.800388, -0.0048569, 0.0483566}, {5, 0.800388,
0.0242973, 0.0424175}}, {{3, 0.810088, 0.0801341, 0.0630511}, {5,
0.810088, 0.0668193, 0.0510046}}, {{5, 0.819675, 0.00868241,
0.10596}}, {{3, 0.82915, -0.0293577, 0.0721665}, {5,
0.82915, -0.00907686, 0.0473577}}, {{3, 0.838525, 0.0299376,
0.0488719}, {5, 0.838525, 0.0330556, 0.0440675}}, {{3,
0.847788, -0.0164008, 0.0556874}, {5, 0.847788, 0.00581662,
0.043372}}};


I'm aware that the fit results from that dataset are rather poor - anyhow that should be irrelevant for the general discussion. Thanks also for the code edit.

EDIT 2: I have not tinkered with things like AccuracyGoal or PrecisionGoal due to my limited knowledge of their impact. However, if that would help already I'd be glad for any advice.

• Please give a minimal example dataset. You have very specialized code and writing an appropriate example dataset would be a pain in the neck. – Henrik Schumacher Aug 27 '18 at 20:46
• @HenrikSchumacher I've added an example set for which Nτ=8 and jack=0. Thanks for looking into it. – gothicVI Aug 27 '18 at 22:07
• @HenrikSchumacher and any ideas from your side? – gothicVI Sep 12 '18 at 8:56
• @gothicVI, when I try copying the code in I get an error that Weights should be a list of real numbers of a pure function. – KraZug Sep 12 '18 at 10:51
• @KraZug I never encountered that. Is it possible that the length of the table becomes zero? Could you try again substituting {beg, 1, Length[formatedDataTrimmed] - 8, 1} with {beg, 1, Length[formatedDataTrimmed] - 10, 1} in the FitF3Tablecombined function? That's one change I made in my local code already to gain a bit of speed up. – gothicVI Sep 12 '18 at 11:02