# Best way to smooth, interpolate and differentiate over a list of data

I have a list given by:

CVlist = {{-14., 1.72277*10^-8}, {-13.9, 1.81891*10^-8}, {-13.8,
1.5097*10^-8}, {-13.7, 1.55127*10^-8}, {-13.6,
2.08915*10^-8}, {-13.5, 2.38538*10^-8}, {-13.4,
1.91246*10^-8}, {-13.3, 2.14892*10^-8}, {-13.2,
2.1801*10^-8}, {-13.1, 2.37498*10^-8}, {-13.,
1.99561*10^-8}, {-12.9, 2.46333*10^-8}, {-12.8,
2.08136*10^-8}, {-12.7, 2.21388*10^-8}, {-12.6,
2.15671*10^-8}, {-12.5, 2.01899*10^-8}, {-12.4,
2.29963*10^-8}, {-12.3, 2.29963*10^-8}, {-12.2,
1.73576*10^-8}, {-12.1, 2.02159*10^-8}, {-12.,
2.07096*10^-8}, {-11.9, 2.04498*10^-8}, {-11.8,
1.96962*10^-8}, {-11.7, 1.72537*10^-8}, {-11.6,
1.88647*10^-8}, {-11.5, 2.1827*10^-8}, {-11.4,
1.96183*10^-8}, {-11.3, 2.07616*10^-8}, {-11.2,
2.07356*10^-8}, {-11.1, 2.07616*10^-8}, {-11.,
1.94884*10^-8}, {-10.9, 2.10214*10^-8}, {-10.8,
1.95923*10^-8}, {-10.7, 1.80852*10^-8}, {-10.6,
1.79293*10^-8}, {-10.5, 2.03458*10^-8}, {-10.4,
2.12293*10^-8}, {-10.3, 2.04498*10^-8}, {-10.2,
1.90466*10^-8}, {-10.1, 2.1723*10^-8}, {-10., 1.97222*10^-8}, {-9.9,
2.09695*10^-8}, {-9.8, 2.03718*10^-8}, {-9.7,
2.29443*10^-8}, {-9.6, 2.22687*10^-8}, {-9.5, 2.10474*10^-8}, {-9.4,
2.05797*10^-8}, {-9.3, 1.82411*10^-8}, {-9.2,
1.74096*10^-8}, {-9.1, 1.81112*10^-8}, {-9., 2.14112*10^-8}, {-8.9,
2.24766*10^-8}, {-8.8, 2.18789*10^-8}, {-8.7, 2.40616*10^-8}, {-8.6,
2.18789*10^-8}, {-8.5, 2.45034*10^-8}, {-8.4,
3.06617*10^-8}, {-8.3, 2.39317*10^-8}, {-8.2, 2.49971*10^-8}, {-8.1,
2.30222*10^-8}, {-8., 2.39577*10^-8}, {-7.9, 2.57766*10^-8}, {-7.8,
2.93625*10^-8}, {-7.7, 3.04018*10^-8}, {-7.6, 3.0142*10^-8}, {-7.5,
3.17011*10^-8}, {-7.4, 3.14412*10^-8}, {-7.3, 3.352*10^-8}, {-7.2,
3.42995*10^-8}, {-7.1, 3.5079*10^-8}, {-7., 3.5079*10^-8}, {-6.9,
3.74177*10^-8}, {-6.8, 3.74177*10^-8}, {-6.7, 3.89767*10^-8}, {-6.6,
4.02759*10^-8}, {-6.5, 4.33941*10^-8}, {-6.4,
4.36539*10^-8}, {-6.3, 4.62524*10^-8}, {-6.2, 4.70319*10^-8}, {-6.1,
4.91107*10^-8}, {-6., 5.06697*10^-8}, {-5.9, 5.27485*10^-8}, {-5.8,
5.56068*10^-8}, {-5.7, 5.66462*10^-8}, {-5.6,
6.00241*10^-8}, {-5.5, 6.21029*10^-8}, {-5.4, 6.57407*10^-8}, {-5.3,
6.80793*10^-8}, {-5.2, 7.06778*10^-8}, {-5.1, 7.5355*10^-8}, {-5.,
7.82133*10^-8}, {-4.9, 8.08117*10^-8}, {-4.8, 8.47094*10^-8}, {-4.7,
8.73079*10^-8}, {-4.6, 9.14654*10^-8}, {-4.5,
9.51032*10^-8}, {-4.4, 9.79615*10^-8}, {-4.3, 1.01859*10^-7}, {-4.2,
1.05757*10^-7}, {-4.1, 1.09655*10^-7}, {-4., 1.12513*10^-7}, {-3.9,
1.15631*10^-7}, {-3.8, 1.18749*10^-7}, {-3.7,
1.21607*10^-7}, {-3.6, 1.24985*10^-7}, {-3.5, 1.28883*10^-7}, {-3.4,
1.30702*10^-7}, {-3.3, 1.3434*10^-7}, {-3.2, 1.37198*10^-7}, {-3.1,
1.40316*10^-7}, {-3., 1.43434*10^-7}, {-2.9, 1.46033*10^-7}, {-2.8,
1.49671*10^-7}, {-2.7, 1.52529*10^-7}, {-2.6,
1.55907*10^-7}, {-2.5, 1.58765*10^-7}, {-2.4, 1.61623*10^-7}, {-2.3,
1.64222*10^-7}, {-2.2, 1.6786*10^-7}, {-2.1, 1.70718*10^-7}, {-2.,
1.73576*10^-7}, {-1.9, 1.76954*10^-7}, {-1.8, 1.79813*10^-7}, {-1.7,
1.8371*10^-7}, {-1.6, 1.87088*10^-7}, {-1.5, 1.90466*10^-7}, {-1.4,
1.93844*10^-7}, {-1.3, 1.97482*10^-7}, {-1.2, 2.0112*10^-7}, {-1.1,
2.04758*10^-7}, {-1., 2.08915*10^-7}, {-0.9, 2.13592*10^-7}, {-0.8,
2.1827*10^-7}, {-0.7, 2.23207*10^-7}, {-0.6, 2.28144*10^-7}, {-0.5,
2.33081*10^-7}, {-0.4, 2.37498*10^-7}, {-0.3,
2.40356*10^-7}, {-0.2, 2.41136*10^-7}, {-0.1, 2.40356*10^-7}, {0.,
2.39577*10^-7}, {0.1, 2.38018*10^-7}, {0.2, 2.35939*10^-7}, {0.3,
2.32561*10^-7}, {0.4, 2.27624*10^-7}, {0.5, 2.21648*10^-7}, {0.6,
2.15411*10^-7}};


The listplot of the data $$y$$ vs $$x$$ looks something like:

Now, I want to smooth the data and then interpolate it so that I can calculate the quantity $$\frac{d(1/y^2)}{dx}$$ as a function of x. What is the best way to go about this data manipulation and how to calculate this derivative?

• Why couldn't you rescale the $y$-coordinates? Jun 11, 2022 at 21:54
• @J.M. We can do that, this is just the raw experimental data. Jun 11, 2022 at 21:55
• Well, after rescaling your data, you could use e.g. a smoothing spline, or you could smooth first with e.g. Savitzky-Golay before throwing the result into Interpolation[]. Jun 11, 2022 at 22:02
• Choosing an appropriate approach really requires more context. Which variations are the wanted signal and which measurement noise? You could try fitting a polynomial Jun 12, 2022 at 11:51
• Related/duplicate: mathematica.stackexchange.com/questions/162118/, mathematica.stackexchange.com/questions/124928/ Jun 14, 2022 at 14:18

qrf = First@ResourceFunction["QuantileRegression"][cvlist, 7, 0.5];

Show[
ListPlot[cvlist, PlotTheme -> "Detailed"],
Plot[{Indeterminate, qrf[x]}, {x, data[[1, 1]], data[[-1, 1]]},
PlotStyle -> Thick]]


Check smoothness (interpolation order is 3, so no surprise; mainly shows that the derivative of qrf can be evaluated):

Plot[qrf'[x], {x, data[[1, 1]], data[[-1, 1]]}]


Inspiration from Anton Antonov's answers to the following and the docs for his ResourceFunction:

Without using any fitting, you can remove the noise from the data using the MovingMap function, like this:

ListLinePlot[CVlist]


ListLinePlot[MovingMap[Mean, CVlist, {3, Center}, "Reflected"]]


Now you can use the Interpolation function to get the derivative.

Well if there is a theoretical model form then definitely start with that form to fit the data. The derivative you are trying to compute could vary drastically depending on the model you choose, so picking one that incorporates the physics or properties of the underlying system would be very important. Since there is no context though I'll proceed with an approximate model shown below to outline the process. (Note: No assumptions of the data were made when choosing this model form, so it likely isn't truly describing the underlying system.)

$$\text{1.5070371729406886\grave{ }*{}^{\wedge}-6}-\text{4.768585174708953\grave{ }*{}^{\wedge}-11} \left(x \left(1. \sqrt[4]{2.71828^x} x-0.5\right)+163.479\right)^2$$

If you have a model form, then it probably isn't necessary to smooth the data unless there are many parameters such that you risk overfitting the noise in the data. Notice how in the plot below, the model essentially is a smooth form of the noisy data.

With a model you can now proceed to compute the derivative. Here you wanted to compute $$d(1/y^2)/dx$$ so we can start by taking our model f(x) and computing $$1/f(x)^2$$. This plot would now look like the example below.

We can then take the derivative using that result as shown below.

deriv = FullSimplify@D[1/((1.5070371729406886*^-6 - 4.768585174708953*^-11 (163.47933475421556 + x (-0.5 + 1. (2.718281828459045^x)^(1/4) x))^2)^2), x]


This gives the following result:

$$\frac{\text{1.9074340698835812\grave{ }*{}^{\wedge}-10} \left(0.25 \sqrt[4]{2.71828^x} x (1. x+8.)-0.5\right) \left(x \left(1. \sqrt[4]{2.71828^x} x-0.5\right)+163.479\right)}{\left(\text{1.5070371729406886\grave{ }*{}^{\wedge}-6}-\text{4.768585174708953\grave{ }*{}^{\wedge}-11} \left(x \left(1. \sqrt[4]{2.71828^x} x-0.5\right)+163.479\right)^2\right)^3}$$

The plot of this derivative looks like this.

• @MichaelE2 The plot is of the derivative of the 1/(f(x)^2) equation. I'll add that plot to the post for comparison. d(1/y^2)/dx
– Nate
Jun 17, 2022 at 15:30
• Oops, I missed that. :) Jun 17, 2022 at 15:44

The simplest solution is :

xmin = Min[CVlist[[All, 1]]];
xmax = Max[CVlist[[All, 1]]];

y = Interpolation[CVlist];

model = a + b x^2 + c x^3 + d x^4 + e x^5 + f x^6 + g x^7 + h x^8 ;

CVfit = FindFit[CVlist, model, {a, b, c, d, e, f, g, h}, x];

modely = Function[{t}, Evaluate[model /. CVfit]];

shift = 10^-8;

Show[Plot[{y[x] + shift, modely[x] - shift}, {x, xmin, xmax},
PlotStyle -> {Black, Gray},
PlotLegends -> {"Interpolated : y(x)", "Model : y(x)"}],
ListLinePlot[CVlist, PlotStyle -> Red, PlotLegends -> {"Data"},
ImageSize -> 600]]



NOTE : Variable shift is used just to show three curves separately. Set this variable to ZERO to get your perfect & smooth fitted curves.

You can use modely to do any further calculations.

My result :