I have an x-y data list. I now need to denoise the data and make it smooth. The next step is to find the position with the largest slope and determine the x coordinate of this point.

The data list is here [1]

  • 2
    $\begingroup$ Please post code representing your attempts. $\endgroup$
    – Alan
    Commented Aug 9, 2023 at 13:37
  • $\begingroup$ See eg Numerical derivative from data points, Numerical differentiation methods, Derivative of an interpolating function, $\endgroup$
    – MarcoB
    Commented Aug 9, 2023 at 14:23
  • 1
    $\begingroup$ You'll probably want to run a low pass filter in your data, interpolate the results, and then use the interpolating function to calculate the derivative to find the highest slope. But derivatives of interpolating functions are often noisy, hence the methods in the links above. $\endgroup$
    – MarcoB
    Commented Aug 9, 2023 at 14:24

2 Answers 2

(* Read data *)
data = Import["C:\\Users\\Desktop\\data.tsv", "Table"];
(* Work only with y data and smooth via SG method *)
smoothed = ListConvolve[SavitzkyGolayMatrix[{10}, 2], data[[All, 2]]];
(* Plot *)
ListLinePlot[{data[[All, 2]], smoothed}]

enter image description here

(* Use SG method to determine smoothed derivative SG method has unexpected inversion so add "-" *)
    diffsmoothed = -ListConvolve[SavitzkyGolayMatrix[{10}, 2, 1], data[[All,2]]];
    ListLinePlot[diffsmoothed, PlotRange -> All]
    (* If you wanted largest slope regardless of sign use Abs[diffsmoothed] in the next line *)
    data[[Ordering[diffsmoothed][[1]], 1]]

enter image description here

Answer is near -13.1125 but will vary with smoothing

  • $\begingroup$ thank you very much! $\endgroup$
    – Mr.2023
    Commented Aug 10, 2023 at 11:36
  • $\begingroup$ Hi, @OpticsMan. Can we use other smoothing methods? I want to get the location of this point of maximum slope, the more precise the better. Besides the SG method, is there another method? What do you recommend? Do we need to preprocess the raw data? Thanks $\endgroup$
    – Mr.2023
    Commented Aug 10, 2023 at 11:44
  • 2
    $\begingroup$ @Mr.2023 Mathematica includes many other filter types Butterwortk, Chebyshev, etc. search "signalProcessing" in help. I use SG primarily due to my personal experience and its understandable polynomial nature and the easy ability to do derivatives with it. Often I am looking for slopes, peaks, troughs, etc. in "low frequency" spectral data and do not have specific concerns about the noise bandwidth etc as many filters can provide similar passbands regardless of optimization. $\endgroup$
    – OpticsMan
    Commented Aug 10, 2023 at 14:09
  • $\begingroup$ Thanks for your warm and detailed help! $\endgroup$
    – Mr.2023
    Commented Aug 11, 2023 at 1:30
data = Import["C:/data.xlsx.tsv", "Table"][[4 ;;]];
filtered = MeanFilter[TimeSeries[data], 0.25];
ListLinePlot[{data, filtered}, PlotStyle -> {Green, Red}]

enter image description here

Now that denoising has been done:

f = Interpolation[filtered];
solmin = NMinimize[f'[t], {t, -15, 0}]

{-399.662, -2.70942}

solmax = NMaximize[f'[t], {t, -15, 0}]

{179.365, {t -> -13.0257}}

Plot[{f[t], f'[t]}
 , {t, -15, 0}
 , PlotStyle -> {Red, GrayLevel[0.6]}
 , Epilog -> {
   Blue, AbsolutePointSize[8],
   Point@{ t /. Last@solmin, First@solmin}
   , Point@{ t /. Last@solmax, First@solmax}

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.