The Stack Overflow podcast is back! Listen to an interview with our new CEO.
6 added 496 characters in body
source | link

I also got interested in this problem and solved it using Quantile Regression. See my blog post Finding local extrema in noisy data using Quantile Regression .

The proposed Quantile Regression algorithm is a version of the polynomial fitting solution proposed by Leonid Shifrin above, and has the following advantages: (i) it requires less parameter tweaking, and (ii) more importantly it is more robust with very noisy and oscillating data.

That robustness is achieved by using two regression fitted curves: one close to the local minima and another close to the local maxima, computed for low and high quantiles respectively. Quantile Regression is uniquely able to do that -- I look for good examples where this feature is of decisive importance.

Here is an example of very noisy and oscillating data that has 10,000 points:

n = 1000;
xs = N@Rescale[Range[n], {1, n}, {0, 60}];
data3 = Flatten[
   Table[Transpose[{xs + 0.1 RandomReal[{-1, 1}, Length[xs]], 
      Map[5 Sinc[#] + Sin[#] + 4 Sin[1/4 #] &, xs] + 
       1.4 RandomVariate[SkewNormalDistribution[0, 1, 0.9], 
         Length[xs]]}], {10}], 1];
ListPlot[data3, PlotRange -> All]

Here is code for finding the regression quantiles functions, local extrema, and making a plot like the ones below:

Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/Applications/QuantileRegressionForLocalExtrema.m"]

{qfuncs3, extrema3} = QRFindExtrema[data3, 24, 2, 150, {0.02, 0.98}];

Show[{ListPlot[Join[{data3}, extrema3], 
   PlotStyle -> {{}, {PointSize[Medium], Red}, {PointSize[Medium], 
      Green}}],
  Plot[Through[qfuncs3[x]], {x, Min[data3[[All, 1]]], 
    Max[data3[[All, 1]]]}, PlotStyle -> {Orange}, 
   PerformanceGoal -> "Speed"]}, Frame -> True, PlotRange -> All]

This image shows the data, the fitted regression quantiles, and the estimated local extrema:

enter image description here

Here is another example with more timid data (again ~ 10,000 points):

enter image description here

Update

I would like to point out that since we find two regression quantile curves we can use two nearest neighbors finding functions: one with the points below the low regression quantile, and one with the points above the high regression quantile.

I changed the implementation linked above to also take an option should the Nearest functions for finding the extrema be constructed using all data points or just the outliers (the points outside of the found regression quantiles).

I also got interested in this problem and solved it using Quantile Regression. See my blog post Finding local extrema in noisy data using Quantile Regression .

The proposed Quantile Regression algorithm is a version of the polynomial fitting solution proposed by Leonid Shifrin above, and has the following advantages: (i) it requires less parameter tweaking, and (ii) more importantly it is more robust with very noisy and oscillating data.

That robustness is achieved by using two regression fitted curves: one close to the local minima and another close to the local maxima, computed for low and high quantiles respectively. Quantile Regression is uniquely able to do that -- I look for good examples where this feature is of decisive importance.

Here is an example of very noisy and oscillating data that has 10,000 points:

n = 1000;
xs = N@Rescale[Range[n], {1, n}, {0, 60}];
data3 = Flatten[
   Table[Transpose[{xs + 0.1 RandomReal[{-1, 1}, Length[xs]], 
      Map[5 Sinc[#] + Sin[#] + 4 Sin[1/4 #] &, xs] + 
       1.4 RandomVariate[SkewNormalDistribution[0, 1, 0.9], 
         Length[xs]]}], {10}], 1];
ListPlot[data3, PlotRange -> All]

Here is code for finding the regression quantiles functions, local extrema, and making a plot like the ones below:

Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/Applications/QuantileRegressionForLocalExtrema.m"]

{qfuncs3, extrema3} = QRFindExtrema[data3, 24, 2, 150, {0.02, 0.98}];

Show[{ListPlot[Join[{data3}, extrema3], 
   PlotStyle -> {{}, {PointSize[Medium], Red}, {PointSize[Medium], 
      Green}}],
  Plot[Through[qfuncs3[x]], {x, Min[data3[[All, 1]]], 
    Max[data3[[All, 1]]]}, PlotStyle -> {Orange}, 
   PerformanceGoal -> "Speed"]}, Frame -> True, PlotRange -> All]

This image shows the data, the fitted regression quantiles, and the estimated local extrema:

enter image description here

Here is another example with more timid data (again ~ 10,000 points):

enter image description here

I also got interested in this problem and solved it using Quantile Regression. See my blog post Finding local extrema in noisy data using Quantile Regression .

The proposed Quantile Regression algorithm is a version of the polynomial fitting solution proposed by Leonid Shifrin above, and has the following advantages: (i) it requires less parameter tweaking, and (ii) more importantly it is more robust with very noisy and oscillating data.

That robustness is achieved by using two regression fitted curves: one close to the local minima and another close to the local maxima, computed for low and high quantiles respectively. Quantile Regression is uniquely able to do that -- I look for good examples where this feature is of decisive importance.

Here is an example of very noisy and oscillating data that has 10,000 points:

n = 1000;
xs = N@Rescale[Range[n], {1, n}, {0, 60}];
data3 = Flatten[
   Table[Transpose[{xs + 0.1 RandomReal[{-1, 1}, Length[xs]], 
      Map[5 Sinc[#] + Sin[#] + 4 Sin[1/4 #] &, xs] + 
       1.4 RandomVariate[SkewNormalDistribution[0, 1, 0.9], 
         Length[xs]]}], {10}], 1];
ListPlot[data3, PlotRange -> All]

Here is code for finding the regression quantiles functions, local extrema, and making a plot like the ones below:

Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/Applications/QuantileRegressionForLocalExtrema.m"]

{qfuncs3, extrema3} = QRFindExtrema[data3, 24, 2, 150, {0.02, 0.98}];

Show[{ListPlot[Join[{data3}, extrema3], 
   PlotStyle -> {{}, {PointSize[Medium], Red}, {PointSize[Medium], 
      Green}}],
  Plot[Through[qfuncs3[x]], {x, Min[data3[[All, 1]]], 
    Max[data3[[All, 1]]]}, PlotStyle -> {Orange}, 
   PerformanceGoal -> "Speed"]}, Frame -> True, PlotRange -> All]

This image shows the data, the fitted regression quantiles, and the estimated local extrema:

enter image description here

Here is another example with more timid data (again ~ 10,000 points):

enter image description here

Update

I would like to point out that since we find two regression quantile curves we can use two nearest neighbors finding functions: one with the points below the low regression quantile, and one with the points above the high regression quantile.

I changed the implementation linked above to also take an option should the Nearest functions for finding the extrema be constructed using all data points or just the outliers (the points outside of the found regression quantiles).

5 Re-arranged text.
source | link

I also got interested in this problem and solved it using Quantile Regression. See my blog post Finding local extrema in noisy data using Quantile Regression .

The proposed Quantile Regression algorithm is a version of the polynomial fitting solution proposed by Leonid Shifrin above, and has the following advantages: (i) it requires less parameter tweaking, and (ii) more importantly it is more robust with very noisy and oscillating data.

That robustness is achieved by using two regression fitted curves: one close to the local minima and another close to the local maxima, computed for low and high quantiles respectively. Quantile Regression is uniquely able to do that -- I look for good examples where this feature is of decisive importance.

Here is an example withof very noisy and oscillating data ofthat has 10,000 points:

n = 1000;
xs = N@Rescale[Range[n], {1, n}, {0, 60}];
data3 = Flatten[
   Table[Transpose[{xs + 0.1 RandomReal[{-1, 1}, Length[xs]], 
      Map[5 Sinc[#] + Sin[#] + 4 Sin[1/4 #] &, xs] + 
       1.4 RandomVariate[SkewNormalDistribution[0, 1, 0.9], 
         Length[xs]]}], {10}], 1];
ListPlot[data3, PlotRange -> All]

This image shows the data, the fitted regression quantiles, and the estimated local extrema:

enter image description here

Here is another example with more timid data (again ~ 10,000 points):

enter image description here

Here is code for finding the regression quantiles functions, local extrema, and making a plot like the ones abovebelow:

Import["https://raw.githubusercontent.com/antononcube/\MathematicaForPredictionMathematicaForPrediction/master/Applications/QuantileRegressionForLocalExtrema.m"]

{qfuncs3, extrema3} = QRFindExtrema[data3, 24, 2, 150, {0.02, 0.98}];

Show[{ListPlot[Join[{data3}, extrema3], 
   PlotStyle -> {{}, {PointSize[Medium], Red}, {PointSize[Medium], 
      Green}}],
  Plot[Through[qfuncs3[x]], {x, Min[data3[[All, 1]]], 
    Max[data3[[All, 1]]]}, PlotStyle -> {Orange}, 
   PerformanceGoal -> "Speed"]}, Frame -> True, PlotRange -> All]

This image shows the data, the fitted regression quantiles, and the estimated local extrema:

enter image description here

Here is another example with more timid data (again ~ 10,000 points):

enter image description here

I also got interested in this problem and solved it using Quantile Regression. See my blog post Finding local extrema in noisy data using Quantile Regression .

The proposed Quantile Regression algorithm is a version of the polynomial fitting solution proposed by Leonid Shifrin above, and has the following advantages: (i) it requires less parameter tweaking, and (ii) more importantly it is more robust with very noisy and oscillating data.

That robustness is achieved by using two regression fitted curves: one close to the local minima and another close to the local maxima, computed for low and high quantiles respectively. Quantile Regression is uniquely able to do that -- I look for good examples where this feature is of decisive importance.

Here is an example with very noisy and oscillating data of 10,000 points:

n = 1000;
xs = N@Rescale[Range[n], {1, n}, {0, 60}];
data3 = Flatten[
   Table[Transpose[{xs + 0.1 RandomReal[{-1, 1}, Length[xs]], 
      Map[5 Sinc[#] + Sin[#] + 4 Sin[1/4 #] &, xs] + 
       1.4 RandomVariate[SkewNormalDistribution[0, 1, 0.9], 
         Length[xs]]}], {10}], 1];
ListPlot[data3, PlotRange -> All]

This image shows the data, the fitted regression quantiles, and the estimated local extrema:

enter image description here

Here is another example with more timid data (again ~ 10,000 points):

enter image description here

Here is code for finding the regression quantiles functions, local extrema, and making a plot like the ones above:

Import["https://raw.githubusercontent.com/antononcube/\MathematicaForPrediction/master/Applications/QuantileRegressionForLocalExtrema.m"]

{qfuncs3, extrema3} = QRFindExtrema[data3, 24, 2, 150, {0.02, 0.98}];

Show[{ListPlot[Join[{data3}, extrema3], 
   PlotStyle -> {{}, {PointSize[Medium], Red}, {PointSize[Medium], 
      Green}}],
  Plot[Through[qfuncs3[x]], {x, Min[data3[[All, 1]]], 
    Max[data3[[All, 1]]]}, PlotStyle -> {Orange}, 
   PerformanceGoal -> "Speed"]}, Frame -> True, PlotRange -> All]

I also got interested in this problem and solved it using Quantile Regression. See my blog post Finding local extrema in noisy data using Quantile Regression .

The proposed Quantile Regression algorithm is a version of the polynomial fitting solution proposed by Leonid Shifrin above, and has the following advantages: (i) it requires less parameter tweaking, and (ii) more importantly it is more robust with very noisy and oscillating data.

That robustness is achieved by using two regression fitted curves: one close to the local minima and another close to the local maxima, computed for low and high quantiles respectively. Quantile Regression is uniquely able to do that -- I look for good examples where this feature is of decisive importance.

Here is an example of very noisy and oscillating data that has 10,000 points:

n = 1000;
xs = N@Rescale[Range[n], {1, n}, {0, 60}];
data3 = Flatten[
   Table[Transpose[{xs + 0.1 RandomReal[{-1, 1}, Length[xs]], 
      Map[5 Sinc[#] + Sin[#] + 4 Sin[1/4 #] &, xs] + 
       1.4 RandomVariate[SkewNormalDistribution[0, 1, 0.9], 
         Length[xs]]}], {10}], 1];
ListPlot[data3, PlotRange -> All]

Here is code for finding the regression quantiles functions, local extrema, and making a plot like the ones below:

Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/Applications/QuantileRegressionForLocalExtrema.m"]

{qfuncs3, extrema3} = QRFindExtrema[data3, 24, 2, 150, {0.02, 0.98}];

Show[{ListPlot[Join[{data3}, extrema3], 
   PlotStyle -> {{}, {PointSize[Medium], Red}, {PointSize[Medium], 
      Green}}],
  Plot[Through[qfuncs3[x]], {x, Min[data3[[All, 1]]], 
    Max[data3[[All, 1]]]}, PlotStyle -> {Orange}, 
   PerformanceGoal -> "Speed"]}, Frame -> True, PlotRange -> All]

This image shows the data, the fitted regression quantiles, and the estimated local extrema:

enter image description here

Here is another example with more timid data (again ~ 10,000 points):

enter image description here

4 Added for finding the regression quantile functions, local extrema, and plotting of results.
source | link

I also got interested in this problem and solved it using Quantile Regression. See my blog post Finding local extrema in noisy data using Quantile Regression .

The proposed Quantile Regression algorithm is a version of the polynomial fitting solution proposed by Leonid Shifrin above, and has the following advantages: (i) it requires less parameter tweaking, and (ii) more importantly it is more robust with very noisy and oscillating data.

That robustness is achieved by using two regression fitted curves: one close to the local minima and another close to the local maxima, computed for low and high quantiles respectively. Quantile Regression is uniquely able to do that -- I look for good examples where this feature is of decisive importance.

Here is an example with very noisy and oscillating data of 10,000 points:

n = 1000;
xs = N@Rescale[Range[n], {1, n}, {0, 60}];
data3 = Flatten[
   Table[Transpose[{xs + 0.1 RandomReal[{-1, 1}, Length[xs]], 
      Map[5 Sinc[#] + Sin[#] + 4 Sin[1/4 #] &, xs] + 
       1.4 RandomVariate[SkewNormalDistribution[0, 1, 0.9], 
         Length[xs]]}], {10}], 1];
ListPlot[data3, PlotRange -> All]

This image shows the data, the fitted regression quantiles, and the estimated local extrema:

enter image description here

Here is another example with more timid data (again ~ 10,000 points):

enter image description here

Here is code for finding the regression quantiles functions, local extrema, and making a plot like the ones above:

Import["https://raw.githubusercontent.com/antononcube/\MathematicaForPrediction/master/Applications/QuantileRegressionForLocalExtrema.m"]

{qfuncs3, extrema3} = QRFindExtrema[data3, 24, 2, 150, {0.02, 0.98}];

Show[{ListPlot[Join[{data3}, extrema3], 
   PlotStyle -> {{}, {PointSize[Medium], Red}, {PointSize[Medium], 
      Green}}],
  Plot[Through[qfuncs3[x]], {x, Min[data3[[All, 1]]], 
    Max[data3[[All, 1]]]}, PlotStyle -> {Orange}, 
   PerformanceGoal -> "Speed"]}, Frame -> True, PlotRange -> All]

I also got interested in this problem and solved it using Quantile Regression. See my blog post Finding local extrema in noisy data using Quantile Regression .

The proposed Quantile Regression algorithm is a version of the polynomial fitting solution proposed by Leonid Shifrin above, and has the following advantages: (i) it requires less parameter tweaking, and (ii) more importantly it is more robust with very noisy and oscillating data.

That robustness is achieved by using two regression fitted curves: one close to the local minima and another close to the local maxima, computed for low and high quantiles respectively. Quantile Regression is uniquely able to do that -- I look for good examples where this feature is of decisive importance.

Here is an example with very noisy and oscillating data of 10,000 points:

n = 1000;
xs = N@Rescale[Range[n], {1, n}, {0, 60}];
data3 = Flatten[
   Table[Transpose[{xs + 0.1 RandomReal[{-1, 1}, Length[xs]], 
      Map[5 Sinc[#] + Sin[#] + 4 Sin[1/4 #] &, xs] + 
       1.4 RandomVariate[SkewNormalDistribution[0, 1, 0.9], 
         Length[xs]]}], {10}], 1];
ListPlot[data3, PlotRange -> All]

This image shows the data, the fitted regression quantiles, and the estimated local extrema:

enter image description here

Here is another example with more timid data (again ~ 10,000 points):

enter image description here

I also got interested in this problem and solved it using Quantile Regression. See my blog post Finding local extrema in noisy data using Quantile Regression .

The proposed Quantile Regression algorithm is a version of the polynomial fitting solution proposed by Leonid Shifrin above, and has the following advantages: (i) it requires less parameter tweaking, and (ii) more importantly it is more robust with very noisy and oscillating data.

That robustness is achieved by using two regression fitted curves: one close to the local minima and another close to the local maxima, computed for low and high quantiles respectively. Quantile Regression is uniquely able to do that -- I look for good examples where this feature is of decisive importance.

Here is an example with very noisy and oscillating data of 10,000 points:

n = 1000;
xs = N@Rescale[Range[n], {1, n}, {0, 60}];
data3 = Flatten[
   Table[Transpose[{xs + 0.1 RandomReal[{-1, 1}, Length[xs]], 
      Map[5 Sinc[#] + Sin[#] + 4 Sin[1/4 #] &, xs] + 
       1.4 RandomVariate[SkewNormalDistribution[0, 1, 0.9], 
         Length[xs]]}], {10}], 1];
ListPlot[data3, PlotRange -> All]

This image shows the data, the fitted regression quantiles, and the estimated local extrema:

enter image description here

Here is another example with more timid data (again ~ 10,000 points):

enter image description here

Here is code for finding the regression quantiles functions, local extrema, and making a plot like the ones above:

Import["https://raw.githubusercontent.com/antononcube/\MathematicaForPrediction/master/Applications/QuantileRegressionForLocalExtrema.m"]

{qfuncs3, extrema3} = QRFindExtrema[data3, 24, 2, 150, {0.02, 0.98}];

Show[{ListPlot[Join[{data3}, extrema3], 
   PlotStyle -> {{}, {PointSize[Medium], Red}, {PointSize[Medium], 
      Green}}],
  Plot[Through[qfuncs3[x]], {x, Min[data3[[All, 1]]], 
    Max[data3[[All, 1]]]}, PlotStyle -> {Orange}, 
   PerformanceGoal -> "Speed"]}, Frame -> True, PlotRange -> All]
3 deleted 2 characters in body
source | link
2 added 693 characters in body
source | link
1
source | link