1
$\begingroup$

My data looks like this:

https://pastebin.com/n59CTB3L

Or if displayed in log chart:

data in log chart

I want to create an interpolation of the data that would remove most of the noise.

What are some good ways to plot a smooth curve through this data?

$\endgroup$
4

2 Answers 2

13
$\begingroup$

Using Quantile regression might produce results you want -- you have to experiment with the number of knots or the knots locations.

Get data:

Get["https://pastebin.com/raw/n59CTB3L"];    
data = plota1;    
Dimensions[data]

Get the package QuantileRegression.m:

Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/QuantileRegression.m"]

Quantile regression application:

knots = 400;

qFunc = First@
   QuantileRegression[data, knots, {0.5}, 
    Method -> {LinearProgramming, Tolerance -> 10^(-7)}];

Plot data and regression quantile in Log-Log scales:

Show[{
  ListLogLogPlot[data, PlotRange -> All, PlotTheme -> "Detailed", 
   PlotStyle -> GrayLevel[0.8]], 
  ListLogLogPlot[{#, qFunc[#]} & /@ data[[All, 1]], Joined -> True]},
 ImageSize -> 800, 
 PlotLabel -> Row[{"QuantileRegression with ", knots, " knots"}]]

enter image description here

enter image description here

$\endgroup$
12
  • $\begingroup$ this is awesome, thank you! $\endgroup$ Dec 18, 2017 at 7:24
  • $\begingroup$ is there any way to make it find the required number of knots automatically? $\endgroup$ Dec 18, 2017 at 15:51
  • 1
    $\begingroup$ Hm... that is in my TODO list for that package. It is not a simple question, several heuristics can be applied that work well in relatively narrow cases. $\endgroup$ Dec 18, 2017 at 16:40
  • $\begingroup$ Another question. Is there any way I can give different points different weights? $\endgroup$ Dec 21, 2017 at 11:37
  • $\begingroup$ I am not sure what you mean, but maybe adding multiple copies of the points you want to have more of an impact can produce results you want. $\endgroup$ Dec 21, 2017 at 13:05
2
$\begingroup$

If I look at the data I would expect a constant value for increasing x-values. So the approximation could be something with Exp[-...t],for example

NonlinearModelFit[plota1,a0 - a1 Exp[-\[Alpha]1 t] - a2  t Exp[-\[Alpha]2 t] , {a0, a1,a2 , \[Alpha]1, \[Alpha]2 }, t] 
Show[{ListPlot[plota1],Plot[Normal[%], {t, Min[plota1[[All, 1]]], Max[plota1[[All,1]]]},PlotRange -> All]}]

gives this result approx

$\endgroup$
3
  • $\begingroup$ good solution. But in log, the curve falls too slowly, and than rises too slowly (you can see that it misses a cluster of points near 50, and than another huge chunk at around 500) $\endgroup$ Dec 17, 2017 at 16:12
  • $\begingroup$ @ Arsen Zahray: Are you looking for a final approximation in Log-space? Please give some information concerning the related problem. $\endgroup$ Dec 17, 2017 at 16:20
  • $\begingroup$ yes, I'm looking at the data in the log scale. as you can see on the chart, there is some activity in the beginning, than it subsides and for the most part of the observation, nothing really is happening $\endgroup$ Dec 17, 2017 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.