4
$\begingroup$

I have to plot a big file with 10 data (data01{x1,y1}, data02{x2,y2}, ...., data10{x10,y10}), for example for data01 (here is the link of data01) I get this plot with some wrong values

enter image description here

ListLogLogPlot[{data01, Table[{t, t}, {t, 0.01, 10, 0.05}]},Joined ->True,Frame -> True, PlotRange -> {{0.01, 10}, {0.01, 10}},PlotRangePadding -> 0,LabelStyle -> Directive[Black],GridLines -> Automatic, GridLinesStyle -> Directive[GrayLevel[0.85]],ImageSize -> 450]

Since I can't detect all these wrong values, how to manipulate these in order to ignore the plot instabilities?

Thank you!

$\endgroup$
2

3 Answers 3

4
$\begingroup$

Discarding points that lie outside the 95% interval.

Clear[x]
lm = LinearModelFit[data01, x, x];
{lower, upper} = lm["SinglePredictionBands", ConfidenceLevel -> 0.95];

Show[ListPlot[data01], Plot[{None, lower, upper},
  {x, data01[[1, 1]], data01[[-1, 1]]}]]

enter image description here

{lowervals, uppervals} = Transpose[
   lm["SinglePredictionConfidenceIntervals", ConfidenceLevel -> 0.95]];

data02 = MapThread[If[And[Last[#1] > #2, Last[#1] < #3], #, Nothing] &,
   {data01, lowervals, uppervals}];

ListLogLogPlot[{data02, Table[{t, t}, {t, 0.01, 10, 0.05}]},
 Joined -> True, Frame -> True, 
 PlotRange -> {{0.01, 10}, {0.01, 10}},
 PlotRangePadding -> 0, LabelStyle -> Directive[Black], GridLines -> Automatic,
 GridLinesStyle -> Directive[GrayLevel[0.85]], ImageSize -> 450]

enter image description here

$\endgroup$
11
  • $\begingroup$ thank you so much @Chris Degnen, I applied this for another data, but it also gives instabilities, in this case we adjust which parameter, ConfidenceLevel -> 0.95?? $\endgroup$
    – Gallagher
    Jun 19 at 12:49
  • 1
    $\begingroup$ Yes, you can vary the confidence level, or use a higher order linear model e.g. LinearModelFit[data, {x, x^2}, x], or use a non-linear model, and/or mean prediction intervals instead of single prediction intervals, e.g. demonstrations.wolfram.com/… $\endgroup$ Jun 19 at 12:55
  • $\begingroup$ This is impressive @Chris, I applied ConfidenceLevel -> 1 for all data and it works perfectly, everything is fixed, I am very grateful to you, thank you so much, I hope this post will help other data manipulators. $\endgroup$
    – Gallagher
    Jun 19 at 13:16
  • $\begingroup$ Can you explain me @Chris the method that you have used here? $\endgroup$
    – Gallagher
    Jun 19 at 14:14
  • 1
    $\begingroup$ From Wikipedia: "a prediction interval is an estimate of an interval in which a future observation will fall, with a certain probability, given what has already been observed. Prediction intervals are often used in regression analysis." Another example here. $\endgroup$ Jun 19 at 16:58
5
$\begingroup$

Perhaps DeleteAnomalies is what you are looking for:

dataO = DeleteAnomalies[LearnDistribution[MovingMedian[data01 , 150],Method -> "Multinormal"], data01 ];
Show[ListLinePlot[data01],ListPlot[dataO, PlotRange -> Full, PlotStyle -> Green]]

enter image description here

$\endgroup$
1
  • $\begingroup$ good method, thank you so much $\endgroup$
    – Gallagher
    Jun 19 at 13:24
5
$\begingroup$

Other than using functions like FindPeak or DeleteAnomalies based on your data we can just fit a line and use it as a clipper to smoothen your data.

fit = FindFit[data01, a x + b, {a, b}, x];
clipper = Function[{x}, Evaluate[a x + b /. fit]]

Function[{x}, -0.0315537 + 0.993757 x]

Now the smoothed data will be like this.

smoothed = If[Abs@(#[[2]] - clipper[#[[1]]]) > 0.5, {#[[1]], line[#[[1]]]}, #] & /@ data01

Now plotting

ListLogLogPlot[{smoothed, Table[{t, t}, {t, 0.01, 10, 0.05}]}, Joined -> True, Frame -> True, PlotRange -> {{0.01, 10}, {0.01, 10}},PlotRangePadding -> 0, LabelStyle -> Directive[Black], GridLines -> Automatic, GridLinesStyle -> Directive[GrayLevel[0.85]],

ImageSize -> 450]

enter image description here

$\endgroup$
1
  • $\begingroup$ good method, many thanks $\endgroup$
    – Gallagher
    Jun 19 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.