4
$\begingroup$

I have to plot a big file with 10 data (data01{x1,y1}, data02{x2,y2}, ...., data10{x10,y10}), for example for data01 (here is the link of data01) I get this plot with some wrong values

enter image description here

ListLogLogPlot[{data01, Table[{t, t}, {t, 0.01, 10, 0.05}]},Joined ->True,Frame -> True, PlotRange -> {{0.01, 10}, {0.01, 10}},PlotRangePadding -> 0,LabelStyle -> Directive[Black],GridLines -> Automatic, GridLinesStyle -> Directive[GrayLevel[0.85]],ImageSize -> 450]

Since I can't detect all these wrong values, how to manipulate these in order to ignore the plot instabilities?

Thank you!

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Exclude data from a list might be of interest $\endgroup$
    – user1066
    Commented Jun 19, 2022 at 11:35
  • $\begingroup$ @user1066 Thank you $\endgroup$
    – Gallagher
    Commented Jun 19, 2022 at 13:21

3 Answers 3

Reset to default
4
$\begingroup$

Discarding points that lie outside the 95% interval.

Clear[x]
lm = LinearModelFit[data01, x, x];
{lower, upper} = lm["SinglePredictionBands", ConfidenceLevel -> 0.95];

Show[ListPlot[data01], Plot[{None, lower, upper},
  {x, data01[[1, 1]], data01[[-1, 1]]}]]

enter image description here

{lowervals, uppervals} = Transpose[
   lm["SinglePredictionConfidenceIntervals", ConfidenceLevel -> 0.95]];

data02 = MapThread[If[And[Last[#1] > #2, Last[#1] < #3], #, Nothing] &,
   {data01, lowervals, uppervals}];

ListLogLogPlot[{data02, Table[{t, t}, {t, 0.01, 10, 0.05}]},
 Joined -> True, Frame -> True, 
 PlotRange -> {{0.01, 10}, {0.01, 10}},
 PlotRangePadding -> 0, LabelStyle -> Directive[Black], GridLines -> Automatic,
 GridLinesStyle -> Directive[GrayLevel[0.85]], ImageSize -> 450]

enter image description here

Improve this answer
$\endgroup$
11
  • $\begingroup$ thank you so much @Chris Degnen, I applied this for another data, but it also gives instabilities, in this case we adjust which parameter, ConfidenceLevel -> 0.95?? $\endgroup$
    – Gallagher
    Commented Jun 19, 2022 at 12:49
  • 1
    $\begingroup$ Yes, you can vary the confidence level, or use a higher order linear model e.g. LinearModelFit[data, {x, x^2}, x], or use a non-linear model, and/or mean prediction intervals instead of single prediction intervals, e.g. demonstrations.wolfram.com/… $\endgroup$
    – Chris Degnen
    Commented Jun 19, 2022 at 12:55
  • $\begingroup$ This is impressive @Chris, I applied ConfidenceLevel -> 1 for all data and it works perfectly, everything is fixed, I am very grateful to you, thank you so much, I hope this post will help other data manipulators. $\endgroup$
    – Gallagher
    Commented Jun 19, 2022 at 13:16
  • $\begingroup$ Can you explain me @Chris the method that you have used here? $\endgroup$
    – Gallagher
    Commented Jun 19, 2022 at 14:14
  • 1
    $\begingroup$ From Wikipedia: "a prediction interval is an estimate of an interval in which a future observation will fall, with a certain probability, given what has already been observed. Prediction intervals are often used in regression analysis." Another example here. $\endgroup$
    – Chris Degnen
    Commented Jun 19, 2022 at 16:58
5
$\begingroup$

Perhaps DeleteAnomalies is what you are looking for:

dataO = DeleteAnomalies[LearnDistribution[MovingMedian[data01 , 150],Method -> "Multinormal"], data01 ];
Show[ListLinePlot[data01],ListPlot[dataO, PlotRange -> Full, PlotStyle -> Green]]

enter image description here

Improve this answer
$\endgroup$
1
  • $\begingroup$ good method, thank you so much $\endgroup$
    – Gallagher
    Commented Jun 19, 2022 at 13:24
5
$\begingroup$

Other than using functions like FindPeak or DeleteAnomalies based on your data we can just fit a line and use it as a clipper to smoothen your data.

fit = FindFit[data01, a x + b, {a, b}, x];
clipper = Function[{x}, Evaluate[a x + b /. fit]]

Function[{x}, -0.0315537 + 0.993757 x]

Now the smoothed data will be like this.

smoothed = If[Abs@(#[[2]] - clipper[#[[1]]]) > 0.5, {#[[1]], line[#[[1]]]}, #] & /@ data01

Now plotting

ListLogLogPlot[{smoothed, Table[{t, t}, {t, 0.01, 10, 0.05}]}, Joined -> True, Frame -> True, PlotRange -> {{0.01, 10}, {0.01, 10}},PlotRangePadding -> 0, LabelStyle -> Directive[Black], GridLines -> Automatic, GridLinesStyle -> Directive[GrayLevel[0.85]],

ImageSize -> 450]

enter image description here

Improve this answer
$\endgroup$
1
  • $\begingroup$ good method, many thanks $\endgroup$
    – Gallagher
    Commented Jun 19, 2022 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.