13
$\begingroup$

I have a set of data that measurements of temperature vs time (date). It is easy to realize when the sensor went wrong as one might see a blip on the data. How to teach to Mathematica to: 1) Detect the "Blip" 2) Delete this Blip 3) Replace the corrupted point for one derived as the mean of the previous data (before the corrupted one) and the one after (after the corrupted one). Thanks in advance !!!

I have tried to setup a test in which each pair of data were compared and if the difference between them was greater than 3 C (or 5C depending how rigorous you want to be). But my reasoning did not work properly.

Thanks in Advance

Ed

The dataset is:

l = {{"2013-11-20 23:00:00", 23.52}, {"2013-11-21 00:00:00", 
  23.55}, {"2013-11-21 01:00:00", 23.62}, {"2013-11-21 02:00:00", 
  23.61}, {"2013-11-21 03:00:00", 23.53}, {"2013-11-21 04:00:00", 
  23.45}, {"2013-11-21 05:00:00", 23.52}, {"2013-11-21 06:00:00", 
  23.4}, {"2013-11-21 07:00:00", 24.02}, {"2013-11-21 08:00:00", 
  26.7}, {"2013-11-21 09:00:00", 27.54}, {"2013-11-21 10:00:00", 
  29.67}, {"2013-11-21 11:00:00", 28.3}, {"2013-11-21 12:00:00", 
  17.94}, {"2013-11-21 13:00:00", 27.42}, {"2013-11-21 14:00:00", 
  25.82}, {"2013-11-21 15:00:00", 24.61}, {"2013-11-21 16:00:00", 
  23.91}, {"2013-11-21 17:00:00", 24.58}, {"2013-11-21 18:00:00", 
  24.31}, {"2013-11-21 19:00:00", 23.18}, {"2013-11-21 20:00:00", 
  28.99}, {"2013-11-21 21:00:00", 22.56}, {"2013-11-21 22:00:00", 
  22.01}}

TempPlot

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ You have TWO outliers DateListPlot[l, PlotRange -> All, PlotStyle -> Large, Joined -> True]} $\endgroup$
    – Dr. belisarius
    Commented Nov 22, 2013 at 21:41
  • 1
    $\begingroup$ @belisarius, there are 2 obvious ones in this signal. How to classify a point as outlier in general? (eg, GDP, DJIA, seismic data...) $\endgroup$
    – alancalvitti
    Commented Oct 7, 2014 at 18:22
  • 1
    $\begingroup$ @alancalvitti Now THAT is a completely different question :D- (and not trivial at all). For a serious outlier classification I think the right place to ask is cross validated, not here. $\endgroup$
    – Dr. belisarius
    Commented Oct 7, 2014 at 18:31
  • $\begingroup$ @alancalvitti Here is link to a movie showing a conversational engine (programmed in Mathematica) that identifies outliers in temperature data, wind data, stocks, and experimental data: youtube.com/watch?v=wlZ5ANglVI4 . You can just watch between 3:00 and 3:40. The outliers are found by fitting regression quantiles at, say, 0.1 and 0.9 and considering an outlier any point outside of these regression quantiles. $\endgroup$
    – Anton Antonov
    Commented Oct 1, 2015 at 16:30

2 Answers 2

Reset to default
10
$\begingroup$

Filtering by the "second difference" and removing the two most prominent outliers. The rest of the data remain unchanged:

dd = Ordering@Differences@Abs@Differences[l[[All, 2]]];
Show@{DateListPlot[l, PlotRange -> All, PlotStyle -> Large, Joined -> True], 
      DateListPlot[ReplacePart[l, Thread[Rule[dd[[1 ;; 2]], Sequence[]]]],
                                        PlotRange -> All, Joined -> True]}

enter image description here

Improve this answer
$\endgroup$
1
  • $\begingroup$ Very good. Thanks $\endgroup$
    – user10737
    Commented Nov 22, 2013 at 22:45
7
$\begingroup$

One common filter to remove outliers is the median:

lt = Transpose[l][[2]];
ListPlot[MedianFilter[lt, 1]]

enter image description here

Here it is retaining the dates:

lt = Transpose[l][[2]];
at = Transpose[l][[1]];
DateListPlot[Transpose[{at, MedianFilter[lt, 1]}], 
      PlotRange -> All, PlotStyle -> Large, Joined -> True]

enter image description here

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks. It seems to be working pretty good. $\endgroup$
    – user10737
    Commented Nov 22, 2013 at 22:36
  • 2
    $\begingroup$ The problem with filtering is that it not only "removes" outliers but also smooths out the rest of the points. Depending on the intended use that could be fatal or a blessing. $\endgroup$
    – Dr. belisarius
    Commented Oct 7, 2014 at 9:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.