Filtering and Replacing outliers

I have a set of data that measurements of temperature vs time (date). It is easy to realize when the sensor went wrong as one might see a blip on the data. How to teach to Mathematica to: 1) Detect the "Blip" 2) Delete this Blip 3) Replace the corrupted point for one derived as the mean of the previous data (before the corrupted one) and the one after (after the corrupted one). Thanks in advance !!!

I have tried to setup a test in which each pair of data were compared and if the difference between them was greater than 3 C (or 5C depending how rigorous you want to be). But my reasoning did not work properly.

Ed

The dataset is:

l = {{"2013-11-20 23:00:00", 23.52}, {"2013-11-21 00:00:00",
23.55}, {"2013-11-21 01:00:00", 23.62}, {"2013-11-21 02:00:00",
23.61}, {"2013-11-21 03:00:00", 23.53}, {"2013-11-21 04:00:00",
23.45}, {"2013-11-21 05:00:00", 23.52}, {"2013-11-21 06:00:00",
23.4}, {"2013-11-21 07:00:00", 24.02}, {"2013-11-21 08:00:00",
26.7}, {"2013-11-21 09:00:00", 27.54}, {"2013-11-21 10:00:00",
29.67}, {"2013-11-21 11:00:00", 28.3}, {"2013-11-21 12:00:00",
17.94}, {"2013-11-21 13:00:00", 27.42}, {"2013-11-21 14:00:00",
25.82}, {"2013-11-21 15:00:00", 24.61}, {"2013-11-21 16:00:00",
23.91}, {"2013-11-21 17:00:00", 24.58}, {"2013-11-21 18:00:00",
24.31}, {"2013-11-21 19:00:00", 23.18}, {"2013-11-21 20:00:00",
28.99}, {"2013-11-21 21:00:00", 22.56}, {"2013-11-21 22:00:00",
22.01}}


• You have TWO outliers DateListPlot[l, PlotRange -> All, PlotStyle -> Large, Joined -> True]} Commented Nov 22, 2013 at 21:41
• @belisarius, there are 2 obvious ones in this signal. How to classify a point as outlier in general? (eg, GDP, DJIA, seismic data...) Commented Oct 7, 2014 at 18:22
• @alancalvitti Now THAT is a completely different question :D- (and not trivial at all). For a serious outlier classification I think the right place to ask is cross validated, not here. Commented Oct 7, 2014 at 18:31
• @alancalvitti Here is link to a movie showing a conversational engine (programmed in Mathematica) that identifies outliers in temperature data, wind data, stocks, and experimental data: youtube.com/watch?v=wlZ5ANglVI4 . You can just watch between 3:00 and 3:40. The outliers are found by fitting regression quantiles at, say, 0.1 and 0.9 and considering an outlier any point outside of these regression quantiles. Commented Oct 1, 2015 at 16:30

Filtering by the "second difference" and removing the two most prominent outliers. The rest of the data remain unchanged:

dd = Ordering@Differences@Abs@Differences[l[[All, 2]]];
Show@{DateListPlot[l, PlotRange -> All, PlotStyle -> Large, Joined -> True],
DateListPlot[ReplacePart[l, Thread[Rule[dd[[1 ;; 2]], Sequence[]]]],
PlotRange -> All, Joined -> True]}


• Very good. Thanks Commented Nov 22, 2013 at 22:45

One common filter to remove outliers is the median:

lt = Transpose[l][[2]];
ListPlot[MedianFilter[lt, 1]]


Here it is retaining the dates:

lt = Transpose[l][[2]];
at = Transpose[l][[1]];
DateListPlot[Transpose[{at, MedianFilter[lt, 1]}],
PlotRange -> All, PlotStyle -> Large, Joined -> True]


• Thanks. It seems to be working pretty good. Commented Nov 22, 2013 at 22:36
• The problem with filtering is that it not only "removes" outliers but also smooths out the rest of the points. Depending on the intended use that could be fatal or a blessing. Commented Oct 7, 2014 at 9:12