# Filtering and Replacing outliers

I have a set of data that measurements of temperature vs time (date). It is easy to realize when the sensor went wrong as one might see a blip on the data. How to teach to Mathematica to: 1) Detect the "Blip" 2) Delete this Blip 3) Replace the corrupted point for one derived as the mean of the previous data (before the corrupted one) and the one after (after the corrupted one). Thanks in advance !!!

I have tried to setup a test in which each pair of data were compared and if the difference between them was greater than 3 C (or 5C depending how rigorous you want to be). But my reasoning did not work properly.

Ed

The dataset is:

l = {{"2013-11-20 23:00:00", 23.52}, {"2013-11-21 00:00:00",
23.55}, {"2013-11-21 01:00:00", 23.62}, {"2013-11-21 02:00:00",
23.61}, {"2013-11-21 03:00:00", 23.53}, {"2013-11-21 04:00:00",
23.45}, {"2013-11-21 05:00:00", 23.52}, {"2013-11-21 06:00:00",
23.4}, {"2013-11-21 07:00:00", 24.02}, {"2013-11-21 08:00:00",
26.7}, {"2013-11-21 09:00:00", 27.54}, {"2013-11-21 10:00:00",
29.67}, {"2013-11-21 11:00:00", 28.3}, {"2013-11-21 12:00:00",
17.94}, {"2013-11-21 13:00:00", 27.42}, {"2013-11-21 14:00:00",
25.82}, {"2013-11-21 15:00:00", 24.61}, {"2013-11-21 16:00:00",
23.91}, {"2013-11-21 17:00:00", 24.58}, {"2013-11-21 18:00:00",
24.31}, {"2013-11-21 19:00:00", 23.18}, {"2013-11-21 20:00:00",
28.99}, {"2013-11-21 21:00:00", 22.56}, {"2013-11-21 22:00:00",
22.01}}


-
You have TWO outliers DateListPlot[l, PlotRange -> All, PlotStyle -> Large, Joined -> True]} – Dr. belisarius Nov 22 '13 at 21:41
@belisarius, there are 2 obvious ones in this signal. How to classify a point as outlier in general? (eg, GDP, DJIA, seismic data...) – alancalvitti Oct 7 '14 at 18:22
@alancalvitti Now THAT is a completely different question :D- (and not trivial at all). For a serious outlier classification I think the right place to ask is cross validated, not here. – Dr. belisarius Oct 7 '14 at 18:31
@alancalvitti Here is link to a movie showing a conversational engine (programmed in Mathematica) that identifies outliers in temperature data, wind data, stocks, and experimental data: youtube.com/watch?v=wlZ5ANglVI4 . You can just watch between 3:00 and 3:40. The outliers are found by fitting regression quantiles at, say, 0.1 and 0.9 and considering an outlier any point outside of these regression quantiles. – Anton Antonov Oct 1 '15 at 16:30

Filtering by the "second difference" and removing the two most prominent outliers. The rest of the data remain unchanged:

dd = Ordering@Differences@Abs@Differences[l[[All, 2]]];
Show@{DateListPlot[l, PlotRange -> All, PlotStyle -> Large, Joined -> True],
PlotRange -> All, Joined -> True]}


-
Very good. Thanks – user10737 Nov 22 '13 at 22:45

One common filter to remove outliers is the median:

lt = Transpose[l][[2]];
ListPlot[MedianFilter[lt, 1]]


Here it is retaining the dates:

lt = Transpose[l][[2]];
at = Transpose[l][[1]];
DateListPlot[Transpose[{at, MedianFilter[lt, 1]}],
PlotRange -> All, PlotStyle -> Large, Joined -> True]


-
Thanks. It seems to be working pretty good. – user10737 Nov 22 '13 at 22:36
The problem with filtering is that it not only "removes" outliers but also smooths out the rest of the points. Depending on the intended use that could be fatal or a blessing. – Dr. belisarius Oct 7 '14 at 9:12