Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I have a set of data that measurements of temperature vs time (date). It is easy to realize when the sensor went wrong as one might see a blip on the data. How to teach to Mathematica to: 1) Detect the "Blip" 2) Delete this Blip 3) Replace the corrupted point for one derived as the mean of the previous data (before the corrupted one) and the one after (after the corrupted one). Thanks in advance !!!

I have tried to setup a test in which each pair of data were compared and if the difference between them was greater than 3 C (or 5C depending how rigorous you want to be). But my reasoning did not work properly.

Thanks in Advance

Ed

The dataset is:

l = {{"2013-11-20 23:00:00", 23.52}, {"2013-11-21 00:00:00", 
  23.55}, {"2013-11-21 01:00:00", 23.62}, {"2013-11-21 02:00:00", 
  23.61}, {"2013-11-21 03:00:00", 23.53}, {"2013-11-21 04:00:00", 
  23.45}, {"2013-11-21 05:00:00", 23.52}, {"2013-11-21 06:00:00", 
  23.4}, {"2013-11-21 07:00:00", 24.02}, {"2013-11-21 08:00:00", 
  26.7}, {"2013-11-21 09:00:00", 27.54}, {"2013-11-21 10:00:00", 
  29.67}, {"2013-11-21 11:00:00", 28.3}, {"2013-11-21 12:00:00", 
  17.94}, {"2013-11-21 13:00:00", 27.42}, {"2013-11-21 14:00:00", 
  25.82}, {"2013-11-21 15:00:00", 24.61}, {"2013-11-21 16:00:00", 
  23.91}, {"2013-11-21 17:00:00", 24.58}, {"2013-11-21 18:00:00", 
  24.31}, {"2013-11-21 19:00:00", 23.18}, {"2013-11-21 20:00:00", 
  28.99}, {"2013-11-21 21:00:00", 22.56}, {"2013-11-21 22:00:00", 
  22.01}}

TempPlot

share|improve this question
1  
You have TWO outliers DateListPlot[l, PlotRange -> All, PlotStyle -> Large, Joined -> True]} –  belisarius Nov 22 '13 at 21:41
1  
@belisarius, there are 2 obvious ones in this signal. How to classify a point as outlier in general? (eg, GDP, DJIA, seismic data...) –  alancalvitti Oct 7 at 18:22
    
@alancalvitti Now THAT is a completely different question :D- (and not trivial at all). For a serious outlier classification I think the right place to ask is cross validated, not here. –  belisarius Oct 7 at 18:31

2 Answers 2

Filtering by the "second difference" and removing the two most prominent outliers. The rest of the data remain unchanged:

dd = Ordering@Differences@Abs@Differences[l[[All, 2]]];
Show@{DateListPlot[l, PlotRange -> All, PlotStyle -> Large, Joined -> True], 
      DateListPlot[ReplacePart[l, Thread[Rule[dd[[1 ;; 2]], Sequence[]]]],
                                        PlotRange -> All, Joined -> True]}

enter image description here

share|improve this answer
    
Very good. Thanks –  user10737 Nov 22 '13 at 22:45

One common filter to remove outliers is the median:

lt = Transpose[l][[2]];
ListPlot[MedianFilter[lt, 1]]

enter image description here

Here it is retaining the dates:

lt = Transpose[l][[2]];
at = Transpose[l][[1]];
DateListPlot[Transpose[{at, MedianFilter[lt, 1]}], 
      PlotRange -> All, PlotStyle -> Large, Joined -> True]

enter image description here

share|improve this answer
    
Thanks. It seems to be working pretty good. –  user10737 Nov 22 '13 at 22:36
1  
The problem with filtering is that it not only "removes" outliers but also smooths out the rest of the points. Depending on the intended use that could be fatal or a blessing. –  belisarius Oct 7 at 9:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.