Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I have a set of data that measurements of temperature vs time (date). It is easy to realize when the sensor went wrong as one might see a blip on the data. How to teach to Mathematica to: 1) Detect the "Blip" 2) Delete this Blip 3) Replace the corrupted point for one derived as the mean of the previous data (before the corrupted one) and the one after (after the corrupted one). Thanks in advance !!!

I have tried to setup a test in which each pair of data were compared and if the difference between them was greater than 3 C (or 5C depending how rigorous you want to be). But my reasoning did not work properly.

Thanks in Advance

Ed

The dataset is:

l = {{"2013-11-20 23:00:00", 23.52}, {"2013-11-21 00:00:00", 
  23.55}, {"2013-11-21 01:00:00", 23.62}, {"2013-11-21 02:00:00", 
  23.61}, {"2013-11-21 03:00:00", 23.53}, {"2013-11-21 04:00:00", 
  23.45}, {"2013-11-21 05:00:00", 23.52}, {"2013-11-21 06:00:00", 
  23.4}, {"2013-11-21 07:00:00", 24.02}, {"2013-11-21 08:00:00", 
  26.7}, {"2013-11-21 09:00:00", 27.54}, {"2013-11-21 10:00:00", 
  29.67}, {"2013-11-21 11:00:00", 28.3}, {"2013-11-21 12:00:00", 
  17.94}, {"2013-11-21 13:00:00", 27.42}, {"2013-11-21 14:00:00", 
  25.82}, {"2013-11-21 15:00:00", 24.61}, {"2013-11-21 16:00:00", 
  23.91}, {"2013-11-21 17:00:00", 24.58}, {"2013-11-21 18:00:00", 
  24.31}, {"2013-11-21 19:00:00", 23.18}, {"2013-11-21 20:00:00", 
  28.99}, {"2013-11-21 21:00:00", 22.56}, {"2013-11-21 22:00:00", 
  22.01}}

TempPlot

share|improve this question
    
You have TWO outliers DateListPlot[l, PlotRange -> All, PlotStyle -> Large, Joined -> True]} –  belisarius Nov 22 '13 at 21:41

2 Answers 2

Filtering by the "second difference" and removing two outliers. The rest of the data remain unchanged:

dd = Ordering@Differences@Abs@Differences[l[[All, 2]]];
Show@{DateListPlot[l, PlotRange -> All, PlotStyle -> Large, Joined -> True], 
      DateListPlot[ReplacePart[l, Thread[Rule[dd[[1 ;; 2]], Sequence[]]]],
                                        PlotRange -> All, Joined -> True]}

enter image description here

share|improve this answer
    
Very good. Thanks –  user10737 Nov 22 '13 at 22:45

One common filter to remove outliers is the median:

lt = Transpose[l][[2]];
ListPlot[MedianFilter[lt, 1]]

enter image description here

Here it is retaining the dates:

lt = Transpose[l][[2]];
at = Transpose[l][[1]];
DateListPlot[Transpose[{at, MedianFilter[lt, 1]}], 
      PlotRange -> All, PlotStyle -> Large, Joined -> True]

enter image description here

share|improve this answer
    
Thanks. It seems to be working pretty good. –  user10737 Nov 22 '13 at 22:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.