Say I have some list a. It is an increasing, noisy dataset; something like $a x + X$ where $X$ is a noisy variable. This is as expected from what I'm interested in. However, once every so often my dataset has an extreme ourlier: if we take a zoom in of the list, a = {1,1.2,1.4,1.1,0.9,4,0.9,1.1}, we see that there is suddenly a 4, which is unphysical in my situation. I know what's causing this issue, but I can't really get rid of it before post processing, so I will have to handle it this way.

In any case, what I'm looking for is a command that I could use to replace this 4, or more generally, every value $a_i > b a_{i-1}$ for some parameter $b$ that I can set. Moreover, I'd like the replacement to be such that $a_i = \frac{a_{i-1}+a_{i+1}}{2}$; I want to replace the point by the mean of its neighbours.

Now, I'm thinking this probably starts with something like creating a list of the fractions, where I divide each element by the value of the previous one. Then the comparison has to be made, and then the replacement. But to be honest I'm not quite sure of the most efficient way to tackle this.

Edit: as pointed out in the comments, the boundaries can be tricky. To keep it simple I'd just assume that they are not the problematic points and leave them as they are.

  • $\begingroup$ @Kuba I mean, in a sense it is both, but I do mean how to do it in WL. I can think of how to do it on paper, that is not a problem. Like I said (ignoring the boundaries) just divide each value by the previous one, find the positions where this is larger than $b$, and replace those values by the mean of their neigbours. $\endgroup$ – user129412 Dec 21 '16 at 13:38
  • $\begingroup$ You may be interested in PartitionMap then. $\endgroup$ – Kuba Dec 21 '16 at 13:42
  • $\begingroup$ What should be the output for {4,0.9,1.1,1}? $\endgroup$ – corey979 Dec 21 '16 at 14:12
  • $\begingroup$ @corey979 Yes, the boundary values. I should have written that they should probably just be left alone, as I can generally avoid them being a problem. The way I've currently phrased it though, we are only comparing to the previous value, so it has no influence. The question of {1,0.9,1.1,4} would then be more tricky.. I would like the code to do nothing in that case, I suppose. Ignore the boundaries and leave them alone. $\endgroup$ – user129412 Dec 21 '16 at 14:15
f[a_List, threshold_] := {First@a}~Join~
   If[#[[2]] > #[[1]]*threshold, Mean@Drop[#, {2}], #[[2]]] &, a, 3, 1]~Join~{Last@a}

f[a, 2]

{1, 1.2, 1.4, 1.1, 0.9, 0.9, 0.9, 1.1}

Employs the criterion $If[a_i>ba_{i-1},....]$, where threshold is the $b$.


If you have a lot of data to process I propose a vectorized numeric approach for efficiency.

fn[dat_List, b_?NumericQ] :=
 Module[{Δ, m, d2 = Developer`ToPackedArray[dat, Real]},
    Δ = UnitStep[Differences[d2] - b] ~Prepend~ 0;
    m = ListCorrelate[{1/2, 0, 1/2}, d2, 2];
    Δ*m + (1 - Δ) d2


fn[{1, 1.2, 1.4, 1.1, 0.9, 4, 0.9, 1.1}, 1]
{1., 1.2, 1.4, 1.1, 0.9, 0.9, 0.9, 1.1}

You may use FindPeaks to refine your peak criteria then assign the Mean to these positions while using Nothing for the indices exceeding the boundaries.


dat = {1, 1.2, 1.4, 1.1, 0.9, 4, 0.9, 1.1};


(dat[[#]] = 
     Mean[dat[[{# - 1 /. {0 -> Nothing}, # + 1 /. Length@dat + 1 -> Nothing}]]]) & /@ 
  FindPeaks[dat, 2][[All, 1]];


{1, 1.2, 1.4, 1.1, 0.9, 0.9, 0.9, 1.1}

You may also find the Linear and Nonlinear Filters guide of interest.

Hope this helps.


Edit to add a bit more explanation and fix error in Mean

I would suggest using Partition to get the sublists, then Map or Apply to do the test then recalculate the value:

test[in_List, n_] := If[
  in[[2]] > n*in[[1]], 
  Mean[{in[[1]], in[[3]]}],

recalculate[a_, n_] := Join[
  test[#, n] & /@ Partition[a, 3, 1], 

recalculate[a, 2]
(*{1.2, 1.4, 1.1, 0.9, 0.9, 0.9, 0.9, 1.1, 1.1}*)

I just left the values at the beginning and end, but you could use that If statement to also process them if you like. The shorthands here are:

# and & are a Pure function - see here for more details

@@@ is Apply at level one, so it treats each sublist as the inputs to the pure function

Alternatively using PartitionMap:

<< Developer`
recalculate[a_, n_] := Join[{First[a]}, PartitionMap[test[#, n] &, a, 3, 1], {Last[a]}];

recalculate[a, 2]
(*{1.2, 1.4, 1.1, 0.9, 0.9, 0.9, 0.9, 1.1, 1.1}*)
  • $\begingroup$ Interesting approach, probably close to what I would like to do! I don't think this is exactly what I thought though; in my original example the list has 8 values, but your output only has 6. The values satisfying the criteria should not be removed, just replaced. Although perhaps that is what you do, it just looks like the boundaries fell off; if possible I'd like to avoid that. Even then, I don't really see how 1.9333 is the mean of 0.9 and 0.9, right? $\endgroup$ – user129412 Dec 21 '16 at 13:51
  • $\begingroup$ Ah oops, I included the middle value in the mean as well, I will edit to fix $\endgroup$ – lowriniak Dec 21 '16 at 13:54
  • $\begingroup$ Thanks for the fix! It does still seem to have some problems with the boundary values though. {1,1.2,1.4,1.1,0.9,4,0.9,1.1} is mapped to {1.2, 1.4, 1.1, 0.9, 0.9, 0.9, 0.9, 1.1, 1.1} from which we see that first of all the first value has disappeared, and an additional 1.1 has been added to the end. I don't really understand what in the code causes this though.. $\endgroup$ – user129412 Dec 21 '16 at 14:10
  • $\begingroup$ So that's mostly a feature of Partition and how it splits things. What you could do instead is just add the start and end values back onto the list once you are done (assuming they are being left alone). I'll edit in the code. $\endgroup$ – lowriniak Dec 21 '16 at 14:18
f1 = With[{x = #, y = MeanFilter[#, 1] - #, u = Prepend[UnitStep[Rest@# - #2 Most@#], 0]},
          x + (3/2) u y] &;

f1[a, 2]

{1., 1.2, 1.4, 1.1, 0.9, 0.9, 0.9, 1.1}

  • $\begingroup$ MeanFilter is 29 times slower than ListCorrelate on a list of packed reals in version 10.1. How is it in the version you use? $\endgroup$ – Mr.Wizard Dec 21 '16 at 22:41
  • $\begingroup$ @Mr.Wizard, in version 9 it is about 23 times slower in for a list of 1m reals. $\endgroup$ – kglr Dec 21 '16 at 22:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.