Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

I have a data list where the elements have the form (time, velocity ). I need to remove some malformed elements in places where the data does not change smoothly and then replace those points by interpolating the elements before and after them.

Here is an example of my data:

{{27.342, -0.01}, {27.443, -0.03}, {27.546, -0.01}, {27.743, -0.01}, {27.945, -0.01}, 
 {28.145, -0.03}, {28.246, -0.05}, {28.346, -0.06}, {28.547, -0.01}, {28.747, -0.01}, 
 {29.149, -0.01}, {29.249, 0}, {29.45, -0.01}, {29.651, -0.05}, {29.852, -0.05}, 
 {30.053, -0.01}, {30.153, -0.01}}

For instance the 7th and 8th elements are malformed since the wind cannot change from -0.3 to -0.05 and -0.6 and then back to -0.01 in 0.1 sec0, so I want to replace them with -0.03 and -0.02.

share|improve this question
1  
@farzane How about providing the condition to keep/substitute them? –  Öskå Aug 8 at 9:54
3  
BTW, see Differences reference.wolfram.com/language/ref/Differences.html –  David Carraher Aug 8 at 9:56
1  
Be more precise and explain your problem clearly. If not, I'm afraid that your question might be closed for being unclear. –  Öskå Aug 8 at 10:06
5  
Here's the thing. If you want a good answer you have to ask a good question. You need to sit back and think about what you need help with, then edit your question and include every important detail that will help us understand clearly. Then we can offer you a solution. Otherwise trying to cut corners in asking a decent question will lead to more back and forth in the comment section that will eventually lead to this question being closed. –  RunnyKine Aug 8 at 10:29
2  
In addition to what @RunnyKine said, here's what actually happened: you ask a question that seems to be answered by Differences. Then it turns out what you actually want is a filter to replace data if (roughly) the gradient is higher than some value by some interpolated value. But this is only clear from reading the comments. It would help in getting an answer if you included the info from the comments in the question (maybe with a short version of your data and some rough attempt to do what you want). Basically, imagine you don't know what the question is. –  acl Aug 8 at 10:37

3 Answers 3

x = y = RandomReal[1, 100];

p = Position[Differences@x, n_Real /; n > 0.1] + 1 // Flatten;

Table[y[[n]] = (y[[n - 1]] + y[[n + 1]])/2, {n, {p}}];

ListLinePlot[{x, y},
 ImageSize -> 600,
 PlotLegends -> {"original", "substituted"}]

enter image description here

share|improve this answer

For example, modifying data if first value is > -0.2 :-

data = {{27.342, -0.01}, {27.443, -0.03}, {27.546, -0.01},
   {27.743, -0.01}, {27.945, -0.01}, {28.145, -0.03},
   {28.246, -0.05}, {28.346, -0.06}, {28.547, -0.01},
   {28.747, -0.01}, {29.149, -0.01}, {29.249, 0},
   {29.45, -0.01}, {29.651, -0.05}, {29.852, -0.05},
   {30.053, -0.01}, {30.153, -0.01}};

Length[data]

17

datapairs = Partition[data, 2, 1];

takedifferences[onepair_] := Module[{a, b},
  {a, b} = onepair;
  a - b]

diffs = Map[takedifferences, datapairs];

Length[diffs]

16

datatriples = Partition[diffs, 3, 1];

process[onetriple_] := Module[{a1, a2, b1, b2, c1, c2},
  {{a1, a2}, {b1, b2}, {c1, c2}} = onetriple;
  If[b1 > -0.2, b1 = Mean[{a1, c1}]];
  {b1, b2}]

processed = Join[{First[diffs]}, Map[process, datatriples], {Last[diffs]}];

Length[processed]

16

ListLinePlot[{First /@ diffs, First /@ processed}]

enter image description here

share|improve this answer
    
Added more thorough processing ;-) –  Chris Degnen Aug 8 at 11:25

Perhaps something along the following lines?

data = {{27.342, -0.01}, {27.443, -0.03}, {27.546, -0.01}, {27.743, -0.01},
        {27.945, -0.01}, {28.145, -0.03}, {28.246, -0.05}, {28.346, -0.06},
        {28.547, -0.01}, {28.747, -0.01}, {29.149, -0.01}, {29.249,  0}, 
        {29.45, -0.01}, {29.651, -0.05}, {29.852, -0.05}, {30.053, -0.01},
        {30.153, -0.01}}; (* from @ChrisDegnen's answer *)

threshold = .02;
t = UnitStep @ PadLeft[Abs@Differences[data[[All, 2]]] - threshold, Length[data]];
mf = MeanFilter[data[[All, 2]], 1];
newdata = Transpose[{data[[All, 1]], (1 - t) data[[All, 2]] + t mf}];
ListLinePlot[{data, newdata},  PlotLegends -> {"data", "newdata"}]

enter image description here

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.