and let me just say thank you for all the help you good people have been providing to stalkers like me through the years.

So, in the next few days or months or years, I'll be acquiring data of the following general form:

enter image description here

My useful signal is the flat, greenish-tint areas, and my problem is the in-between, quite regular transient spikes, marked in red (-ish, of sorts).

Here's some sample data, too.

Now, due to the width and significant weight of the spikes, and the two-step structure of my data, usual filtering techniques don't seem to produce satisfactory results, and I couldn't conjure any other brilliant, generic ways to throw away the spikes, though it does seem rather trivial.

(Bear in mind, however, that everything but the general form of these data is subject to change: more than two useful levels might be present, the periodicity of the signal might vary etc., so I'm looking for a rather generic approach)

Any ideas would be greatly appreciated.

Thank you all.

EDIT: Well, just after I posted the question, I did come up with something that works for my current datasets. Interpolation of data, and then check the derivative for all x against some appropriate value:

fint = Interpolation[data]; (* data is in the form of {x, y} pairs *)
stepData = Select[data, Abs[fint'[#[[1]]]] < 0.05 &]; 
(* TODO: don't use a hardcoded test value here *)

I don't know where this will fail, but regardless, all robust and computationally efficient solutions are warmly welcome! Thanks.


If "good data" is data surrounded by similar values, then using the variance, standard deviation or range of 3 or more neighboring points might be an approach. Below I've used the standard deviation of 3 consecutive points to decide on whether to keep the middle point.

sd = Table[{i, data[[i]], StandardDeviation[data[[i - 1 ;; i + 1]]]}, {i, 2, Length[data] - 1}];
d2 = Select[sd, #[[3]] < 0.005 &][[All, {1, 2}]];
ListPlot[{data, d2}, AspectRatio -> 1/8, ImageSize -> Full, PlotStyle -> {LightGray, Red}]

Original data and cleaned data

  • $\begingroup$ Thank you for your response. This is somewhat similar to what I did with the interpolation, in the sense that you check the local variance against some value. I'll throw some large datasets on both to see which performs better. Thanks! $\endgroup$ – kalt Oct 25 '19 at 10:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.