The data referred to as testdat in this question is here if you would like to fiddle with it. Here it is as a ListPlot.


I am trying to plot the "average" of this data and other data like it (and then plot a line using Interpolate), allowing for discontinuities where they are obvious to a human. If there were no jumps anywhere then any of MovingAverage, MovingMedian, MeanShiftFilter get me what I want, but they deal badly with the jump in their own ways, as shown below.



A crude answer that I've thought of is to use MeanShiftFilter, then delete points that aren't near any others. However, I then lose lots of data near the jump and so lose some precision in where the jump is when interpolating. Using MeanShiftFilter with a tighter radius, say 0.1, works quite well for the jump, but then gives too much noise elsewhere in the data where I would like a smooth line.

Better might be to use MovingAverage (with probably a smaller run length than 30), delete the ones in the slope somehow and then put a discontinuity on the middle, but I'm not sure actually how I might do this.

Using something like FindClusters I don't really like as sometimes the jumps are less obvious than this and there may be multiple in a data set, in fact for this reason I feel approaches that look at all the data at once are unlikely to work, and it will be better to deal with each jump as it comes.

  • 1
    $\begingroup$ Why not first locate the jump point(s) using one of your techniques above like MeanShiftFilter. Once they are located, go back to the original data and fit lines to the data between each pair of jump points. $\endgroup$
    – bill s
    Aug 11 '17 at 19:01
  • $\begingroup$ I like this idea, but do you have any suggestions as to what might be the most robust way to locate them? $\endgroup$ Aug 13 '17 at 9:12

The test-data you provided looks different than what you have posted as image

data = Get["https://pastebin.com/raw/ifndXGCW"];

Mathematica graphics

Not so much the general structure of the signal, but the noise seems to have vastly different characteristics. So this is the first thing you need to inspect: What is the noise model behind your data.

After that, you should use a filter that is able to get rid of this kind of noise without affecting real features inside your signal. One idea here is to consider the TotalVariationFilter that has support for different kinds of noise.

Consider for instance this example, which assumes Gaussian noise (which is not what your data suggests!):

{xdata, ydata} = Transpose[data];
    TotalVariationFilter[ydata, reg, Method -> "Gaussian"]
   ], PlotRange -> {Automatic, {0, 1}}
 , {reg, 0, 10}

Mathematica graphics

Which seems to have a sweet-spot at a regularizer value of about 1.6.

Remember that your values are not equidistant but by feeding only y-values into the filter, it assumes so.

  • $\begingroup$ Crap I'm sorry that's just a mistake, I've already filtered the noise in the data for the images in a way that I'm happy with. I'll change the provided data now so that it's the same as what's in the images. $\endgroup$ Aug 11 '17 at 14:14
  • $\begingroup$ I can't help but notice that you work in data processing and seem to know a lot about removing noise. What in your opinion is the noise model? I didn't really assume anything and just sort of attacked it, I'm new to this. Edit: I think this is a question in its own right and I'd need to give lots more details, so maybe ignore me. $\endgroup$ Aug 11 '17 at 14:24
  • $\begingroup$ @George, for determining the noise model, some background information/domain knowledge would be helpful. In other words: where did these data come from? $\endgroup$
    – J. M.'s torpor
    Aug 11 '17 at 14:32
  • $\begingroup$ @GeorgeMoore As J.M. said, it depends on where the data comes from. If you have exact knowledge about the capturing device you used, then it is maybe known, what type of noise it produces. Another idea is to capture a ground truth. Typically, one uses a signal where you are sure there is no signal in it. In images, we can use regions of background for this. Inspecting the values there, you can look at the histogram. Often the noise is a mixture of different kinds. Search for "CCD noise model" for instance. It's rarely possible to eliminate all noise without affecting signal features. $\endgroup$
    – halirutan
    Aug 11 '17 at 14:46
  • $\begingroup$ Thanks both, when I'm in on Monday I'll ask more about the machine used. For context, this is genome sequencing and I'm a recent maths grad doing a summer research project before I start a Master's in Systems Biology. Lots of exciting new stuff for me. $\endgroup$ Aug 13 '17 at 9:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.