# Applying filters to data of the form {{x1,y1},{x2,y2},...}

When working with any kind of measurement data there is (at least) for me always a phase where I have to play around with different filters (such as MedianFilter, MeanFilter, LowpassFilter ...) to figure out how to improve my data in some aspect (filtering noise, detecting outliers, detecting edges...). Two things have always bugged me when using the build-in filter functions:

1. filters expect simple list like {y1,y2,y3,y4} when (more often than not) measurement data is of the form {{x1,y1}, {x2, y2}, {x3, y3}, {x4, y4}} where $x_i$ is some index (e.g. time or frequency) and $y_i$ is the respective measurement (e.g. voltage or force)

2. filters have a syntax of the form someFilter[data, parameters] and not an operator form someFilter[parameters][data]

This leads often to a Kuddelmuddel of [[]] mixed with a bunch of Transpose and/or intermediate (global) variables

What is a stylistically good way to deal with this?

• Recommend that you include a sample data set and a specific example of how you are filtering it now. It is easier to offer alternate methods if there is a specific example with which to compare. Sep 13 '16 at 16:19
• Is a "kuddelmuddel" similar to a "muddle puddle tweetle poodle beetle noodle bottle paddle battle"? Sep 13 '16 at 16:26
• @BobHanlon I included an example in my own answer Sep 13 '16 at 16:43
• @JasonB probably more like "schamozzle" or "hodge podge" Sep 13 '16 at 16:45

The answer I came up with is

Clear@applyFilter;
applyFilter[filter_] := Function[data,
Module[{freq, value},
{freq, value} = Transpose@data;
Transpose[{freq, filter@value}]
]
];

applyFilter[filters__] := RightComposition @@ (applyFilter /@ {filters})

applyFilter[{filter_, n_}] := Nest[applyFilter[filter], # , n] &


The features are:

1. Applying filters in operator form

applyFilter[MedianFilter[#, 5] &] @ someData

2. Chaining filters together

myfilter = applyFilter[
MedianFilter[#, 5] &,
MeanFilter[#,2 ] &
]
(*used with myfilter @ someData *)

3. Multiple filter passes

applyFilter[{MeanFilter[#, 2] &, numberOfPasses}]


and can be used on some example data

example = Transpose[{Range@100, Accumulate@(RandomVariate[NormalDistribution[0, 1], 100])}]


with for instance a MedianFilter to flatten the peaks/remove outliers

applyFilter[MedianFilter[#, 3] &] @ example


or a MedianFilter followed up by a MeanFilter (this can be advantageous compared to using only a MeanFilter if there are huge outliers in the data)

applyFilter[MedianFilter[#, 3] &, MeanFilter[#, 2] &] @ example


or multiple passes of some filter (1 pass through MedianFilter and 10 passes through MeanFilter)

applyFilter[MedianFilter[#, 3] &, {MeanFilter[#, 2] &, 10}] @ example


For me at least, something like applyFilter makes using the build-in filters a lot more user-friendy when experimenting with data.

Manipulate[
ListLinePlot[{
example,
applyFilter[MedianFilter[#, r] &, {MeanFilter[#, r1] &, n}]@example}],

{{r, 0, "Medianfilter radius"}, 0, 10, 1}, Delimiter,
{{n, 0, "Meanfilter passes"}, 0, 10, 1},
{{r1, 0, "Meanfilter radius"}, 0, 10, 1}]


• you can use MapAt in your applyFilter to save from copying the data to temporary variables. applyFilter[filter_] := Function[data, Transpose@MapAt[filter, Transpose@data, {2}]] Sep 13 '16 at 16:45
• @george2079 Thanks for your suggestion! If you want I can add it to my answer. Sep 13 '16 at 16:52