Applying filters to data of the form {{x1,y1},{x2,y2},...}

Question

When working with any kind of measurement data there is (at least) for me always a phase where I have to play around with different filters (such as MedianFilter, MeanFilter, LowpassFilter ...) to figure out how to improve my data in some aspect (filtering noise, detecting outliers, detecting edges...). Two things have always bugged me when using the build-in filter functions:

1. filters expect simple list like {y1,y2,y3,y4} when (more often than not) measurement data is of the form {{x1,y1}, {x2, y2}, {x3, y3}, {x4, y4}} where $x_i$ is some index (e.g. time or frequency) and $y_i$ is the respective measurement (e.g. voltage or force)

2. filters have a syntax of the form someFilter[data, parameters] and not an operator form someFilter[parameters][data]

This leads often to a Kuddelmuddel of [[]] mixed with a bunch of Transpose and/or intermediate (global) variables

What is a stylistically good way to deal with this?

Answer 1

The answer I came up with is

Clear@applyFilter;
applyFilter[filter_] := Function[data, 
Module[{freq, value},
{freq, value} = Transpose@data;
Transpose[{freq, filter@value}]
]
];

applyFilter[filters__] := RightComposition @@ (applyFilter /@ {filters})

applyFilter[{filter_, n_}] := Nest[applyFilter[filter], # , n] &

The features are:

1. Applying filters in operator form

   applyFilter[MedianFilter[#, 5] &] @ someData
   

2. Chaining filters together

   myfilter = applyFilter[
   MedianFilter[#, 5] &,
   MeanFilter[#,2 ] &
   ]  
   (*used with myfilter @ someData *)
   

3. Multiple filter passes

   applyFilter[{MeanFilter[#, 2] &, numberOfPasses}]
   

and can be used on some example data

example = Transpose[{Range@100, Accumulate@(RandomVariate[NormalDistribution[0, 1], 100])}]

with for instance a MedianFilter to flatten the peaks/remove outliers

applyFilter[MedianFilter[#, 3] &] @ example

or a MedianFilter followed up by a MeanFilter (this can be advantageous compared to using only a MeanFilter if there are huge outliers in the data)

applyFilter[MedianFilter[#, 3] &, MeanFilter[#, 2] &] @ example

or multiple passes of some filter (1 pass through MedianFilter and 10 passes through MeanFilter)

applyFilter[MedianFilter[#, 3] &, {MeanFilter[#, 2] &, 10}] @ example

For me at least, something like applyFilter makes using the build-in filters a lot more user-friendy when experimenting with data.

Manipulate[
ListLinePlot[{
example,
applyFilter[MedianFilter[#, r] &, {MeanFilter[#, r1] &, n}]@example}],

{{r, 0, "Medianfilter radius"}, 0, 10, 1}, Delimiter,
{{n, 0, "Meanfilter passes"}, 0, 10, 1},
{{r1, 0, "Meanfilter radius"}, 0, 10, 1}]