# Using TotalVariationFilter on dataset with missing data

TotalVariationFilter does not work on a dataset with missing data. Does anybody has a good idea to make it work?

I don't know if first splitting the dataset into parts with no missing data, applying the filter on each part and then gathering them is a good idea?

• If there is only few data values missing, you can try to replace the missing values by other values, e.g. the average of the dataset or by noise (Gaussian or salt and pepper? Should be adjusted to the expected noise model in the dataset). These filters are made for removing noise so interpreting missing values as such might be a good idea. If course, it would be better if one could convince TotalVariationFilter to ignore missing values for the fidelity term in the objective function but I doubt that this is possible. – Henrik Schumacher Aug 23 '18 at 7:41

This is a workaround in some cases. Imagine data:

data = {1, 3, 3, Missing[], 4, 3, 2};


TotalVariationFilter would not work:

TotalVariationFilter[data]


TotalVariationFilter::arg1: Expecting an image or a non-empty real numeric array instead of {1,3,3,Missing[],4,3,2}.

Try the trick:

repairMissing[data_, n_] :=TimeSeries[data, MissingDataMethod ->
{"Interpolation", InterpolationOrder -> n}]["Values"]


Now this will work:

Grid[{repairMissing[data, 1],
TotalVariationFilter[repairMissing[data, 1]]}]


Alternatively you could work just simply in TimeSeries format:

{ListLinePlot[data, PlotTheme -> "Business"],
ListLinePlot[
TotalVariationFilter[
TimeSeries[data,
MissingDataMethod -> {"Interpolation", InterpolationOrder -> 1}]],