4
$\begingroup$

TotalVariationFilter does not work on a dataset with missing data. Does anybody has a good idea to make it work?

I don't know if first splitting the dataset into parts with no missing data, applying the filter on each part and then gathering them is a good idea?

Thanks for your help?

$\endgroup$
  • $\begingroup$ If there is only few data values missing, you can try to replace the missing values by other values, e.g. the average of the dataset or by noise (Gaussian or salt and pepper? Should be adjusted to the expected noise model in the dataset). These filters are made for removing noise so interpreting missing values as such might be a good idea. If course, it would be better if one could convince TotalVariationFilter to ignore missing values for the fidelity term in the objective function but I doubt that this is possible. $\endgroup$ – Henrik Schumacher Aug 23 '18 at 7:41
3
$\begingroup$

This is a workaround in some cases. Imagine data:

data = {1, 3, 3, Missing[], 4, 3, 2};

TotalVariationFilter would not work:

TotalVariationFilter[data]

TotalVariationFilter::arg1: Expecting an image or a non-empty real numeric array instead of {1,3,3,Missing[],4,3,2}.

Try the trick:

repairMissing[data_, n_] :=TimeSeries[data, MissingDataMethod -> 
{"Interpolation", InterpolationOrder -> n}]["Values"]

Now this will work:

Grid[{repairMissing[data, 1],
  TotalVariationFilter[repairMissing[data, 1]]}]

enter image description here

Alternatively you could work just simply in TimeSeries format:

{ListLinePlot[data, PlotTheme -> "Business"],
 ListLinePlot[
  TotalVariationFilter[
   TimeSeries[data, 
    MissingDataMethod -> {"Interpolation", InterpolationOrder -> 1}]],
   PlotTheme -> "Business"]}

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.