# Applying TemporalData to array before filtering gives wrong results

I had a task of filtering data with various timestamps on it, and proceeded as instructed. The first option was to use TimeSeries which works fine but somewhat slow, and the second one TemporalData which operated much faster. However sometimes the second option gives completely bogus results, looking nothing like filtered values at all. Is there a way to fix it?

data = TemporalData[Table[{(i/100.)^2, i/100 + SquareWave[(i/100.)^2] + RandomReal[{-0.1, 0.1}]}, {i, 1,1000}]];
ans = AbsoluteTiming[LowpassFilter[data, Quantity[1, "Hertz"]]];
ans[[1]]
ListLinePlot[{data, ans[[2]]}]


data = Table[{(i/100.)^2, i/100 + SquareWave[(i/100.)^2] + RandomReal[{-0.1, 0.1}]}, {i, 1, 1000}];
ans = AbsoluteTiming[LowpassFilter[TimeSeries@data, Quantity[1, "Hertz"]]];
ans[[1]]
ListLinePlot[{data, ans[[2]]}]


This is how it should look like:

However if you put 1500 points in array instead of 1000 results look exactly the same.

As suggested in comments, this issue can be solved by adding "Path" after the data array, however I was unable to filter an Example of real data using this fix:

\[Tau]1 = TemporalData[a0];
f1 = LowpassFilter[\[Tau]1["Path"], Quantity[1, "Hertz"]];
ListPlot[{\[Tau]1, f1}]


This filtering doesn't even change the data:

When I remove "Path" weird results appear yet again (yellow points are supposed to be filtered):

Note that if you swap it for TimeSeries@ it works as intended (but much slower, which is my main problem):

\[Tau]1 = TemporalData[a0];
f1 = LowpassFilter[TimeSeries@\[Tau]1, Quantity[1, "Hertz"]];
ListPlot[{\[Tau]1, f1}]


• Please write a clearer, searchable title. Jul 2, 2018 at 0:07
• use ans = AbsoluteTiming[LowpassFilter[data["Path"], Quantity[1, "Hertz"]]]; in the first code block?
– kglr
Jul 2, 2018 at 1:26
• @kglr thanks, if fixes the toy example but does not work for real data. I updated the question. Jul 2, 2018 at 15:06

## Diagnosis

Here is an explanation:

1. the real data is not regularly spaced,

2. TemporalData and TimeSeries use different default resampling methods ("HoldValueFromLeft" and "LinearInterpolation" respectively), and

3. with TemporalData's default resampling method "HoldValueFromLeft" the low-pass filtering is faster but produces "the wrong" results.

The code below demonstrates those points.

ClearAll[t1, t2, t3, f1, f2, f3]

t1 = TemporalData[a0];
RepeatedTiming[
f1 = LowpassFilter[t1, Quantity[1, "Hertz"]];
]
(* {0.121, Null} *)

t2 = TemporalData[a0, ResamplingMethod -> {"Interpolation", InterpolationOrder -> 1}];
RepeatedTiming[
f2 = LowpassFilter[t2, Quantity[1, "Hertz"]];
]
(* {0.9496, Null} *)

t3 = TimeSeries[a0];
RepeatedTiming[
f3 = LowpassFilter[t3, Quantity[1, "Hertz"]];
]
(* {0.94, Null} *)

MapThread[ListPlot@*List, {{t1, t2, t3}, {f1, f2, f3}}]


## Fix

A possible fix is to re-sample the time series over a regular grid. Or just force a regular grid assumption.

As OP mentioned in a comment we can do:

t4 = TimeSeries@a0;
RepeatedTiming[int = Table[{i, t2[i]}, {i, 160.1, 174.8, 0.01}];]
(* {0.10, Null} *)

RepeatedTiming[
f4 = LowpassFilter[TemporalData[int], Quantity[1, "Hertz"]];]
(* {0.012, Null} *)


Alternatives of using Table and point evaluations are:

1. using the option setting TemporalRegularity -> True, or

2. resampling with TimeSeriesResample.

t5 = TemporalData[a0, TemporalRegularity -> True];
RepeatedTiming[f5 = LowpassFilter[t5, Quantity[1, "Hertz"]];]
(* {0.010, Null} *)

t6 = TimeSeriesResample[TimeSeries[a0]];
RepeatedTiming[f6 = LowpassFilter[t6, Quantity[1, "Hertz"]];]
(* {0.2, Null} *)


It is not clear to me from the TemporalRegularity function page what exactly is the effect of TemporalRegularity->True. After some experimentation I think it just sets a regularity flag, which is respected by algorithms like LowpassFilter.

It seems that TimeSeriesResample uses a regular grid generated with the smallest difference of the time axis coordinates. (At least for the type of time series considered here.) So, I think that is the best way to fix the observed problems.

• So how exactly the filtered data looks like with option "holdvaluefromleft"? Does it drops down to zero frequently? Anyways, is this t2 = TimeSeries@a0; int = Table[{i, t2[i]}, {i, 160.1, 174.8, 0.01}]; RepeatedTiming[ f2 = LowpassFilter[TemporalData[int], Quantity[1, "Hertz"]]] the right way to do fast filtering then? Jul 4, 2018 at 3:49
• @VsevolodA. Yes, I think re-sampling the time series over a regular grid is the right approach. Further comments are given in the answer update. Jul 4, 2018 at 11:43
• This solves my issue I guess, but still the question remains: why filter fails on certain kinds of interpolation. They all should work fine. Maybe that's a bug. Jul 5, 2018 at 4:23
• @VsevolodA. I am not sure the low pass filtering fails. The behavior we observe might be explained given the data. Jul 5, 2018 at 9:33
• Filtered value cannot lie outside of the interval of the unfiltered one, whatever the interpolation is. Jul 5, 2018 at 16:56