Applying TemporalData to array before filtering gives wrong results

Question

I had a task of filtering data with various timestamps on it, and proceeded as instructed. The first option was to use TimeSeries which works fine but somewhat slow, and the second one TemporalData which operated much faster. However sometimes the second option gives completely bogus results, looking nothing like filtered values at all. Is there a way to fix it?

data = TemporalData[Table[{(i/100.)^2, i/100 + SquareWave[(i/100.)^2] + RandomReal[{-0.1, 0.1}]}, {i, 1,1000}]];
ans = AbsoluteTiming[LowpassFilter[data, Quantity[1, "Hertz"]]];
ans[[1]]
ListLinePlot[{data, ans[[2]]}]

data = Table[{(i/100.)^2, i/100 + SquareWave[(i/100.)^2] + RandomReal[{-0.1, 0.1}]}, {i, 1, 1000}];
ans = AbsoluteTiming[LowpassFilter[TimeSeries@data, Quantity[1, "Hertz"]]];
ans[[1]]
ListLinePlot[{data, ans[[2]]}]

This is how it should look like:

However if you put 1500 points in array instead of 1000 results look exactly the same.

As suggested in comments, this issue can be solved by adding "Path" after the data array, however I was unable to filter an Example of real data using this fix:

\[Tau]1 = TemporalData[a0];
f1 = LowpassFilter[\[Tau]1["Path"], Quantity[1, "Hertz"]];
ListPlot[{\[Tau]1, f1}] 

This filtering doesn't even change the data:

When I remove "Path" weird results appear yet again (yellow points are supposed to be filtered):

Note that if you swap it for TimeSeries@ it works as intended (but much slower, which is my main problem):

\[Tau]1 = TemporalData[a0];
f1 = LowpassFilter[TimeSeries@\[Tau]1, Quantity[1, "Hertz"]];
ListPlot[{\[Tau]1, f1}] 

Answer 1

Diagnosis

Here is an explanation:

1. the real data is not regularly spaced,

2. TemporalData and TimeSeries use different default resampling methods ("HoldValueFromLeft" and "LinearInterpolation" respectively), and

3. with TemporalData's default resampling method "HoldValueFromLeft" the low-pass filtering is faster but produces "the wrong" results.

The code below demonstrates those points.

ClearAll[t1, t2, t3, f1, f2, f3]

t1 = TemporalData[a0];
RepeatedTiming[
 f1 = LowpassFilter[t1, Quantity[1, "Hertz"]];
]
(* {0.121, Null} *)


t2 = TemporalData[a0, ResamplingMethod -> {"Interpolation", InterpolationOrder -> 1}];
RepeatedTiming[
 f2 = LowpassFilter[t2, Quantity[1, "Hertz"]];
]
(* {0.9496, Null} *)

t3 = TimeSeries[a0];
RepeatedTiming[
 f3 = LowpassFilter[t3, Quantity[1, "Hertz"]];
]
(* {0.94, Null} *)

MapThread[ListPlot@*List, {{t1, t2, t3}, {f1, f2, f3}}]

Fix

A possible fix is to re-sample the time series over a regular grid. Or just force a regular grid assumption.

As OP mentioned in a comment we can do:

t4 = TimeSeries@a0; 
RepeatedTiming[int = Table[{i, t2[i]}, {i, 160.1, 174.8, 0.01}];]
(* {0.10, Null} *)

RepeatedTiming[
 f4 = LowpassFilter[TemporalData[int], Quantity[1, "Hertz"]];]   
(* {0.012, Null} *)

Alternatives of using Table and point evaluations are:

1. using the option setting TemporalRegularity -> True, or

2. resampling with TimeSeriesResample.

t5 = TemporalData[a0, TemporalRegularity -> True];
RepeatedTiming[f5 = LowpassFilter[t5, Quantity[1, "Hertz"]];]
(* {0.010, Null} *)

t6 = TimeSeriesResample[TimeSeries[a0]];
RepeatedTiming[f6 = LowpassFilter[t6, Quantity[1, "Hertz"]];]
(* {0.2, Null} *)

It is not clear to me from the TemporalRegularity function page what exactly is the effect of TemporalRegularity->True. After some experimentation I think it just sets a regularity flag, which is respected by algorithms like LowpassFilter.

It seems that TimeSeriesResample uses a regular grid generated with the smallest difference of the time axis coordinates. (At least for the type of time series considered here.) So, I think that is the best way to fix the observed problems.