3
$\begingroup$

I had a task of filtering data with various timestamps on it, and proceeded as instructed. The first option was to use TimeSeries which works fine but somewhat slow, and the second one TemporalData which operated much faster. However sometimes the second option gives completely bogus results, looking nothing like filtered values at all. Is there a way to fix it?

data = TemporalData[Table[{(i/100.)^2, i/100 + SquareWave[(i/100.)^2] + RandomReal[{-0.1, 0.1}]}, {i, 1,1000}]];
ans = AbsoluteTiming[LowpassFilter[data, Quantity[1, "Hertz"]]];
ans[[1]]
ListLinePlot[{data, ans[[2]]}]

enter image description here

data = Table[{(i/100.)^2, i/100 + SquareWave[(i/100.)^2] + RandomReal[{-0.1, 0.1}]}, {i, 1, 1000}];
ans = AbsoluteTiming[LowpassFilter[TimeSeries@data, Quantity[1, "Hertz"]]];
ans[[1]]
ListLinePlot[{data, ans[[2]]}]

This is how it should look like: enter image description here

However if you put 1500 points in array instead of 1000 results look exactly the same.

As suggested in comments, this issue can be solved by adding "Path" after the data array, however I was unable to filter an Example of real data using this fix:

\[Tau]1 = TemporalData[a0];
f1 = LowpassFilter[\[Tau]1["Path"], Quantity[1, "Hertz"]];
ListPlot[{\[Tau]1, f1}] 

This filtering doesn't even change the data: enter image description here

When I remove "Path" weird results appear yet again (yellow points are supposed to be filtered): enter image description here

Note that if you swap it for TimeSeries@ it works as intended (but much slower, which is my main problem):

\[Tau]1 = TemporalData[a0];
f1 = LowpassFilter[TimeSeries@\[Tau]1, Quantity[1, "Hertz"]];
ListPlot[{\[Tau]1, f1}] 

enter image description here

$\endgroup$
  • 2
    $\begingroup$ Please write a clearer, searchable title. $\endgroup$ – David G. Stork Jul 2 '18 at 0:07
  • $\begingroup$ use ans = AbsoluteTiming[LowpassFilter[data["Path"], Quantity[1, "Hertz"]]]; in the first code block? $\endgroup$ – kglr Jul 2 '18 at 1:26
  • $\begingroup$ @kglr thanks, if fixes the toy example but does not work for real data. I updated the question. $\endgroup$ – Vsevolod A. Jul 2 '18 at 15:06
5
+200
$\begingroup$

Diagnosis

Here is an explanation:

  1. the real data is not regularly spaced,

  2. TemporalData and TimeSeries use different default resampling methods ("HoldValueFromLeft" and "LinearInterpolation" respectively), and

  3. with TemporalData's default resampling method "HoldValueFromLeft" the low-pass filtering is faster but produces "the wrong" results.

The code below demonstrates those points.

ClearAll[t1, t2, t3, f1, f2, f3]

t1 = TemporalData[a0];
RepeatedTiming[
 f1 = LowpassFilter[t1, Quantity[1, "Hertz"]];
]
(* {0.121, Null} *)


t2 = TemporalData[a0, ResamplingMethod -> {"Interpolation", InterpolationOrder -> 1}];
RepeatedTiming[
 f2 = LowpassFilter[t2, Quantity[1, "Hertz"]];
]
(* {0.9496, Null} *)

t3 = TimeSeries[a0];
RepeatedTiming[
 f3 = LowpassFilter[t3, Quantity[1, "Hertz"]];
]
(* {0.94, Null} *)

MapThread[ListPlot@*List, {{t1, t2, t3}, {f1, f2, f3}}]

enter image description here

enter image description here

Fix

A possible fix is to re-sample the time series over a regular grid. Or just force a regular grid assumption.

As OP mentioned in a comment we can do:

t4 = TimeSeries@a0; 
RepeatedTiming[int = Table[{i, t2[i]}, {i, 160.1, 174.8, 0.01}];]
(* {0.10, Null} *)

RepeatedTiming[
 f4 = LowpassFilter[TemporalData[int], Quantity[1, "Hertz"]];]   
(* {0.012, Null} *)

Alternatives of using Table and point evaluations are:

1. using the option setting TemporalRegularity -> True, or

2. resampling with TimeSeriesResample.

t5 = TemporalData[a0, TemporalRegularity -> True];
RepeatedTiming[f5 = LowpassFilter[t5, Quantity[1, "Hertz"]];]
(* {0.010, Null} *)

t6 = TimeSeriesResample[TimeSeries[a0]];
RepeatedTiming[f6 = LowpassFilter[t6, Quantity[1, "Hertz"]];]
(* {0.2, Null} *)

It is not clear to me from the TemporalRegularity function page what exactly is the effect of TemporalRegularity->True. After some experimentation I think it just sets a regularity flag, which is respected by algorithms like LowpassFilter.

It seems that TimeSeriesResample uses a regular grid generated with the smallest difference of the time axis coordinates. (At least for the type of time series considered here.) So, I think that is the best way to fix the observed problems.

$\endgroup$
  • $\begingroup$ So how exactly the filtered data looks like with option "holdvaluefromleft"? Does it drops down to zero frequently? Anyways, is this t2 = TimeSeries@a0; int = Table[{i, t2[i]}, {i, 160.1, 174.8, 0.01}]; RepeatedTiming[ f2 = LowpassFilter[TemporalData[int], Quantity[1, "Hertz"]]] the right way to do fast filtering then? $\endgroup$ – Vsevolod A. Jul 4 '18 at 3:49
  • $\begingroup$ @VsevolodA. Yes, I think re-sampling the time series over a regular grid is the right approach. Further comments are given in the answer update. $\endgroup$ – Anton Antonov Jul 4 '18 at 11:43
  • $\begingroup$ This solves my issue I guess, but still the question remains: why filter fails on certain kinds of interpolation. They all should work fine. Maybe that's a bug. $\endgroup$ – Vsevolod A. Jul 5 '18 at 4:23
  • $\begingroup$ @VsevolodA. I am not sure the low pass filtering fails. The behavior we observe might be explained given the data. $\endgroup$ – Anton Antonov Jul 5 '18 at 9:33
  • $\begingroup$ Filtered value cannot lie outside of the interval of the unfiltered one, whatever the interpolation is. $\endgroup$ – Vsevolod A. Jul 5 '18 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.