How can I manipulate TemporalData?

Question

Version 9 introduced TemporalData but, truth be told, the documentation doesn't suggest that one can do much with it. Many of us have probably assumed we'd be better off sticking with the date-value pairs familiar from use in DateListPlot and elsewhere.

Are there some (perhaps undocumented) functions I can use to manipulate temporal data after putting it in TemporalData form?

Answer 1

The short answer is, yes! There is a whole undocumented package TemporalData` containing some useful functions.

The results below are from my own spelunking. Feel free to add/amend as appropriate.

Let's set up some simple TemporalData objects to explore them:

fakedata = 
  Transpose@{DatePlus[{2001, 1}, {#, "Month"}] & /@ Range[0, 99], 
    Accumulate[RandomVariate[NormalDistribution[0, 1], {100}]] - 2};
temp = TemporalData[fakedata];
fakedatalater = 
  Transpose@{DatePlus[{2010, 5}, {#, "Month"}] & /@ Range[0, 99], 
    Accumulate[RandomVariate[NormalDistribution[0, 1], {100}]] - 2};
templater = TemporalData[fakedatalater]

DateListPlot[temp["Paths"], Joined -> True]

Here are the functions revealed by the command ?TemporalData`*. Some of them have usage messages (these are given where exist, with all the developer typos) and are all ReadProtected.

Aggregate

Aggregate[td,dt,f] aggregates each path over time intervals of width dt using aggregating function f, where dt can be a number, a date increment such as "Month" or a list {n,t} where n is a number and t is a date increment.

Aggregate[td,dt] uses Mean of the intervals of width dt to aggregate.

aggd = TemporalData`Aggregate[temp, {3, "Month"}];
DateListPlot[aggd["Paths"], Joined -> True]

You can aggregate several ways. The default is Mean, but Variance, StandardDeviation, Total and Median are also possible, as are some more obscure aggregation methods like Quantile[#, 0.95] &, Skewness,Kurtosis, TrimmedMean[#, 0.2] &, GeometricMean, HarmonicMean, ContraharmonicMean (including things like ContraharmonicMean[#, 4] &):

aggd = TemporalData`Aggregate[temp, {3, "Month"}, Total];
DateListPlot[aggd["Paths"], Joined -> True]

In fact as far as I can tell, pretty much anything that condenses a vector of numeric values to a single number works, e.g. Mean[Abs[#]] &.

Caveat: aggregating at the "Month" frequency might introduce shifts in starting days!

TemporalData assumes that a month is always 31 days long (it simply adds 2678400 seconds to the AbsoluteTime values at each step):

td = TemporalData[{DatePlus[{2001, 1}, # - 1], #} & /@ Range@500];
Differences[TemporalData`Aggregate[td, {1, "Month"}]["Times"][[1]]]/3600/24

(* {31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31, 31} *)

which results in accumulating error in days (i.e. next aggregate won't start at the 1st of the month).

DateSpecification

DateSpecification is an internal wrapper for implicit TemporalData time specification, not available for direct use. According to the definition of ExtendTimes, TemporalData[...]["UnexpandedRawTimes"] can return the time for each path in one of the following formats:

1. implicit {mint, maxt, dt} in AbsoluteTime format
2. a vector of explicit time values, e.g. {1, 2, 3, ...}
3. implicit DateSpecification[mint, maxt, dt] in DateList format

The double-wrapped time specification is interpreted as the first 3 entries in calendar-date-format, and is expanded correctly, assuming 1 day-increment and automatically calculating the end-date from the number of data points:

TemporalData[Range@5, {{2001, 10, 2}}]["UnexpandedRawTimes"]

(* {TemporalData`DateSpecification[{2001, 10, 2, 0, 0, 0.},
                                   {2001, 10, 6, 0, 0, 0}, {1, "Day"}]} *)

Compare it with a single-wrapped time specification, that is interpreted as any iterator: "from 1 to 10 with steps of 2":

td = TemporalData[Range@5, {1, 10, 2}];
td["UnexpandedRawTimes"]

(* {{1, 10, 2}} *)

td["Times"]

(* {{1, 3, 5, 7, 9}} *)

It uses the same argument specification DateListPlot accepts, where $end$ and $start$ dates must comply with the $stepsize$ AND with the number of datapoints, so a lot of naive combinations won't work (see further details under ExtendTimes). Increment by 1 week works, as the time span is 5 weeks, and there are exactly 5 datapoints:

DateList /@ 
  TemporalData[Range@5, {{2001, 1, 1}, {2001, 1, 29}, {1, "Week"}}]["Times"][[1]]

(* {{2001, 1, 1, 0, 0, 0.}, {2001, 1, 8, 0, 0, 0.}, {2001, 1, 15, 0, 0, 0.},
    {2001, 1, 22, 0, 0, 0.}, {2001, 1, 29, 0, 0, 0.}} *)

Increment by 2 days and Automatic endpoint (number of time divisions is defined by number of datapoints):

DateList /@ 
  TemporalData[Range@5, {{2001, 1, 1, 0, 0, 0}, Automatic, {2, "Day"}}]["Times"][[1]]

(* {{2001, 1, 1, 0, 0, 0.}, {2001, 1, 3, 0, 0, 0.}, {2001, 1, 5, 0, 0, 0.},
    {2001, 1, 7, 0, 0, 0.}, {2001, 1, 9, 0, 0, 0.}} *)

More detail is under DateListPlot's specification, Details and Options section.

DropTimes

DropTimes drops data points from a discrete time series. By default it works if time values are single numbers (e.g. {1, 2, 3, ...}). If time is specified in e.g. DateList format, it has to be converted to AbsoluteTime for DropTimes, as TemporalData automatically converts DateList-type date specifications to AbsoluteTime-format:

temp["Times"][[1, 1 ;; 5]]

(*  {3187296000, 3189974400, 3192393600, 3195072000, 3197664000} *)

By using AbsoluteTime on the time value to be removed, it works:

DateList /@ TemporalData`DropTimes[temp, AbsoluteTime@{2001, 2, 1}]["Times"][[1, ;; 5]]

(* {{2001, 1, 1, 0, 0, 0.}, {2001, 3, 1, 0, 0, 0.}, {2001, 4, 1, 0, 0, 0.},
    {2001, 5, 1, 0, 0, 0.}, {2001, 6, 1, 0, 0, 0.}} *)

For comparison, below is the original 5 time values. Note that {2002, 1, 1, ...} is missing, and a new date is added at the end.

DateList /@ temp["Times"][[1, ;; 5]]

(* {{2001, 1, 1, 0, 0, 0.}, {2001, 2, 1, 0, 0, 0.}, {2001, 3, 1, 0, 0, 0.},
    {2001, 4, 1, 0, 0, 0.}, {2001, 5, 1, 0, 0, 0.}} *)

EnsembleApply

It allows one to do various kinds of arithmetic on time series without having to muck around with custom functions to avoid changing the dates.

EnsembleApply[f,td] apply the function f to the state values of td.

EnsembleApply[f,td,lev] apply f to the states at level lev.

DateListPlot[TemporalData`EnsembleApply[#1 + #1^2 &, temp]["Paths"],
  Joined -> True]

EnsembleFold, EnsembleMap, EnsembleMapThread, EnsembleMovingMap and EnsembleTimeMap work similarly.

EnsembleFold[f,td] folds the function f over the state values in td.

EnsembleMap[f,td] map the function f over the state values of td.

EnsembleMap[f,td,lev] map f over the states at level lev.

EnsembleMapThread[f,td] resamples by "Union" and creates a single path with states {f[{s11,s21,...}],f[{s12,s22,...}],...}.

EnsembleMovingMap[f,td,r] computes running version of f over the states in td of order r.

EnsembleTimeMap[f,td] maps the function f over the time stamps in td.

ExtendTimes

This seems to work out what the $n$-th-next data point's time value would be, but it does it in AbsoluteTime space, which is not what you want for calendar data. As you can see, the last time period in my fakedata (or temp once converted to TemporalData form) is 1 April 2009, but three periods later is June 24, not July 1.

Map[DateList, (temp["Times"]), {2}][[1, -5 ;;]]
(* {{2008, 12, 1, 0, 0, 0.}, {2009, 1, 1, 0, 0, 0.}, {2009, 2, 1, 0, 0, 
  0.}, {2009, 3, 1, 0, 0, 0.}, {2009, 4, 1, 0, 0, 0.}} *)

DateList[TemporalData`ExtendTimes[temp, 3][[1, 1]]]
(* {2009, 6, 24, 0, 0, 0.} *)

To extend times according to calendar dates, one has to specify calendar dates for TemporalData (implicit or explicit):

td = TemporalData[Range@5, {{2001, 1, 1}, {2001, 1, 29}, {1, "Week"}}];
td["UnexpandedRawTimes"]

(* {TemporalData`DateSpecification[{2001, 1, 1, 0, 0, 0.},
                                   {2001, 1, 29, 0, 0, 0.}, {1, "Week"}]} *)

DateList /@ td["Times"][[1]]

(* {{2001, 1, 1, 0, 0, 0.}, {2001, 1, 8, 0, 0, 0.}, {2001, 1, 15, 0, 0, 0.},
    {2001, 1, 22, 0, 0, 0.}, {2001, 1, 29, 0, 0, 0.}} *)

TemporalData`ExtendTimes[td, 2]

(* {{{2001, 2, 12, 0, 0, 0.}}} (* two weeks are correctly added *)*)

Resample

Resamples data according to the bin width specification.

Resample[td,t,f] maps the function f over the state values and resamples so that the paths have equivalent time stamps specified by t where t can be "Union", "Intersection",a number, list of numbers, a date increment such as "Month" or a list {n,t} where n is a number and t is a date increment.

Resample[td,t] is equivalent to Resample[td,t,Identity]

Resample[td] is equivalent to Resample[td,"Union"]

Simulate a random walk for 200 steps and then resample it for bins of width 13:

td = Block[{i=0}, TemporalData[{#, i = i+RandomChoice@{-1, 1}} & /@ Range@200]];
new = TemporalData`Resample[td, 13];
new["Times"]

(* {{1, 14, 27, 40, 53, 66, 79, 92, 105, 118, 131, 144, 157, 170, 183, 196}} *)

{ListLinePlot@td, ListLinePlot@new}

With calendar dates, resample from 500 days to 2 months resolution:

td = TemporalData[{DatePlus[{2001, 1}, # - 1], #} & /@ Range@500];
DateList /@ TemporalData`Resample[td, {2, "Month"}]["Times"][[1]]

(* {{2001, 1, 1, 0, 0, 0.}, {2001, 3, 1, 0, 0, 0.}, {2001, 5, 1, 0, 0, 0.},
    {2001, 7, 1, 0, 0, 0.}, {2001, 9, 1, 0, 0, 0.}, {2001, 11, 1, 0, 0, 0.},
    {2002, 1, 1, 0, 0, 0.}, {2002, 3, 1, 0, 0, 0.}, {2002, 5, 1, 0, 0, 0.}} *)

RescaleTimes

RescaleTimes[td,{tmin,tmax}] rescales the paths to run from tmin to tmax.

If a single value is given instead of a pair, it is taken to be the new starting date, and the end date is shifted accordingly:

td = TemporalData[{#, #} & /@ Range@10];
td["Times"]

(* {{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}} *)

TemporalData`RescaleTimes[td, 21]["Times"]

(* {{21, 22, 23, 24, 25, 26, 27, 28, 29, 30}} *)

TemporalData`RescaleTimes[td, {101, 200}]["Times"]

(* {{101, 112, 123, 134, 145, 156, 167, 178, 189, 200}} *)

With calendar dates:

td = TemporalData[Range@4, {{2013, 10, 31}}];
DateList /@ First@td["Times"]

(* {{2013, 10, 31, 0, 0, 0.}, {2013, 11, 1, 0, 0, 0.},
    {2013, 11, 2, 0, 0, 0.}, {2013, 11, 3, 0, 0, 0.}} *)

new = TemporalData`RescaleTimes[td, {{1099, 12, 31}, {1110, 1, 5}}];
DateList /@ First@new["Times"]

(* {{1099, 12, 31, 0, 0, 0.}, {1103, 5, 4, 0, 0, 0.},
    {1106, 9, 4, 0, 0, 0.}, {1110, 1, 5, 0, 0, 0.}} *)

{DateListPlot@td@"Path", DateListPlot@new@"Path"}

ShiftTimes

ShiftTimes[td,dt] shifts the paths in td by dt units where dt can be a number t or a date increment.

This one works the same way as DatePlus. Simulate a random walk starting at today, and then shift timestaps by 700 days:

td = TemporalData[NestList[RandomChoice@{-1, 1} + # &, 0, 100],
         {NestList[DatePlus[#, {1, "Month"}] &, {2013, 10, 31, 0, 0, 0}, 100]}];
new = TemporalData`ShiftTimes[td, {700, "Day"}];

{DateListPlot@td@"Path", DateListPlot@new@"Path"}

TDListConvolve

TDListConvolve[ker,td] performs a convolution on the state values in td using the kernel ker.

td = TemporalData[NestList[RandomChoice@{-1, 1} + # &, 0, 100], {Range@101}];
new = TemporalData`TDListConvolve[{.2, .3, .5}, td];
{ListLinePlot@td, ListLinePlot@new}

TemporalDataInsert

TemporalDataInsert[td,{t,x},p] inserts element {t,x} into td at path p where p is specified as an integer, list of integers or All.

This adds the pair {3, 11} to the second path:

td = TemporalData[{{1, 1, 1, 1}, {2, 2, 2, 2}}, {{1, 2, 5, 10}}];
td["Paths"]

(* {{{1, 1}, {2, 1}, {5, 1}, {10, 1}}, {{1, 2}, {2, 2}, {5, 2}, {10, 2}}} *)

new = TemporalData`TemporalDataInsert[td, {3, 11}, 2];
new["Paths"]

(* {{{1, 1}, {2, 1}, {5, 1}, {10, 1}}, {{1, 2}, {2, 2}, {3, 11}, {5, 2}, {10, 2}}} *)

Works with calendar dates as well.

TemporalDataQ

TemporalDataQ[td] test whether td is TemporalData and structurally valid.

TemporalData`TemporalDataQ[temp]
(* True *)    
TemporalData`TemporalDataQ[fakedata]
(* False *)

TemporallyAlignedQ

Returns True when all paths have same starting and ending times, othewise returns False.

TemporalData`TemporallyAlignedQ[{temp, templater}]
(* False *)

TimeSeriesConcatenate

TimeSeriesConcatenate[td1, td2,...] concatentates that paths of the tdi.

concated = TemporalData`TimeSeriesConcatenate[temp, templater];
DateListPlot[concated["Paths"], Joined -> True]

UniformlySpacedQ

Again, as TemporalData converts DateList-format to seconds, UniformlySpacedQ might return False for data that is intuitively uniformly spaced, but not in the absolute sense, e.g. if time step size is given in "Month":

TemporalData`UniformlySpacedQ@
   TemporalData[{DatePlus[{2001, 1}, {#, "Day"}], #} & /@ Range@10]

(* True *)

TemporalData`UniformlySpacedQ@
   TemporalData[{DatePlus[{2001, 1}, {#, "Month"}], #} & /@ Range@10]

(* False *)

ValidTemporalDataQ

This does what you expect:

TemporalData`ValidTemporalDataQ[temp]
(* True *)
TemporalData`ValidTemporalDataQ[fakedata]
(* False *)