2
$\begingroup$

It is possible to do simple math between TemporalSeries objects. For example

es=EventSeries[{{{2016, 1, 1}, 2}, {{2016, 1, 3}, 2.1}}];
td=TemporalData[{{{2016, 1, 1}, 2}, {{2016, 1, 3}, 3.1}}];
es*td (* works fine *)

This returns a TemporalData object with the path {{3660595200, 4}, {3660768000, 6.51}}.

This method does not work however if the objects being manipulated have differing dimensions.

es=EventSeries[{{{2016, 1, 1}, 2}, {{2012, 1, 3}, 1.9}, {{2016, 1, 3}, 2.1}}];
td=TemporalData[{{{2016, 1, 1}, 2}, {{2016, 1, 3}, 3.1}}];
es*td (* ka-boom *)

With the data I'm using, one series or the other may be missing data at various time stamps, and it would be acceptable to either interpolate for those missing points or to simply omit them from the computation and the output.

I've been dealing with this problem in several steps,

  1. Make a list of the dates that all the series have in common with intersectionDates=Apply[Intersection, Map[#["Path"][[All, 1]] &, listOfTD]
  2. Extract the points for only those dates with alignedData=Map[Select[#, MemberQ[intersectiondates, #[[1]]] &]&, listOfTD]
  3. And finally turn the output back into TemporalData with Map[TimeSeries[#] &, alignedData]

In practice I have additional code to remove duplicate dates, if any exist, to convert EventSeries back to EventSeries, and some other bells and whistles, not shown above.

The problem is, this method is unusably slow for large datasets. With a few hundred points, it works fine, but with 5000+, it becomes intolerable.

I welcome any suggestions for approaches that would allow me to complete simple math between TemporalData objects of differing dimensions much faster.

$\endgroup$

4 Answers 4

5
$\begingroup$

One of the regular tasks in statistical arbitrage is to compute correlations between a large universe of stocks, such as the S&P500 index members, for example. Mathematica/WL has some very nice features for obtaining financial data and manipulating time series. And of course it offers all the commonly required statistical functions, including correlation. But the WL Correlation function is missing one vital feature - the ability to handle data series of unequal length. This arises, of course, because stock data series do not all share a common start date and (very occasionally) omit data for dates in the middle of the series. This creates an issue for the Correlation function, which can only handle series of equal length. The usual way of handling this is to apply pairwise correlation, in which each pair of data vectors is truncated to include only the dates common to both series. Of course this can easily be done in WL; but it is very inefficient.

Let's take an example. We start with the last 10 symbols in the S&P 500 index membership:

    In[1]:= tickers = Take[FinancialData["^GSPC", "Members"], -10]

Out[1]= {"NASDAQ:WYNN", "NASDAQ:XEL", "NYSE:XRX", "NASDAQ:XLNX", \
"NYSE:XYL", "NYSE:YUM", "NASDAQ:ZBRA", "NYSE:ZBH", "NASDAQ:ZION", \
"NYSE:ZTS"}

Next we obtain the returns series for these stocks, over the last 3 years. By default, FinancialData retrieves the data as TimeSeries Objects. This is very elegant, but slows the processing of the data, as we shall see.

tsStocks = 
 FinancialData[tickers, "Return", 
  DatePlus[Today, {-2753, "BusinessDay"}]];

Not all the series contain the same number of date-return pairs. So using Correlation is out of the question:

In[282]:= Table[Length@tsStocks[[i]]["Values"], {i, 10}]

Out[282]= {2762, 2762, 2762, 2762, 2388, 2762, 2762, 2762, 2762, 2060}

Since Correlation doesn't offer a pairwise option, we have to create the required functionality in WL. Let's start with:

PairsCorrelation[ts_] := Module[{td, correl},
   If[ts[[1]]["PathLength"] == ts[[2]]["PathLength"], 
    correl = Correlation @@ ts,
    td = TimeSeriesResample[ts, "Intersection"];
    correl = Correlation @@ td[[All, All, 2]]]];

We first check to see if the two arguments are of equal length, in which case we can Apply the Correlation function directly. If not, we use the "Intersection" option of the TSResample function to reduce the series to a set of common observation dates. The function is designed to be deployed using parallelization, as follows:

PairsListCorrelation[tslist_] := Module[{pairs, i, td, c, correl = {}},
  pairs = Subsets[Range[Length@tslist], {2}];
  correl = 
   ParallelTable[
    PairsCorrelation[tslist[[pairs[[i]]]]], {i, 1, Length@pairs}];
  {correl, pairs}]

The Subsets function is used to generate a non-duplicative list of index pairs and then a correlation table is built in parallel using PairsCorrelation function on each pair of series.

When we apply the function to the ten stock time series, we get the following results:

In[263]:= AbsoluteTiming[{correl, pairs} = 
   PairsListCorrelation[tsStocks];]

Out[263]= {13.4791, Null}

In[270]:= Length@correl

Out[270]= 45

In[284]:= Through[{Mean, Median, Min, Max}[correl]]

Out[284]= {0.381958, 0.396429, 0.200828, 0.536383}

So far, so good. But look again at the timing of the PairsListCorrelation function. It takes 13.5 seconds to calculate the 45 correlation coefficients for 10 series. To carry out an equivalent exercise for the entire S&P 500 universe would entail computing 124,750 coefficients, taking approximately 10.5 hours! This is far too slow to be practically useful in the given context.

Some speed improvement is achievable by retrieving the stock returns data in legacy (i.e. list rather than time series) format, but it still takes around 10 seconds to calculate the coefficients for our 10 stocks. Perhaps further speed improvements are possible through other means (e.g. compilation), but what is really required is a core language function to handle series of unequal length (or a Pairwise method for the Correlation function).

For comparison, I can produce the correlation coefficients for all 500 S&P member stocks in under 3 seconds using the 'Rows', 'pairwise' options of the equivalent correlation function in another scientific computing language.

$\endgroup$
1
  • $\begingroup$ The correlation computation can be sped up considerably by storing stock returns in a list of associations. See my answer below. $\endgroup$
    – sakra
    Apr 12, 2021 at 18:02
3
$\begingroup$

For some applications, an alternative to using TimeSeries expressions is using associations which support both a fast KeyIntersection and KeyUnion operation.

The example by Jonathan Kinlay from another answer to this question can be rewritten using associations in the following way:

First retrieve market data as a list of associations by using the "Legacy" method:

tickers = Take[FinancialData["^GSPC", "Members"], -10];
tickerData = FinancialData[tickers, "Return", 
    DatePlus[Today, {-2753, "BusinessDay"}], Method -> "Legacy"];
assocStocks = Apply[Rule, tickerData, {2}] // Map[Association];

Then compute the correlations for unique pairs of stocks with KeyIntersection:

pairs = Subsets[Range@Length@assocStocks, {2}];
correl = Map[Correlation @@ Values@KeyIntersection[assocStocks[[#]]] &, pairs];

This reduces the required time to compute the correlations to a fraction of a second on my machine.

$\endgroup$
1
  • $\begingroup$ Excellent solution! $\endgroup$ Apr 13, 2021 at 6:58
1
$\begingroup$

Given two TimeSeries (ts1 and ts2), I was able to speed up my results about 60x with the following:

  1. paths=Map[#["Path"] &, {ts1,ts2}];
  2. commonDates=Intersection[paths[[1, All, 1]], paths[[2, All, 1]]];
  3. fakeDatepath=Transpose[{commonDates, Table[-1, {Length[commonDates]}]}];
  4. shortPaths=Map[(TemporalData[Intersection[#, fakeDatepath, SameTest -> (#1[[1]] == #2[[1]] &)]]) &, paths];
  5. Map[TemporalData[#]&, shortPaths]

My production code works with an arbitrary number of TimeSeries simultaneously, using a generalized form of this approach.

$\endgroup$
2
  • 1
    $\begingroup$ paths = Through[{ts1, ts2}["Path"]]; is slightly neater. $\endgroup$ Jul 7, 2016 at 23:39
  • $\begingroup$ @J.M. OK, but any suggestions on making it faster? Each "intersection" still takes several minutes. $\endgroup$ Jul 8, 2016 at 10:44
1
$\begingroup$

If you want to interpolate then use TimeSeriesThread to combine TimeSeries (by default TimeSeries uses linear interpolation):

In[96]:= ts1 = TimeSeries[{{{2016, 1, 1}, 2}, {{2012, 1, 3}, 1.9}, {{2016, 1, 3}, 
     2.1}}];
ts2 = TimeSeries[{{{2016, 1, 1}, 2}, {{2016, 1, 3}, 3.1}}];

In[100]:= res = TimeSeriesThread[Times @@ # &, {ts1, ts2}];

During evaluation of In[100]:= InterpolatingFunction::dmval: Input value {3534537600} lies outside the range of data in the interpolating function. Extrapolation will be used.

In[101]:= Normal[res]

Out[101]= {{DateObject[{2012, 1, 3}], -1520.86}, {DateObject[{2016, 1,
     1}], 4.}, {DateObject[{2016, 1, 3}], 6.51}}

If you use TemporalData, then default is the interpolation with order 0.

If you use EventSeries, no interpolation is applied and Missing[] is created:

    In[87]:= 
es1 = EventSeries[{{{2016, 1, 1}, 2}, {{2012, 1, 3}, 1.9}, {{2016, 1, 3}, 
         2.1}}];
es2 = EventSeries[{{{2016, 1, 1}, 2}, {{2016, 1, 3}, 3.1}}];

In[94]:= res = TimeSeriesThread[Times @@ # &, {es1, es2}];

In[95]:= Normal[res]

Out[95]= {{DateObject[{2012, 1, 3}], 
  1.9 Missing[]}, {DateObject[{2016, 1, 1}], 
  4}, {DateObject[{2016, 1, 3}], 6.51}}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.