Access pairs of vectors, rather than the matrices, in partitioned data structure at a levelspec

Question

I have some old code to analyze correlation between 2 time series by looking at windows of different lengths shifting across the time series data. The code:

• gets 2 sets of time series data (in this case using FinancialData[] for convenience);
• sets a list of windows on which to calculate the correlations;
• partitions the data into matrices by applying each window over time;
• calculates correlations;
• pads the data; and
• plots

Code:

data = Transpose[Drop[FinancialData[#, "Aug. 4, 2006"] & /@ {"SPY", "GLD"}, None, None, 1][[All, All, 1]]];
windows = Range[100, Round[Length[data], 100], 100];
partitionedData = Partition[data, #, 1] & /@ windows;
correlations = Map[Correlation, partitionedData, {2}];
plotData = PadLeft[correlations[[All, All, 1, 2]]];
ListPlot3D[
 plotData,
 ImageSize -> 400,
 AxesLabel -> {"Days", "Window", "Correlation"},
 PlotRange -> {Automatic, Automatic, {-1, 1}},
 BoxRatios -> {1, 1, 0.75},
 ViewPoint -> {2, 0, 1}]

Output

Now typically I'd run these correlations on log returns rather than price, but I wanted to keep the example as simple as possible.

In this case the windows have lengths of 100, 200, 300, ... 1500 corresponding to the 15 windows in the plot.

This all works fine, but now I want to look at other correlation measures:

SpearManRankCorrelation[]
KendallRankCorrelation[]

Neither of these take a matrix as input, they both take 2 vectors:

So my problem comes down to this single line of code:

correlations = Map[Correlation, partitionedData, {2}];

I need to alter this line of code so that it will map to the pairs of vectors in each of the matrices at level 2 of partitionedData.

As Correlation[] can also use vectors as inputs this will give me a more generalized approach.

Some additional background...

The structure of partitionedData looks like this:

Dimensions[partitionedData]
{15}

Dimensions[#] & /@ partitionedData
{{1410, 100, 2}, {1310, 200, 2}, {1210, 300, 2}, {1110, 400, 2}, {1010, 500, 2}, {910, 600, 2}, {810, 700, 2}, {710, 800, 2}, {610, 900, 2}, {510, 1000, 2}, {410, 1100, 2}, {310, 1200, 2}, {210, 1300, 2}, {110, 1400, 2}, {10, 1500, 2}}

I've fiddled around with this all morning, but keep confusing myself. I don't see a clear way to use the same form of Map[] using levelspec, so I keep trying variations on pure functions but keep getting into a muddle.

I've tried lots of variations of this basic approach:

Correlation[#[[1]], #[[2]]] & /@ partitionedData[[All, 2]];

but I don't get myself anywhere.

Thoughts and guidance appreciated.

Answer 1

You appear to have an inefficiency in your algorithm. You generate a symmetrical correlation matrix and only use one element (correlations[[All, All, 1, 2]]). You also partition everything before processing which takes a lot of memory:

ByteCount[partitionedData]

505899832

I suggest reformulating your code to produce only the significant correlation and use PartitionMap to create these correlations as the data is partitioned, as the correlations take much less space than the partitioned data that is used to create them.

In the code below Correlation is used in the two vector argument format, therefore your other correlation functions may be substituted directly.

windows = Range[100, Round[Length[data], 100], 100];

correlations = 
  Developer`PartitionMap[Correlation @@ (#\[Transpose]) &, data, #, 1] & /@ windows;

plotData = PadLeft[correlations];

ListPlot3D[plotData,
 ImageSize -> 400, 
 AxesLabel -> {"Days", "Window", "Correlation"}, 
 PlotRange -> {Automatic, Automatic, {-1, 1}}, 
 BoxRatios -> {1, 1, 0.75},
 ViewPoint -> {2, 0, 1}
]

The small amount of memory used by correlations:

ByteCount[correlations]

256952

Answer 2

If I understand you correctly, is the following what you want?

correlations2 = Map[
   Outer[Correlation, #, #, 1] &[Transpose[#]] &,
   partitionedData, {2}];

Now the Correlation receives two vector as its input, and you can replace it with SpearmanRankCorrelation or KendallRankCorrelation legally.

To verify it, compare correlations2 with your original correlations:

Union[Flatten[Chop[correlations2 - correlations]]] == {0}

True