# Access pairs of vectors, rather than the matrices, in partitioned data structure at a levelspec

I have some old code to analyze correlation between 2 time series by looking at windows of different lengths shifting across the time series data. The code:

• gets 2 sets of time series data (in this case using FinancialData[] for convenience);
• sets a list of windows on which to calculate the correlations;
• partitions the data into matrices by applying each window over time;
• calculates correlations;
• plots

Code:

data = Transpose[Drop[FinancialData[#, "Aug. 4, 2006"] & /@ {"SPY", "GLD"}, None, None, 1][[All, All, 1]]];
windows = Range[100, Round[Length[data], 100], 100];
partitionedData = Partition[data, #, 1] & /@ windows;
correlations = Map[Correlation, partitionedData, {2}];
plotData = PadLeft[correlations[[All, All, 1, 2]]];
ListPlot3D[
plotData,
ImageSize -> 400,
AxesLabel -> {"Days", "Window", "Correlation"},
PlotRange -> {Automatic, Automatic, {-1, 1}},
BoxRatios -> {1, 1, 0.75},
ViewPoint -> {2, 0, 1}]


Output

Now typically I'd run these correlations on log returns rather than price, but I wanted to keep the example as simple as possible.

In this case the windows have lengths of 100, 200, 300, ... 1500 corresponding to the 15 windows in the plot.

This all works fine, but now I want to look at other correlation measures:

SpearManRankCorrelation[]
KendallRankCorrelation[]


Neither of these take a matrix as input, they both take 2 vectors:

So my problem comes down to this single line of code:

correlations = Map[Correlation, partitionedData, {2}];


I need to alter this line of code so that it will map to the pairs of vectors in each of the matrices at level 2 of partitionedData.

As Correlation[] can also use vectors as inputs this will give me a more generalized approach.

The structure of partitionedData looks like this:

Dimensions[partitionedData]
{15}

Dimensions[#] & /@ partitionedData
{{1410, 100, 2}, {1310, 200, 2}, {1210, 300, 2}, {1110, 400, 2}, {1010, 500, 2}, {910, 600, 2}, {810, 700, 2}, {710, 800, 2}, {610, 900, 2}, {510, 1000, 2}, {410, 1100, 2}, {310, 1200, 2}, {210, 1300, 2}, {110, 1400, 2}, {10, 1500, 2}}


I've fiddled around with this all morning, but keep confusing myself. I don't see a clear way to use the same form of Map[] using levelspec, so I keep trying variations on pure functions but keep getting into a muddle.

I've tried lots of variations of this basic approach:

Correlation[#[[1]], #[[2]]] & /@ partitionedData[[All, 2]];


but I don't get myself anywhere.

Thoughts and guidance appreciated.

-

You appear to have an inefficiency in your algorithm. You generate a symmetrical correlation matrix and only use one element (correlations[[All, All, 1, 2]]). You also partition everything before processing which takes a lot of memory:

ByteCount[partitionedData]


505899832

I suggest reformulating your code to produce only the significant correlation and use PartitionMap to create these correlations as the data is partitioned, as the correlations take much less space than the partitioned data that is used to create them.

In the code below Correlation is used in the two vector argument format, therefore your other correlation functions may be substituted directly.

windows = Range[100, Round[Length[data], 100], 100];

correlations =
DeveloperPartitionMap[Correlation @@ (#\[Transpose]) &, data, #, 1] & /@ windows;

ListPlot3D[plotData,
ImageSize -> 400,
AxesLabel -> {"Days", "Window", "Correlation"},
PlotRange -> {Automatic, Automatic, {-1, 1}},
BoxRatios -> {1, 1, 0.75},
ViewPoint -> {2, 0, 1}
]


The small amount of memory used by correlations:

ByteCount[correlations]


256952

-
Clearer, more concise, much faster and very much appreciated. – Jagra Aug 3 '12 at 14:31
Yep, I do know and will ;-) I wanted to study your code first to better understand what you did in it. I never had occasion to know about the "Developer Utilities" package. Nice. Also can you discuss other uses for that very interesting construction "#[Transpose]" which when I pasted it into a notebook shows up as "#"superscript T"? Does the documentation have anything on it's more general application like use with other functions? – Jagra Aug 3 '12 at 15:30
@Jagra that notation is described in the More Information section of Transpose documentation, and also briefly under \[Transpose]. It has low precedence (binding power) so () were necessary here. – Mr.Wizard Aug 3 '12 at 15:36

If I understand you correctly, is the following what you want?

correlations2 = Map[
Outer[Correlation, #, #, 1] &[Transpose[#]] &,
partitionedData, {2}];


Now the Correlation receives two vector as its input, and you can replace it with SpearmanRankCorrelation or KendallRankCorrelation legally.

To verify it, compare correlations2 with your original correlations:

Union[Flatten[Chop[correlations2 - correlations]]] == {0}


True

-
Thanks for the quick solution. It works flawlessly. – Jagra Jul 31 '12 at 20:02
@Jagra Thank you :) – Silvia Jul 31 '12 at 20:26
I believe you can replace Correlation[#1, #2] & with Correlation`. – Mr.Wizard Aug 2 '12 at 22:49
@Mr.Wizard oh yes of course! Guess my mind trapped in a pure-function pattern at that moment LOL – Silvia Aug 3 '12 at 8:30