2
$\begingroup$

I would like to determine which pattern has the greatest correlation with each sublist.

I have the following data:

data = {{0.6043664420589688`, 0.9857728765270476`, 
0.5425821703751481`, 0.15504309122370524`, 
0.8276861413622156`}, {0.5425821703751481`, 0.15504309122370524`, 
0.8276861413622156`, 0.3117288541104692`, 
0.4230554403012381`}, {0.8276861413622156`, 0.3117288541104692`, 
0.4230554403012381`, 0.05368581732943603`, 
0.1285684468267231`}, {0.4230554403012381`, 0.05368581732943603`, 
0.1285684468267231`, 0.568487372004134`, 
0.9595968642236314`}, {0.1285684468267231`, 0.568487372004134`, 
0.9595968642236314`, 0.748598034746261`, 
0.5760672090376757`}, {0.9595968642236314`, 0.748598034746261`, 
0.5760672090376757`, 0.8226326906088341`, 
0.08882777194355806`}, {0.5760672090376757`, 0.8226326906088341`, 
0.08882777194355806`, 0.21758856941067833`, 
0.6896577869437517`}, {0.08882777194355806`, 0.21758856941067833`,
 0.6896577869437517`, 0.3515760303307617`, 
0.8666214369119132`}, {0.6896577869437517`, 0.3515760303307617`, 
0.8666214369119132`, 0.0015663158972453776`, 
0.6448673801479466`}, {0.8666214369119132`, 
0.0015663158972453776`, 0.6448673801479466`, 0.6775921340346936`, 
0.38830057970317`}, {0.6448673801479466`, 0.6775921340346936`, 
0.38830057970317`, 0.0875520713672823`, 
0.04050093808897781`}, {0.38830057970317`, 0.0875520713672823`, 
0.04050093808897781`, 0.6918192575076461`, 
0.8457184093280218`}, {0.04050093808897781`, 0.6918192575076461`, 
0.8457184093280218`, 0.9325089801435771`, 
0.21281479672676218`}, {0.8457184093280218`, 0.9325089801435771`, 
0.21281479672676218`, 0.3800904033971768`, 
0.42266296902678374`}, {0.21281479672676218`, 0.3800904033971768`,
 0.42266296902678374`, 0.878823162814141`, 
0.08424634990003906`}, {0.42266296902678374`, 0.878823162814141`, 
0.08424634990003906`, 0.8116030313930428`, 
0.46306610480315225`}, {0.08424634990003906`, 0.8116030313930428`,
 0.46306610480315225`, 0.1302251280678801`, 
0.5081791408623634`}, {0.46306610480315225`, 0.1302251280678801`, 
0.5081791408623634`, 0.9889703407842088`, 
0.3742383328595942`}, {0.5081791408623634`, 0.9889703407842088`, 
0.3742383328595942`, 0.9126365586572017`, 
0.8185213539186117`}, {0.3742383328595942`, 0.9126365586572017`, 
0.8185213539186117`, 0.6373943104534471`, 
0.507616895947681`}, {0.8185213539186117`, 0.6373943104534471`, 
0.507616895947681`, 0.9110702427599564`, 
0.17365397377066502`}, {0.507616895947681`, 0.9110702427599564`, 
0.17365397377066502`, 0.9598021764187534`, 
0.11931631624451103`}, {0.17365397377066502`, 0.9598021764187534`,
 0.11931631624451103`, 0.8235181713926741`, 
0.1331530356816872`}, {0.11931631624451103`, 0.8235181713926741`, 
0.1331530356816872`, 0.2679829189111074`, 0.2735979069164892`}};

I have a set of patterns which may fit certain sublists, found by taking the 3/4 and 1/4 quantiles at each position within each sublist, as follows:

aa = Quantile[data[[All, #]], 3/4] & /@ Table[i, {i, 1, 5}];
bb = Quantile[data[[All, #]], 1/4] & /@ Table[i, {i, 1, 5}];
patterns = Tuples[Transpose@{aa, bb}]

I would like to find which pattern (such as patterns[[1,All]]) has the greatest Absolute correlation (ie. correlation of -1 can be the greatest correlation) with each sublist in data.

I would like the output to follow the following format:

{{1,30}...{24,5}}

Where {1,30} represents sublist 1 having the highest correlation with pattern 30 (take into account that I did not check if this statement is true).

The data provided in this example is random as I am working with a much larger dataset and stackexchange will not allow me to post all of it. Thank you for your help!

EDIT: So far I have been able to calculate the correlations the following code found in a different answer on stackexchange:

N[Table[Map[Correlation[data[[i, All]], #] &, patterns], {i,Length[data]}]]

I can also find the Maximum points with:

datapatterncor =N[Table[Map[Correlation[data[[i, All]], #] &, patterns], {i, 
 Length[data]}]];

But I still cannot find a way to automate the process of finding the positions of the values with the greatest distance from zero in each sublist.

$\endgroup$
  • $\begingroup$ Have a look at ListCorrelate. $\endgroup$ – bill s Sep 10 '16 at 0:46
  • $\begingroup$ @bills ListCorrelate seems to be multiplying each value in each pattern by the value in the equivalent position in data. Am I doing something wrong? It isn't returning the correlation between two lists. $\endgroup$ – Alejandro Braun Sep 10 '16 at 1:42
  • $\begingroup$ Correction, it seems to be multiplying as stated before and then adding the values in each resulting sublist. $\endgroup$ – Alejandro Braun Sep 10 '16 at 1:52
  • $\begingroup$ Have you tried Correlation ? $\endgroup$ – Simon Woods Sep 10 '16 at 12:24
  • $\begingroup$ Your comment actually reminded me of a previous recommendation from a past question to map the correlation function over the data (shown in the edited portion of the question). Now I just need to find the position of the value with the greatest distance from zero each sublist. $\endgroup$ – Alejandro Braun Sep 10 '16 at 14:09
1
$\begingroup$

Does this give the result you need?

Transpose[{Range[24], Last@*Ordering /@ Outer[Abs@*Correlation, data, patterns, 1]}]

(* {{1, 3}, {2, 5}, {3, 17}, {4, 4}, {5, 16}, {6, 31}, {7, 7}, {8, 27}, 
   {9, 11}, {10, 23}, {11, 4}, {12, 4}, {13, 15}, {14, 8}, {15, 30},
   {16, 5}, {17, 19}, {18, 3}, {19, 21}, {20, 20}, {21, 14}, {22, 32},
   {23, 22}, {24, 24}} *)
|improve this answer|||||
$\endgroup$
  • $\begingroup$ This works too! Thank you! $\endgroup$ – Alejandro Braun Sep 12 '16 at 21:16
4
$\begingroup$

I apologize if I have misunderstood the aim. In the following are two approaches. The first uses absolute correlation as distance function to generate a distance matrix and the major diagonal is removed. Thereafter, the position of element with maximal correlation for each list. The second approach uses Nearest and -absolute correlation as distance to identify 'closes'list.

dm = DistanceMatrix[data, 
    DistanceFunction -> (Abs@Correlation[#1, #2] &)] - 
   IdentityMatrix[24];
mx = Max /@ dm;
un = MapThread[{#1, Position[#2, #3][[1, 1]]} -> #3 &, {Range[24], dm,
     mx}];
dmr = ReplacePart[dm, Thread[Keys[un] -> 10]];
Legended[MatrixPlot[dmr, ColorFunction -> "Rainbow", 
  ColorRules -> {10 -> Black, 0. -> White}], BarLegend["Rainbow"]]
nf[u_] := {u, 
  Position[data, 
    Nearest[data, data[[u]], 2, 
      DistanceFunction -> (-Abs@Correlation[#1, #2] &)][[-1]]][[1, 1]]}
nfv = nf /@ Range[24];
Keys[un] == nfv

enter image description here

|improve this answer|||||
$\endgroup$
  • $\begingroup$ This is really neat! I will take a look at it after breakfast to see if I can breakdown how the code works and I'll let you know if it is returning the solutions I am aiming for! $\endgroup$ – Alejandro Braun Sep 10 '16 at 14:11
  • $\begingroup$ @AlejandroBraun I may have misunderstood. It is after midnight in my timezone so I am off to bed.:) $\endgroup$ – ubpdqn Sep 10 '16 at 14:16
  • $\begingroup$ It seems like what I was looking for, I will update if it isn't. Thank you for your help! $\endgroup$ – Alejandro Braun Sep 10 '16 at 14:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.