# Computing Correlations and p-values

I have two vectors $A$ and $B$ of length (say) $50$ or so, and I want to determine whether there is any correlation between their entries. I computed their correlation directly and found it to be positive, but I also wanted to compute a $p$-value to confirm the implication that the entries are not completely independent.

However, when I asked Mathematica to compute the following:

PearsonCorrelationTest[A, B, "TestDataTable"]


It gave an answer, but it also gave the following warning:

PearsonCorrelationTest::nortst : "At least one of the p-values in {0.508..., 0} resulting from a test for normality, is below 0.025. The tests in {"PearsonCorrelation"} require that the data is normally distributed.

I couldn't find any further documentation on what Mathematica was doing. What assumptions on the vectors $A$ and $B$ is required for PearsonCorrelationTest to give a sensible answer?

• It sounds like the data isn't normally distributed. Is that so? You might need to use something like MannWhitneyTest if that's the case. – Jonathan Shock May 13 '13 at 5:31
• Take a look at this post. It might help. – Rod May 13 '13 at 5:49
• @RodLm, thanks for that, that's useful. – Jonathan Shock May 13 '13 at 5:52
• Pearson's correlation does assume Normality, while Spearman's correlation is a rank based correlation measure and does not assume Normality. – Rod May 13 '13 at 5:54
• Or even SpearmanRankTest[], why not? – Rod May 13 '13 at 6:06

Pearson's correlation does assume Normality, while Spearman's correlation is a rank based correlation measure and does not assume Normality.

So, instead of using PearsonCorrelationTest[A, B, "TestDataTable"] you should use

SpearmanRankTest[A, B, "TestDataTable"]


# EDITED

Pearson's correlation does not assume Normality, however the sampling distribution for Pearson's correlation does assume Normality.

Maybe that's why you're getting this error message from Mathematica.

• Citing from the highest ranked answer in the link you posted above in comment yourself: "It does not assume normality although it does assume finite variances and finite covariance". It looks like the jury is still out on this. – Sjoerd C. de Vries May 13 '13 at 11:55
• These assumptions must also be true for the Pearson correlation test... – Rod May 13 '13 at 13:20
• I feel you're missing the part "does not assume normality" which is the opposite of what you write above. – Sjoerd C. de Vries May 13 '13 at 13:43
• OK, let me correct myself: the sampling distribution for Pearson's correlation does assume Normality, while the measure itself does not assume Normality. That's why on should use Spearman correlation instead of Pearson. – Rod May 13 '13 at 16:20