3
$\begingroup$

I have two vectors $A$ and $B$ of length (say) $50$ or so, and I want to determine whether there is any correlation between their entries. I computed their correlation directly and found it to be positive, but I also wanted to compute a $p$-value to confirm the implication that the entries are not completely independent.

However, when I asked Mathematica to compute the following:

PearsonCorrelationTest[A, B, "TestDataTable"]

It gave an answer, but it also gave the following warning:

PearsonCorrelationTest::nortst : "At least one of the p-values in {0.508..., 0} resulting from a test for normality, is below 0.025. The tests in {"PearsonCorrelation"} require that the data is normally distributed.

I couldn't find any further documentation on what Mathematica was doing. What assumptions on the vectors $A$ and $B$ is required for PearsonCorrelationTest to give a sensible answer?

| improve this question | |
$\endgroup$
  • $\begingroup$ It sounds like the data isn't normally distributed. Is that so? You might need to use something like MannWhitneyTest if that's the case. $\endgroup$ – Jonathan Shock May 13 '13 at 5:31
  • 1
    $\begingroup$ Take a look at this post. It might help. $\endgroup$ – Rod May 13 '13 at 5:49
  • $\begingroup$ @RodLm, thanks for that, that's useful. $\endgroup$ – Jonathan Shock May 13 '13 at 5:52
  • $\begingroup$ Pearson's correlation does assume Normality, while Spearman's correlation is a rank based correlation measure and does not assume Normality. $\endgroup$ – Rod May 13 '13 at 5:54
  • 1
    $\begingroup$ Or even SpearmanRankTest[], why not? $\endgroup$ – Rod May 13 '13 at 6:06
3
$\begingroup$

Pearson's correlation does assume Normality, while Spearman's correlation is a rank based correlation measure and does not assume Normality.

So, instead of using PearsonCorrelationTest[A, B, "TestDataTable"] you should use 

SpearmanRankTest[A, B, "TestDataTable"]

EDITED

Pearson's correlation does not assume Normality, however the sampling distribution for Pearson's correlation does assume Normality.

Maybe that's why you're getting this error message from Mathematica.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Citing from the highest ranked answer in the link you posted above in comment yourself: "It does not assume normality although it does assume finite variances and finite covariance". It looks like the jury is still out on this. $\endgroup$ – Sjoerd C. de Vries May 13 '13 at 11:55
  • $\begingroup$ These assumptions must also be true for the Pearson correlation test... $\endgroup$ – Rod May 13 '13 at 13:20
  • $\begingroup$ I feel you're missing the part "does not assume normality" which is the opposite of what you write above. $\endgroup$ – Sjoerd C. de Vries May 13 '13 at 13:43
  • $\begingroup$ OK, let me correct myself: the sampling distribution for Pearson's correlation does assume Normality, while the measure itself does not assume Normality. That's why on should use Spearman correlation instead of Pearson. $\endgroup$ – Rod May 13 '13 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.