0
$\begingroup$

I have two sets of data points,

pts1 = {{14, 66}, {33.03, 76.97}, {26.03, 55}, {14.08, 42}, {29.05, 
    49.97}, {40.94, 39.97}, {33.92, 19.05}, {20.92, 14.89}, {21, 
    21}, {51.03, 46.97}, {68.03, 84.22}, {64.89, 83.06}, {60.97, 
    29.97}, {79.97, 14.86}, {102.94, 22.08}};

pts2 = {{14, 66}, {33, 77}, {26, 55}, {14, 42}, {29, 50}, {41, 
    40}, {34, 19}, {21, 15}, {21, 21}, {51, 47}, {68, 84}, {65, 
    83}, {61, 30}, {80, 15}, {103, 22}};

How can I use Mathematica to calculate the standard deviation between the two sets of the data points?

$\endgroup$
  • $\begingroup$ Can you clarify "standard deviation between the two sets of the data points"? Do you mean the standard deviation of the difference? $\endgroup$ – dr.blochwave Jun 4 '15 at 7:00
  • $\begingroup$ yes, the standard deviation of the difference between the two sets of data points. $\endgroup$ – Keith Lim Jun 4 '15 at 7:01
  • $\begingroup$ In which case, perhaps the information here might help you get started? How to compute standard deviation of difference between two data sets? $\endgroup$ – dr.blochwave Jun 4 '15 at 7:06
  • 1
    $\begingroup$ You have a link to the formula for this, right? Why not look it up and try to implement it in Mathematica? $\endgroup$ – J. M. will be back soon Jun 4 '15 at 7:07
4
$\begingroup$

The second set is just the Round-ed version of the first set, which means there is a strong correlation between both sets (correlation close to 1).

A naive and rough (and incorrect) approach would be to say that the variances of both sets will be very close as the differences caused by rounding are relatively small compared to the values involved.

The variance of the difference of two correlated random variables with correlation coefficient $\rho$ is given by:

$ Var(X-Y) = Var(X) + Var(Y) - 2 \rho*SD(X)* SD (Y) $

which, with $Var(X) \approx Var(Y)$ and $\rho=1$, equals

$Var(X) + Var(X) - 2*1*Var(X) = 0 $

To be exact, and in Mathematica code:

Variance@pts1[[All, 1]] + Variance@pts2[[All, 1]] - 
 2 Correlation[pts1[[All, 1]], pts2[[All, 1]]] StandardDeviation@
   pts1[[All, 1]]*StandardDeviation@pts2[[All, 1]]
(* 0.00312381 *)

However, setting both variances equal is actually unacceptable in this case and another approach would be to study the individual differences exactly. If it is just a rounding process then X-Round(X) should be distributed as a UniformDistribution[{-1/2, 1/2}] (unless the decimal digits have some kind of bias, which ideally should not be the case), so the standard deviation will be given by:

StandardDeviation[UniformDistribution[{-1/2, 1/2}]]

1/(2 Sqrt[3])

which is about 0.29.

This is what you would predict. In the concrete case of the given data sets it would be:

StandardDeviation /@ Transpose[pts1 - pts2]

{0.0558911, 0.0832781}

for the first and second coordinates values, respectively.

This deviates considerable from our prediction. The reason is that the fractional part of the numbers is not uniformely distributed as assumed:

DistributionFitTest[#, UniformDistribution[{-1/2, 1/2}]] & /@  Transpose[pts1 - pts2]

{0.0000134701, 0.000166425}

Very low p-values, so it is unlikely the decimals came from a uniform distribution. If these are measured values, perhaps you should start worrying about your data acquisition process.

$\endgroup$
  • $\begingroup$ To be exact, the two set of data points are coordinates in 2D. pts2 seems to be a rounded version of pts1 but actually its not. pts1 comes from my defuzzification model while pts2 is the crisp model. The closer between pts1 and pts2 the better it is. Standard deviation I mean is to know how much pts1 deviate from the original pts2, the lesser, the better. $\endgroup$ – Keith Lim Jun 5 '15 at 14:51
  • $\begingroup$ In that case the line StandardDeviation /@ Transpose[pts1 - pts2] in the middle of the post should do the trick for you. $\endgroup$ – Sjoerd C. de Vries Jun 5 '15 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.