0
$\begingroup$

Motivation

I have the Mean and StandardDeviation of several lists of equal size but I don't know the elements of those lists.

I want to find what the Mean and StandardDeviation would be if I combined those lists together.

The Mean of the combined lists is trivial, but I am finding the StandardDeviation more complicated...

I am not very confident in the use of Solve especially with statistical functions and so could use some help

Problem

Let r be a list of real numbers of length m, where m is an even number. (r is the combined list from above)

r is partitioned into x sublists of equal length where x is an even number. (Call them r1, r2,...,rx. The length of each ri is n).

How can I generate a function,f, that generates the analytic solution that finds the StandardDeviation of r, given only:

  • the Mean[r] (call it u)

  • the Mean of each sublist,Mean[r1],Mean[r2]...(Call itu1,u2...)

  • and the StandardDeviation of each sublist, StandardDeviation[r1],StandardDeviation[r2]...(Call it s1, s1...)

ie. I want to find a function g[x_] that generates f

Example

eg. we can solve this by hand for x=2 which gives

f[n_,u_,{s1_, s2_}, {u1_, u2_}] := 
 Sqrt[(((n - 1)*(s1^2 + s2^2)) + n*(u1^2 + u2^2) - 2*n*u^2)/(2*n - 1)]

Check formula on an example

SeedRandom[1];
m = 100;
n = m/2;
r = RandomReal[{-1, 1}, m];
r1 = r[[;; n]];
r2 = r[[n + 1 ;;]];

u = Mean[r];
u1 = Mean[r1];
u2 = Mean[r2];

s1 = StandardDeviation[r1];
s2 = StandardDeviation[r2];

StandardDeviation[r] == f[n, u, {u1, u2}, {s1, s2}];

True

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ It turns out this can be solved by "The exact pooled variance is the mean of the variances plus the variance of the means of the component data sets." from math.stackexchange.com/a/4567292/1221434. The original paper is "Calculating the Exact Pooled Variance J. W. Rudmin, Physics Dept., James Madison University" arxiv.org/ftp/arxiv/papers/1007/1007.1012.pdf $\endgroup$
    – IntroductionToProbability
    Commented Sep 16, 2023 at 15:43
  • $\begingroup$ I am still interested in seeing if we can use Mathematica to solve this problem $\endgroup$
    – IntroductionToProbability
    Commented Sep 16, 2023 at 15:44
  • $\begingroup$ The STD is the square root of the variance. The variance of a sum x+y is var(x+y)= var(x)+var(y)+2 cov[x,y], where cov is the covariance. If x and y are independent, the covariance is zero. $\endgroup$
    – Daniel Huber
    Commented Sep 16, 2023 at 16:48
  • 1
    $\begingroup$ The general formula is not restricted to even numbers of groups. m can be even or odd. $\endgroup$
    – JimB
    Commented Sep 16, 2023 at 17:41
  • 2
    $\begingroup$ Note that the referenced paper uses a definition of variance where one divides by the sample size rather than the sample size minus 1. That's not necessarily wrong but Mathematica divides by the sample size minus 1. Which do you want? Also that referenced paper uses the term "pooled" in a non-standard way. $\endgroup$
    – JimB
    Commented Sep 16, 2023 at 18:14

1 Answer 1

Reset to default
4
$\begingroup$

Here is one way to do so. Note that this is a data manipulation exercise and doesn't imply any ability to make inferences with the resulting standard deviation.

m = 5; (* Number of groups *)
n = 7;  (* Sample size per group *)
(* Random sample of size n from m normal distributions *) 
r = RandomVariate[NormalDistribution[0, 1], {m, n}];
sd = StandardDeviation[#] & /@ r
(* {1.12549, 0.827586, 0.762233, 0.973844, 0.84024} *)
mean = Mean[#] & /@ r
(* {-0.145108, -0.0999453, -0.788001, 0.43927, -0.303919} *)

(* Overall mean by formula *)
overallMean = Total[mean]/m
(* -0.179541 *)
(* Overall mean from raw data *)
Mean[Flatten[r]]
(* -0.179541 *)

(* Overall standard deviation by formula *)
Sqrt[(Total[sd^2 (n - 1) + n mean^2] - n m (Total[mean]/m)^2)/(n m - 1)]
(* Total sums of squares minus total number of observations*square of 
   overall mean divided by total sample size minus 1 *)
(* 0.947985 *)
(* Standard deviation from raw data *)
StandardDeviation[Flatten[r]]
(* 0.947985 *)

Putting all of this into a single function:

g[n_, mean_, sd_] := {Mean[mean], 
  Sqrt[(Total[sd^2 (n - 1) + n mean^2] - n Length[mean] Mean[mean]^2)/(n Length[mean] - 1)]}

g[n, mean, sd]
(* {-0.179541, 0.947985} *)
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.