# Fast way to subtract the mean from a dataset [duplicate]

I have a dataset, say:

data =
{{10, 8.04}, {8, 6.95}, {13, 7.58}, {9, 8.81}, {11, 8.33}, {14, 9.96},
{6, 7.24}, {4, 4.26}, {12, 10.84}, {7, 4.82}, {5, 5.68}}


and I want to subtract the mean from it, producing:

correcteddata =
{{1, 0.539091}, {-1, -0.550909}, {4, 0.0790909}, {0, 1.30909}, {2, 0.829091},
{5, 2.45909}, {-3, -0.260909}, {-5, -3.24091}, {3, 3.33909}, {-2, -2.68091},
{-4, -1.82091}}


However, I'm looking to do this on a much larger data-set as efficiently as possible.

I've gone through several attempts:

This was very slow, but simple:

correcteddata = # - Mean[data] & /@ data


This was significantly faster, but involves adding another variable:

m = Mean[data];
correcteddata = # - m & /@ data


The fastest/best I've found so far is this:

correcteddata = Transpose[# - Mean /@ # &[Transpose[data]]]


But I'm wondering if there is something better.

• I think the notation of the question is a bit confusing. You do not have a dataset ... you have sample bivariate $(X_i,Y_i)$ data. And rather than the stated desire to 'subtract the mean from it, you wish to subtract the sample means from the data. – wolfies Jul 2 '17 at 8:26

Do this:

Standardize[data, Mean, 1 &]


Here are the timings for comparison, with f1 and f2 as defined in Mr.Wizard's answer:

big = RandomReal[{0, 50}, {1*^7, 2}];

f1[big] // RepeatedTiming // First
f2[big] // RepeatedTiming // First
Standardize[big, Mean, 1 &] // RepeatedTiming // First


0.274

0.232

0.12

• What version and platform are you using? In 10.1 under Windows this is much slower than either f1 or f2 in my post at 0.85 seconds on the same test. – Mr.Wizard Jul 1 '17 at 15:49
• Strange - on my Mac with MMA 11.0.1 my answer is twice as fast as yours... – Jens Jul 1 '17 at 15:51
• On the same big data? If so I guess this function received a needed upgrade. – Mr.Wizard Jul 1 '17 at 15:53
• Yes, I added the timings with your method. – Jens Jul 1 '17 at 15:54

This is a modest improvement on your code:

f2[data_] := (data\[Transpose] - Mean[data])\[Transpose]


With your code as a function f1 for reference:

f1[data_] := Transpose[# - Mean /@ # &[Transpose[data]]]


And now with Simon Woods's f3 calling the internal function used by Standardize in v11:

f3[data_] :=
Module[{a = data}, StatisticsLibraryMatrixRowTranslate[a, -Mean[a]]; a]


Timings:

big = RandomReal[{0, 50}, {1*^7, 2}];

f1[big] // RepeatedTiming // First
f2[big] // RepeatedTiming // First
f3[big] // RepeatedTiming // First

0.193

0.15

0.0936

f1[big] === f2[big] === f3[big]

True


In Mathematica 10.1 under Windows x64 Standardize performs rather poorly. I get:

Standardize[big, Mean, 1 &] // RepeatedTiming // First

0.852


I am curious to know if this is version or platform dependent, or perhaps both.

• In V10.4.1 and V9.0, Standardize took 1.6 sec and in V11.1.1 it took 0.14 sec. – Michael E2 Jul 1 '17 at 16:19
• Try f3[data_] := Block[{a = data}, StatisticsLibraryMatrixRowTranslate[a, -Mean[data]]; a] - this is what Standardize does in 11.1.1 – Simon Woods Jul 1 '17 at 21:28
• @Simon Thanks, that gives the performance one would have anticipated for a built-in! (I used Module as I feel that is appropriate.) – Mr.Wizard Jul 2 '17 at 6:16
• @Simon beat me to it. I figured that out, which led to this answer, but haven't had time to come back here. – Michael E2 Jul 3 '17 at 0:34
• @Michael would you please use that to answer mathematica.stackexchange.com/q/103527/121 ? – Mr.Wizard Jul 3 '17 at 8:27