13
$\begingroup$

I have a dataset, say:

data = 
  {{10, 8.04}, {8, 6.95}, {13, 7.58}, {9, 8.81}, {11, 8.33}, {14, 9.96}, 
   {6, 7.24}, {4, 4.26}, {12, 10.84}, {7, 4.82}, {5, 5.68}}

and I want to subtract the mean from it, producing:

correcteddata = 
  {{1, 0.539091}, {-1, -0.550909}, {4, 0.0790909}, {0, 1.30909}, {2, 0.829091}, 
   {5, 2.45909}, {-3, -0.260909}, {-5, -3.24091}, {3, 3.33909}, {-2, -2.68091}, 
   {-4, -1.82091}}

However, I'm looking to do this on a much larger data-set as efficiently as possible.

I've gone through several attempts:

This was very slow, but simple:

correcteddata = # - Mean[data] & /@ data

This was significantly faster, but involves adding another variable:

m = Mean[data];
correcteddata = # - m & /@ data

The fastest/best I've found so far is this:

correcteddata = Transpose[# - Mean /@ # &[Transpose[data]]]

But I'm wondering if there is something better.

$\endgroup$
  • 2
    $\begingroup$ I think the notation of the question is a bit confusing. You do not have a dataset ... you have sample bivariate $(X_i,Y_i)$ data. And rather than the stated desire to 'subtract the mean from it, you wish to subtract the sample means from the data. $\endgroup$ – wolfies Jul 2 '17 at 8:26
14
$\begingroup$

Do this:

Standardize[data, Mean, 1 &]

Here are the timings for comparison, with f1 and f2 as defined in Mr.Wizard's answer:

big = RandomReal[{0, 50}, {1*^7, 2}];

f1[big] // RepeatedTiming // First
f2[big] // RepeatedTiming // First
Standardize[big, Mean, 1 &] // RepeatedTiming // First

0.274

0.232

0.12

$\endgroup$
  • 1
    $\begingroup$ What version and platform are you using? In 10.1 under Windows this is much slower than either f1 or f2 in my post at 0.85 seconds on the same test. $\endgroup$ – Mr.Wizard Jul 1 '17 at 15:49
  • 1
    $\begingroup$ Strange - on my Mac with MMA 11.0.1 my answer is twice as fast as yours... $\endgroup$ – Jens Jul 1 '17 at 15:51
  • $\begingroup$ On the same big data? If so I guess this function received a needed upgrade. $\endgroup$ – Mr.Wizard Jul 1 '17 at 15:53
  • 2
    $\begingroup$ Yes, I added the timings with your method. $\endgroup$ – Jens Jul 1 '17 at 15:54
11
$\begingroup$

This is a modest improvement on your code:

f2[data_] := (data\[Transpose] - Mean[data])\[Transpose]

With your code as a function f1 for reference:

f1[data_] := Transpose[# - Mean /@ # &[Transpose[data]]]

And now with Simon Woods's f3 calling the internal function used by Standardize in v11:

f3[data_] := 
  Module[{a = data}, Statistics`Library`MatrixRowTranslate[a, -Mean[a]]; a]

Timings:

big = RandomReal[{0, 50}, {1*^7, 2}];

f1[big] // RepeatedTiming // First
f2[big] // RepeatedTiming // First
f3[big] // RepeatedTiming // First
0.193

0.15

0.0936
f1[big] === f2[big] === f3[big]
True

In Mathematica 10.1 under Windows x64 Standardize performs rather poorly. I get:

Standardize[big, Mean, 1 &] // RepeatedTiming // First
0.852

I am curious to know if this is version or platform dependent, or perhaps both.

$\endgroup$
  • 4
    $\begingroup$ In V10.4.1 and V9.0, Standardize took 1.6 sec and in V11.1.1 it took 0.14 sec. $\endgroup$ – Michael E2 Jul 1 '17 at 16:19
  • 3
    $\begingroup$ Try f3[data_] := Block[{a = data}, Statistics`Library`MatrixRowTranslate[a, -Mean[data]]; a] - this is what Standardize does in 11.1.1 $\endgroup$ – Simon Woods Jul 1 '17 at 21:28
  • $\begingroup$ @Simon Thanks, that gives the performance one would have anticipated for a built-in! (I used Module as I feel that is appropriate.) $\endgroup$ – Mr.Wizard Jul 2 '17 at 6:16
  • $\begingroup$ @Simon beat me to it. I figured that out, which led to this answer, but haven't had time to come back here. $\endgroup$ – Michael E2 Jul 3 '17 at 0:34
  • $\begingroup$ @Michael would you please use that to answer mathematica.stackexchange.com/q/103527/121 ? $\endgroup$ – Mr.Wizard Jul 3 '17 at 8:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.