# Uncorrected Versus Corrected Measures of StandardDeviation in PredictorMeasurements' "StandardDeviation"

When using Predict have noticed that the standard deviation of the residuals computed manually is not equal to the output from PredictorMeasurements[predictor,testset,"StandardDeviation"], and that the relationship between the two outputs is $\sqrt{\frac{n-1}{n}}$; the factor which relates an the uncorrected estimate of the standard deviation to a the corrected one. I was taught that the uncorrected version is more biased than the corrected version, although both are not completely unbiased (See wikipedia). Of course, this is not a big deal when learning on large sets of data, as this factor is asymptotically equal to one. Is there a reason why Mathematica uses the uncorrected version in the Predict package?

Code to reproduce this is below:

data = {{1, 0, 1, 0, 0, 0., 0, 0.} ->
0., {1, 0, 2, 0, 0,
7737.568001411986, -44, -0.08999999999999986} ->
0.1, {1, 0, 3, 0, 0, 33415.08588019777,
73, -0.11999999999999988} -> -0.24, {1, 0, 4, 0, 0,
30097.577460046155, 186, -0.010000000000000009} -> -0.34, {1,
0, 5, 0, 0, 31134.576744789716,
55, -0.29000000000000004} -> -0.41, {1, 0, 6, 0, 0,
36282.2124880825, 44, -0.33999999999999986} -> -0.56, {1, 0,
7, 0, 0, 37865.342476767975,
40, -0.3699999999999999} -> -0.75, {1, 0, 0, 1, 0,
20998.538503301235, 132, -0.3599999999999999} -> -0.94, {1, 0,
1, 1, 0, 11193.4298923167, -135,
0.20999999999999996} -> -0.38, {1, 0, 2, 1, 0,
14.054760156282418, -262, -0.06000000000000005} ->
0.03, {1, 0, 3, 1, 0,
5431.369223994962, -190, -0.6200000000000001} ->
0.09, {1, 0, 4, 1, 0,
12304.262919022143, -193, -0.3599999999999999} -> -0.06, {1,
0, 5, 1, 0,
18897.84256887328, -182, -0.6600000000000001} -> -0.31, {1, 0,
6, 1, 0,
16173.385156583805, -86, -0.7399999999999998} -> -0.62, {1, 0,
7, 1, 0, 5477.184071455756,
92, -0.6600000000000001} -> -0.93, {1, 0, 0, 2, 0,
243.7633660095809, 184, -0.3899999999999999} -> -1.18, {1, 0,
1, 2, 0, 10422.835510030174, -153, 0.24} -> -0.14, {1, 0, 2,
2, 0, 13.010723450765466, -314, 0.040000000000000036} ->
0.25, {1, 0, 3, 2, 0,
4950.114728299646, -391, -0.8199999999999998} ->
0.35, {1, 0, 4, 2, 0,
10065.359304805765, -307, -0.3599999999999999} ->
0.2, {1, 0, 5, 2, 0,
16722.50558676344, -320, -0.6600000000000001} -> -0.04, {1, 0,
6, 2, 0,
14787.278172121176, -202, -0.6600000000000001} -> -0.37, {1,
0, 7, 2, 0,
4614.924306404395, -96, -0.7399999999999998} -> -0.72, {1, 0,
0, 2, 0, 279.6333051769322, 100, -1.} -> -1.05, {2, 0, 1, 0,
0, 8104.458036890476, 44, 0.08999999999999986} -> -0.49, {2,
0, 2, 0, 0, 0., 0, 0.} ->
0., {2, 0, 3, 0, 0, 6145.525506385167,
117, -0.030000000000000027} ->
0.3, {2, 0, 4, 0, 0, 6221.250864883095, 230,
0.07999999999999985} ->
0.41, {2, 0, 5, 0, 0, 6470.596480554151,
99, -0.20000000000000018} ->
0.36, {2, 0, 6, 0, 0, 9522.573513820591, 88, -0.25} ->
0.15, {2, 0, 7, 0, 0, 10521.76548433661,
84, -0.28} -> -0.12, {2, 0, 0, 1, 0, 3926.3955501863484,
176, -0.27} -> -0.54, {2, 0, 1, 1, 0, 39844.33791120307, -91,
0.2999999999999998} -> -1.08, {2, 0, 2, 1, 0,
7115.147461376305, -218, 0.029999999999999805} -> -0.41, {2,
0, 3, 1, 0, 1874.3121073918378, -146, -0.5300000000000002} ->
0.1, {2, 0, 4, 1, 0, 48.418012701984985, -149, -0.27} ->
0.36, {2, 0, 5, 1, 0,
406.51424189891, -138, -0.5700000000000003} ->
0.39, {2, 0, 6, 1, 0,
110.1144682381671, -42, -0.6499999999999999} ->
0.13, {2, 0, 7, 1, 0, 328.5518770771627,
136, -0.5700000000000003} -> -0.28, {2, 0, 0, 2, 0,
3959.5439585105346, 228, -0.30000000000000004} -> -0.75, {2,
0, 1, 2, 0, 44879.00727307302, -109,
0.32999999999999985} -> -1., {2, 0, 2, 2, 0,
11002.73223367998, -270, 0.1299999999999999} -> -0.23, {2, 0,
3, 2, 0, 2978.9176889788305, -347, -0.73} ->
0.31, {2, 0, 4, 2, 0, 595.9036944939947, -263, -0.27} ->
0.62, {2, 0, 5, 2, 0,
25.196469494367616, -276, -0.5700000000000003} ->
0.68, {2, 0, 6, 2, 0,
14.19991960061888, -158, -0.5700000000000003} ->
0.51, {2, 0, 7, 2, 0,
920.7768070374773, -52, -0.6499999999999999} ->
0.09, {2, 0, 0, 2, 0, 3664.638021662793,
144, -0.9100000000000001} -> -0.39, {3, 0, 1, 0, 0,
22462.07903234322, -73, 0.11999999999999988} -> -0.72, {3, 0,
2, 0, 0, 3674.021300271403, -117,
0.030000000000000027} -> -0.15};
sizes2 = {10, 15, 20, 30};
numSamps2 = 3;
For[i = 1, i <= Length[sizes2], i++,
n = sizes2[[i]];
Print["n=", n];
samples2 = Table[RandomSample[data, n], {j, 1, numSamps2}];
Print["samples taken, sample:"];
Print[samples2[[1, 1 ;; 5]]];
For[j = 1, j <= Length[samples2], j++,
network = Predict[samples2[[j]], Method -> "NearestNeighbors"];
Print[{sizes2[[i]],
PredictorMeasurements[network, samples2[[j, ;;]],
"StandardDeviation"], (Sqrt[(n - 1)/n])*
StandardDeviation@
Table[network[samples2[[j, i, 1]]] - samples2[[j, i, 2]], {i,
1, Length[samples2[[j]]]}]}]

];
];

• I guess this is because the Sqrt [n-1] correction is applied on samples from a population. In the PredictorMeasurement case there is no sample only a population. The standard deviation of the residuals is just that. It's not a sample from a larger, unknown set of residuals. Sep 23 '15 at 19:55
• I would agree in the case of summarizing the standard deviation from a training set, in which case the entire population is defined and fixed. However, in general, when using a function like PredictorMeasurements, the user is interested in the generalization error, which we estimate for the larger population using a smaller validation or test set. Therefore, in the case of greatest interest, the correction should be applied. There are also names for the uncorrected quantity, like root-mean-square deviation. Sep 23 '15 at 20:17