1
$\begingroup$

When using Predict have noticed that the standard deviation of the residuals computed manually is not equal to the output from PredictorMeasurements[predictor,testset,"StandardDeviation"], and that the relationship between the two outputs is $\sqrt{\frac{n-1}{n}}$; the factor which relates an the uncorrected estimate of the standard deviation to a the corrected one. I was taught that the uncorrected version is more biased than the corrected version, although both are not completely unbiased (See wikipedia). Of course, this is not a big deal when learning on large sets of data, as this factor is asymptotically equal to one. Is there a reason why Mathematica uses the uncorrected version in the Predict package?

Code to reproduce this is below:

data = {{1, 0, 1, 0, 0, 0.`, 0, 0.`} -> 
    0.`, {1, 0, 2, 0, 0, 
     7737.568001411986`, -44, -0.08999999999999986`} -> 
    0.1`, {1, 0, 3, 0, 0, 33415.08588019777`, 
     73, -0.11999999999999988`} -> -0.24`, {1, 0, 4, 0, 0, 
     30097.577460046155`, 186, -0.010000000000000009`} -> -0.34`, {1, 
     0, 5, 0, 0, 31134.576744789716`, 
     55, -0.29000000000000004`} -> -0.41`, {1, 0, 6, 0, 0, 
     36282.2124880825`, 44, -0.33999999999999986`} -> -0.56`, {1, 0, 
     7, 0, 0, 37865.342476767975`, 
     40, -0.3699999999999999`} -> -0.75`, {1, 0, 0, 1, 0, 
     20998.538503301235`, 132, -0.3599999999999999`} -> -0.94`, {1, 0,
      1, 1, 0, 11193.4298923167`, -135, 
     0.20999999999999996`} -> -0.38`, {1, 0, 2, 1, 0, 
     14.054760156282418`, -262, -0.06000000000000005`} -> 
    0.03`, {1, 0, 3, 1, 0, 
     5431.369223994962`, -190, -0.6200000000000001`} -> 
    0.09`, {1, 0, 4, 1, 0, 
     12304.262919022143`, -193, -0.3599999999999999`} -> -0.06`, {1, 
     0, 5, 1, 0, 
     18897.84256887328`, -182, -0.6600000000000001`} -> -0.31`, {1, 0,
      6, 1, 0, 
     16173.385156583805`, -86, -0.7399999999999998`} -> -0.62`, {1, 0,
      7, 1, 0, 5477.184071455756`, 
     92, -0.6600000000000001`} -> -0.93`, {1, 0, 0, 2, 0, 
     243.7633660095809`, 184, -0.3899999999999999`} -> -1.18`, {1, 0, 
     1, 2, 0, 10422.835510030174`, -153, 0.24`} -> -0.14`, {1, 0, 2, 
     2, 0, 13.010723450765466`, -314, 0.040000000000000036`} -> 
    0.25`, {1, 0, 3, 2, 0, 
     4950.114728299646`, -391, -0.8199999999999998`} -> 
    0.35`, {1, 0, 4, 2, 0, 
     10065.359304805765`, -307, -0.3599999999999999`} -> 
    0.2`, {1, 0, 5, 2, 0, 
     16722.50558676344`, -320, -0.6600000000000001`} -> -0.04`, {1, 0,
      6, 2, 0, 
     14787.278172121176`, -202, -0.6600000000000001`} -> -0.37`, {1, 
     0, 7, 2, 0, 
     4614.924306404395`, -96, -0.7399999999999998`} -> -0.72`, {1, 0, 
     0, 2, 0, 279.6333051769322`, 100, -1.`} -> -1.05`, {2, 0, 1, 0, 
     0, 8104.458036890476`, 44, 0.08999999999999986`} -> -0.49`, {2, 
     0, 2, 0, 0, 0.`, 0, 0.`} -> 
    0.`, {2, 0, 3, 0, 0, 6145.525506385167`, 
     117, -0.030000000000000027`} -> 
    0.3`, {2, 0, 4, 0, 0, 6221.250864883095`, 230, 
     0.07999999999999985`} -> 
    0.41`, {2, 0, 5, 0, 0, 6470.596480554151`, 
     99, -0.20000000000000018`} -> 
    0.36`, {2, 0, 6, 0, 0, 9522.573513820591`, 88, -0.25`} -> 
    0.15`, {2, 0, 7, 0, 0, 10521.76548433661`, 
     84, -0.28`} -> -0.12`, {2, 0, 0, 1, 0, 3926.3955501863484`, 
     176, -0.27`} -> -0.54`, {2, 0, 1, 1, 0, 39844.33791120307`, -91, 
     0.2999999999999998`} -> -1.08`, {2, 0, 2, 1, 0, 
     7115.147461376305`, -218, 0.029999999999999805`} -> -0.41`, {2, 
     0, 3, 1, 0, 1874.3121073918378`, -146, -0.5300000000000002`} -> 
    0.1`, {2, 0, 4, 1, 0, 48.418012701984985`, -149, -0.27`} -> 
    0.36`, {2, 0, 5, 1, 0, 
     406.51424189891`, -138, -0.5700000000000003`} -> 
    0.39`, {2, 0, 6, 1, 0, 
     110.1144682381671`, -42, -0.6499999999999999`} -> 
    0.13`, {2, 0, 7, 1, 0, 328.5518770771627`, 
     136, -0.5700000000000003`} -> -0.28`, {2, 0, 0, 2, 0, 
     3959.5439585105346`, 228, -0.30000000000000004`} -> -0.75`, {2, 
     0, 1, 2, 0, 44879.00727307302`, -109, 
     0.32999999999999985`} -> -1.`, {2, 0, 2, 2, 0, 
     11002.73223367998`, -270, 0.1299999999999999`} -> -0.23`, {2, 0, 
     3, 2, 0, 2978.9176889788305`, -347, -0.73`} -> 
    0.31`, {2, 0, 4, 2, 0, 595.9036944939947`, -263, -0.27`} -> 
    0.62`, {2, 0, 5, 2, 0, 
     25.196469494367616`, -276, -0.5700000000000003`} -> 
    0.68`, {2, 0, 6, 2, 0, 
     14.19991960061888`, -158, -0.5700000000000003`} -> 
    0.51`, {2, 0, 7, 2, 0, 
     920.7768070374773`, -52, -0.6499999999999999`} -> 
    0.09`, {2, 0, 0, 2, 0, 3664.638021662793`, 
     144, -0.9100000000000001`} -> -0.39`, {3, 0, 1, 0, 0, 
     22462.07903234322`, -73, 0.11999999999999988`} -> -0.72`, {3, 0, 
     2, 0, 0, 3674.021300271403`, -117, 
     0.030000000000000027`} -> -0.15`};
sizes2 = {10, 15, 20, 30};
numSamps2 = 3;
For[i = 1, i <= Length[sizes2], i++,
  n = sizes2[[i]];
  Print["n=", n];
  samples2 = Table[RandomSample[data, n], {j, 1, numSamps2}];
  Print["samples taken, sample:"];
  Print[samples2[[1, 1 ;; 5]]];
  For[j = 1, j <= Length[samples2], j++,
   network = Predict[samples2[[j]], Method -> "NearestNeighbors"];
   Print[{sizes2[[i]], 
     PredictorMeasurements[network, samples2[[j, ;;]], 
      "StandardDeviation"], (Sqrt[(n - 1)/n])*
      StandardDeviation@
       Table[network[samples2[[j, i, 1]]] - samples2[[j, i, 2]], {i, 
         1, Length[samples2[[j]]]}]}]

   ];
  ];
$\endgroup$
  • 1
    $\begingroup$ I guess this is because the Sqrt [n-1] correction is applied on samples from a population. In the PredictorMeasurement case there is no sample only a population. The standard deviation of the residuals is just that. It's not a sample from a larger, unknown set of residuals. $\endgroup$ – Sjoerd C. de Vries Sep 23 '15 at 19:55
  • $\begingroup$ I would agree in the case of summarizing the standard deviation from a training set, in which case the entire population is defined and fixed. However, in general, when using a function like PredictorMeasurements, the user is interested in the generalization error, which we estimate for the larger population using a smaller validation or test set. Therefore, in the case of greatest interest, the correction should be applied. There are also names for the uncorrected quantity, like root-mean-square deviation. $\endgroup$ – mikeagibson Sep 23 '15 at 20:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.