20
$\begingroup$

I am interested in using random forests for some prediction, so I have been using Predict. I find that "RandomForest" method tends to create biased fits of data sets, as demonstrated by predicted vs. actual plots where the slope of the data cloud is not equal to unity; this is unusual for a random forest so I investigated a bit further. An example is below for a small data set:

url1 = "https://www.dropbox.com/s/u5g997u337ysuv1/";
url2 = "exampleRegressionDataForML.m?dl=1";
allData = Import[url1 <> url2]
data = RandomSample[allData];
training = data[[;; Floor[Length[allData]*0.7]]];
validation = data[[Floor[Length[allData]*0.7] + 1 ;;]];
Length[training];
Length[validation];
pTrain = 
  Predict[training, 
    PerformanceGoal -> "Quality", 
    Method -> "RandomForest"];
pMeasuresTrain = PredictorMeasurements[pTrain, training];
pMeasuresTrain["ComparisonPlot"]

which gives a plot that looks like:

Comparison of predicted and actual values from random forest fitting in Mathematica

I get the same type of biased fitting for several different data sets. This biased result is not sensitive to the number of trees, number of data points used to determine terminal leaves, or the performance goal (LeafSize, TreenNumber, and PerformanceGoal, respectively). These seem to be all of the levers we have in Predict to modify the random forest.

I can run a random forest on the same data set in R and get good results, as shown below (using the randomForest, which wraps Breiman and Cutler's RF package). These un-biased fits in R are robust to perturbation of the additional variables controllable in R such as mtry, etc:

A random forest fit to the data in R shows no bias

Can anyone explain the source of this bias and how I might go about fixing it? It is difficult to diagnose because Predict is somewhat of a black box.

Etienne Bernard or Taliesin Beynon might be interested in this post

$\endgroup$
26
$\begingroup$

Can anyone explain the source of this bias and how I might go about fixing it?

The source of the bias is the low number of features used to grow the trees (mtry in R). Both R and Mathematica currently use Max[1, Floor[number of features/3]] as the heuristic for choosing this number (Mathematica might change this in the future). As your data has 5 features, this is an mtry = 1. You can change this in Mathematica using

    Predict[trainingData, Method -> {"RandomForest", "VariableSampleSize" -> 3}];

Plotting this now gives:

enter image description here

I can run a random forest on the same data set in R and get good results ... These un-biased fits in R are robust to perturbation of the additional variables controllable in R such as mtry

I think you are a victim of deceptive plots. Lets use RLink to generate the R plot with the default mtry value (mtry = 1):

    RSet["input", training[[;;, 1]]];
    RSet["output", training[[;;, 2]]];
    REvaluate["{
     library(randomForest)
     p = randomForest(input, output)
    }"];
    predictedRM = First@REvaluate["predict(p, input)"];

Plotting predictR versus training[[;;, 1]], I get:

The predictor performance using R package randomForest Now combine the plot above with the Mathematica plot for "VariableSampleSize" -> Automatic: Red dots are from R, blue from Mathematica.

They give virtually identical results! (although one wouldn't expect them to be exactly the same, given the randomness inherent in the method).

The R plot you gave is probably generated with mtry = 3: enter image description here

Combine the R mtry = 1 (blue) and mtry = 3 plots: enter image description here

This demonstrates that the result is indeed very sensitive to mtry (or equivalently the Mathematica option "VariableSampleSize").

$\endgroup$
  • $\begingroup$ Thank you, Sebastian! I guess I mist have absentmindedly set mtry = 3 in R without realizing it. $\endgroup$ – mikeagibson Mar 11 '15 at 0:02
  • 1
    $\begingroup$ Could you please tell me how you found out about the "VariableSampleSize" -> 3 option? It must be a silly question but…? $\endgroup$ – chris May 9 '15 at 14:38
  • 1
    $\begingroup$ @chris among other ways, the simplest is PredictorInformation[p, "Options"] which gives the full form of the method used, so it is quite useful. $\endgroup$ – rcollyer Sep 8 '15 at 13:15
  • $\begingroup$ @Sebastian, it appeasrs that "VariableSampleSize" is no longer a valid option? $\endgroup$ – SumNeuron Aug 29 '16 at 17:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.