Biased results when using Method -> "RandomForest" in Predict

Question

I am interested in using random forests for some prediction, so I have been using Predict. I find that "RandomForest" method tends to create biased fits of data sets, as demonstrated by predicted vs. actual plots where the slope of the data cloud is not equal to unity; this is unusual for a random forest so I investigated a bit further. An example is below for a small data set:

url1 = "https://www.dropbox.com/s/u5g997u337ysuv1/";
url2 = "exampleRegressionDataForML.m?dl=1";
allData = Import[url1 <> url2]
data = RandomSample[allData];
training = data[[;; Floor[Length[allData]*0.7]]];
validation = data[[Floor[Length[allData]*0.7] + 1 ;;]];
Length[training];
Length[validation];
pTrain = 
  Predict[training, 
    PerformanceGoal -> "Quality", 
    Method -> "RandomForest"];
pMeasuresTrain = PredictorMeasurements[pTrain, training];
pMeasuresTrain["ComparisonPlot"]

which gives a plot that looks like:

I get the same type of biased fitting for several different data sets. This biased result is not sensitive to the number of trees, number of data points used to determine terminal leaves, or the performance goal (LeafSize, TreenNumber, and PerformanceGoal, respectively). These seem to be all of the levers we have in Predict to modify the random forest.

I can run a random forest on the same data set in R and get good results, as shown below (using the randomForest, which wraps Breiman and Cutler's RF package). These un-biased fits in R are robust to perturbation of the additional variables controllable in R such as mtry, etc:

Can anyone explain the source of this bias and how I might go about fixing it? It is difficult to diagnose because Predict is somewhat of a black box.

Etienne Bernard or Taliesin Beynon might be interested in this post

Answer 1

Can anyone explain the source of this bias and how I might go about fixing it?

The source of the bias is the low number of features used to grow the trees (mtry in R). Both R and Mathematica currently use Max[1, Floor[number of features/3]] as the heuristic for choosing this number (Mathematica might change this in the future). As your data has 5 features, this is an mtry = 1. You can change this in Mathematica using

    Predict[trainingData, Method -> {"RandomForest", "VariableSampleSize" -> 3}];

Plotting this now gives:

I can run a random forest on the same data set in R and get good results ... These un-biased fits in R are robust to perturbation of the additional variables controllable in R such as mtry

I think you are a victim of deceptive plots. Lets use RLink to generate the R plot with the default mtry value (mtry = 1):

    RSet["input", training[[;;, 1]]];
    RSet["output", training[[;;, 2]]];
    REvaluate["{
     library(randomForest)
     p = randomForest(input, output)
    }"];
    predictedRM = First@REvaluate["predict(p, input)"];

Plotting predictR versus training[[;;, 1]], I get:

Now combine the plot above with the Mathematica plot for "VariableSampleSize" -> Automatic: Red dots are from R, blue from Mathematica.

They give virtually identical results! (although one wouldn't expect them to be exactly the same, given the randomness inherent in the method).

The R plot you gave is probably generated with mtry = 3:

Combine the R mtry = 1 (blue) and mtry = 3 plots:

This demonstrates that the result is indeed very sensitive to mtry (or equivalently the Mathematica option "VariableSampleSize").