I am interested in using random forests for some prediction, so I have been using
Predict. I find that
"RandomForest" method tends to create biased fits of data sets, as demonstrated by predicted vs. actual plots where the slope of the data cloud is not equal to unity; this is unusual for a random forest so I investigated a bit further. An example is below for a small data set:
url1 = "https://www.dropbox.com/s/u5g997u337ysuv1/"; url2 = "exampleRegressionDataForML.m?dl=1"; allData = Import[url1 <> url2] data = RandomSample[allData]; training = data[[;; Floor[Length[allData]*0.7]]]; validation = data[[Floor[Length[allData]*0.7] + 1 ;;]]; Length[training]; Length[validation]; pTrain = Predict[training, PerformanceGoal -> "Quality", Method -> "RandomForest"]; pMeasuresTrain = PredictorMeasurements[pTrain, training]; pMeasuresTrain["ComparisonPlot"]
which gives a plot that looks like:
I get the same type of biased fitting for several different data sets. This biased result is not sensitive to the number of trees, number of data points used to determine terminal leaves, or the performance goal (
PerformanceGoal, respectively). These seem to be all of the levers we have in
Predict to modify the random forest.
I can run a random forest on the same data set in R and get good results, as shown below (using the randomForest, which wraps Breiman and Cutler's RF package). These un-biased fits in R are robust to perturbation of the additional variables controllable in R such as mtry, etc:
Can anyone explain the source of this bias and how I might go about fixing it? It is difficult to diagnose because
Predict is somewhat of a black box.