Predict: How to customize model construction? (e.g. cross fold validation, mtry, number features used, ROC curve)

Question

Question Overview

In general this question is in regards to how to tweak the parameters used when one specifies a Method when calling the Predict function. Herein the examples focus more on Random Forests - although the question is valid for the other Method options of Predict. The specific questions follow the the example for better clarify.

Backgound

Mathematica's Predict function is pretty nifty. It quickly makes a model depending on the "Method" given as an option. The parameters that underly that, however, to me are obscure.

Example

Consider one wants to construct a Random Forest, on their Dataset, data, to predict a column called "output." Then the code for implementing this would be:

rf=Predict[data->"output", Method->"RandomForest"];

To get information about the model:

PredictorInformation[rf];

And we see something like:

Predictor information

Method | Random Forest

Number of features | Dimensions[data][[2]]-1

Number of training examples | Dimensions[data][[1]]

Number of tress | 50

Note

The method Random Forest, as stated in the Mathematica documentaiton, follows the Breiman-Cutler ensembles of decision trees. Click for Breiman-Cutler Documentation , or Click for Mathematica's Predict Documentation

Question 1

How can we modify this model? e.g. what if I want 100 trees? How can I use mtry? What if I wanted to use only 85 features?

Question 2

What are the underlying methods of this model? e.g. Does it make use of cross fold validation? If not, can we average 10 models together? Does it automatically scale the data?

Question 3

ROC curve?

So in general, knowing that Method -> "RandomForest" uses Breiman-Cutler ensembles, how can we tweak Predict?

Extension

Likewise, how could one do this for the other methods of Predict?

Answer 1

Question 3

ROC curve?

So in general, knowing that Method -> "RandomForest" uses Breiman-Cutler ensembles, how can we tweak Predict?

For basic ROC usage see this blog post "Basic example of using ROC with Linear regression".

For ROC for classifier ensembles (using Classify not Predict) see the blog post "ROC for classifier ensembles, bootstrapping, damaging, and interpolation".

Here is an example of ROC plot for and ensemble of classifiers over the Adult dataset:

References

Here are the packages used for ROC and classifier ensembles in the blog posts above:

[1] Anton Antonov, Receiver operating characteristic functions Mathematica package, (2016), source code at MathematicaForPrediction at GitHub, package ROCFunctions.m.

[2] Anton Antonov, Classifier ensembles functions Mathematica package, (2016), source code at MathematicaForPrediction at GitHub, package ClassifierEnsembles.m.

Answer 2

Answer on question 1

You can find possible options via Options[p] where p = Predict[data->output,"Method"->method]

Method->{"RandomForest", "TreeNumber" -> 100, "LeafSize" -> 10, "VariableSampleSize" -> 10}

Method->{"NeuralNetwork", "HiddenLayers"-> {{4, "RectifiedLinear"}, {3, "Tanh"}}}

(* Possible layers are: {"LogisticSigmoid", "RectifiedLinear", "Tanh", "SoftRectifiedLinear", "Linear"} *)

Method -> {"NearestNeighbors", "NeighborsNumber" -> 6}

Method -> {"LinearRegression", "L1Regularization" -> 0., "L2Regularization" -> 0.}

If you want to use only 85 features, you can select only them:

p = Predict[data[[;;,features]]->output]

Answer on question 2

There is function KFoldSplit in the package MachineLearning. So probably 'yes' - Mathematica does cross-validation.

?MachineLearning`PackageScope`*

KFoldSplit[data, nsplit] outputs nsplit pairs {trainingset, validationset} according to the K-Fold cross validation procedure.

KFoldSplit[data, nsplit, i] outputs the i-th split.

data can be X or {X, Y} where X is the list of feature vectors, and Y is the response vector.