How to do $ n $-fold cross validation with Classify?

Question

Other machine learning libraries have utilities to generate indices that can be used to generate dataset splits according to different cross-validation strategies. How do I tell Classify to optimize regularization parameters with a cross-validation strategy?

Answer 1

If you look closely at the documentation you will see that classify and predict both perform Cross validation of some description - unfortunately it's not clear from the help what type or degree is being done. I think it's reasonable to assume it's probably a low order n fold validation but there is no way to be sure it's meeting your needs without a query to Wolfram tech support.

See these questions for a bit more detail - particularly the comments from Etienne and Taliesin, note the override option that's mentioned :

How to know the internal algorithms of functions like Predict or Classify?

How to split a Dataset into training and testing for machine learning?

I would also add it's worth doing some spelunking on the Machine Learning package - you get some interesting insights into what's going on. There is a lot going on under the surface.

Just came from a Wolfram conference today 10.1 is imminent will be interesting to see if machine learning is more configurable.

Edit: Here's some insights just from

Options[classifier function name]

Answer 2

Cross validation is normally used to overcome the problem of overfitting instead of to optimize regularization parameters of a classifier. Although we can combine cross validation and othe techinques like Grid search to optimize the parameters.

Mathematica uptill V11 seems do not cantain built-in support n cross validation support, but one can easily implement this functionality. The following example show how to do n fold cross validation.

nCrossValidation[dataset_, n_] := 
 Module[{trainSet, testSet, size, i, accScores = {}, c},
  size =  IntegerPart[ Length[dataset] / n];
  For[i = 0, i < n, i++, 
   trainSet = 
    Join[dataset[[1 ;; i * size ]], dataset[[(i + 1) * size + 1 ;; ]]];
   testSet = dataset[[i * size + 1 ;; (i + 1) * size]];
   c = Classify[trainSet];
   AppendTo[accScores, ClassifierMeasurements[c, testSet, "Accuracy"]];
   ];
  <| "folds_scores" -> accScores, "avg_scores" -> Mean[accScores] |>
  ]

In[66]:= nCrossValidation[train, 5]

Out[66]= <|"folds_scores" -> {0.859076, 0.807215, 0.781285, 0.894025, 
   0.860203}, "avg_scores" -> 0.840361|>