7
$\begingroup$

Other machine learning libraries have utilities to generate indices that can be used to generate dataset splits according to different cross-validation strategies. How do I tell Classify to optimize regularization parameters with a cross-validation strategy?

$\endgroup$
  • 1
    $\begingroup$ Cross-validation is a "meta" learning method, one that describes how to use true learning methods (e.g., neural networks, Support Vector Machines, nearest-neighbor algorithm, etc.). You perform $n$-fold cross validation by writing a loop that repeatedly trains the classifier (e.g., SupportVectorMachine) with a portion of the data and tests with the rest. As such, cross-validation is not (and should not be) implemented as 'part' of a particular learning method. $\endgroup$ – David G. Stork Mar 5 '15 at 21:23
  • $\begingroup$ @DavidG.Stork correct, I didn't mean to confuse by mentioning svm's. I'll post my data to help! $\endgroup$ – M.R. Mar 5 '15 at 21:26
7
$\begingroup$

If you look closely at the documentation you will see that classify and predict both perform Cross validation of some description - unfortunately it's not clear from the help what type or degree is being done. I think it's reasonable to assume it's probably a low order n fold validation but there is no way to be sure it's meeting your needs without a query to Wolfram tech support.

See these questions for a bit more detail - particularly the comments from Etienne and Taliesin, note the override option that's mentioned :

How to know the internal algorithms of functions like Predict or Classify?

How to split a Dataset into training and testing for machine learning?

I would also add it's worth doing some spelunking on the Machine Learning package - you get some interesting insights into what's going on. There is a lot going on under the surface.

Just came from a Wolfram conference today 10.1 is imminent will be interesting to see if machine learning is more configurable.

Edit: Here's some insights just from

Options[classifier function name]

Options on Classifier

$\endgroup$
  • $\begingroup$ Interesting. Would it be possible to make that word "imminent" slightly more specific? $\endgroup$ – Sjoerd C. de Vries Mar 6 '15 at 14:56
  • 1
    $\begingroup$ @SjoerdC.deVries, I wish I could. At the conference it changed during the day from "right now" until "soon" depending on who you spoke to :). If it helps they were actively demo-ing it via VPN back to WRI Oxford not sure why unless it was to get to UAT paclet servers and the like. If it helps distract you they've also sneaked out a paclet update to support the new DataDrop product that's just gone live on wolfram cloud. See Databin in the help pages. They also re-iterated they were committed to new functionality releases in point releases. JonM mentioned ~3 pages of new or improved stuff $\endgroup$ – Gordon Coale Mar 6 '15 at 16:18
  • $\begingroup$ I happened to stumble upon Datadrop a couple of weeks ago. My first bin just retired. Stephen Wolfram blogged about data dropping yesterday. $\endgroup$ – Sjoerd C. de Vries Mar 6 '15 at 17:31
  • $\begingroup$ I would add that there seem to have been a lot of changes in Predict/Classify in V10.1. Chief of which is that the learning method is now chosen on a feature by feature basis. $\endgroup$ – Gordon Coale Apr 13 '15 at 10:54
3
$\begingroup$

Cross validation is normally used to overcome the problem of overfitting instead of to optimize regularization parameters of a classifier. Although we can combine cross validation and othe techinques like Grid search to optimize the parameters.

Mathematica uptill V11 seems do not cantain built-in support n cross validation support, but one can easily implement this functionality. The following example show how to do n fold cross validation.

nCrossValidation[dataset_, n_] := 
 Module[{trainSet, testSet, size, i, accScores = {}, c},
  size =  IntegerPart[ Length[dataset] / n];
  For[i = 0, i < n, i++, 
   trainSet = 
    Join[dataset[[1 ;; i * size ]], dataset[[(i + 1) * size + 1 ;; ]]];
   testSet = dataset[[i * size + 1 ;; (i + 1) * size]];
   c = Classify[trainSet];
   AppendTo[accScores, ClassifierMeasurements[c, testSet, "Accuracy"]];
   ];
  <| "folds_scores" -> accScores, "avg_scores" -> Mean[accScores] |>
  ]

In[66]:= nCrossValidation[train, 5]

Out[66]= <|"folds_scores" -> {0.859076, 0.807215, 0.781285, 0.894025, 
   0.860203}, "avg_scores" -> 0.840361|>
$\endgroup$
  • $\begingroup$ I am very new to Mathematica. Could you help me understand how to implement the solution presented by m00nlight above? For example, how would I adapt m00nlight's code to perform cross-validation on the following dataset using an SVM classifier? TrainingData = {{0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, $\endgroup$ – Jamie Dec 21 '18 at 14:14
  • $\begingroup$ @JamieDixson please do not use answers for follow up questions. Ask a separate one instead, link add all relevant information there. $\endgroup$ – Kuba Dec 21 '18 at 15:28
  • $\begingroup$ @JamieDixson cross validation dose not depend on which classifier you use. The idea is only to split the dataset into n equal size, and use n-1 as training and the remaining one to test for n time. For example, if you want to use SVM, you could change the line of c = Classify[trainSet] to Classify[trainSet, Methdo -> "SupportVectorMachine"] which will force to use SVM classifier. $\endgroup$ – m00nlight Dec 22 '18 at 17:45
  • $\begingroup$ Do you have more info about what a train dataset looks like in line 66? MMA seems to want a list of assocations according to the errors I'm getting when I try to reverse engineering the code. I'm not sure what a list of zeros from above would show me about the error I'm getting so thought I'd ask. $\endgroup$ – BBirdsell Apr 23 at 6:09
  • $\begingroup$ A train data set should be a list of association, the left part of the association is the data, and the right part is the class value of the output. Please refer to reference.wolfram.com/language/ref/Classify.html $\endgroup$ – m00nlight Apr 23 at 13:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.