4
$\begingroup$
 trainingset = 
ExampleData[{"MachineLearning","Mushroom"},"TrainingData"];
testset = ExampleData[{"MachineLearning","Mushroom"},"TestData"];
Train a classifier on a part of the training set:
SeedRandom[1234];
trainingSample = RandomSample[trainingset, 100];
c=Classify[trainingSample,Method->"GradientBoostedTrees"]
samples = testset[[All, 1]];

probs1=c[samples,"Probabilities"];
probs2=c[#,"Probabilities"]&/@samples;
probs1==probs2
True

enter image description here

If we train the model without setting the Method option, and Mathematica happen choose the GradientBoostTrees Method, then in prediction process, model gives two different results.

Both two methods to predict are useful.

Is this a bug? Following code: Batch Run Until one method is GradientBoostTrees

trainingset = ExampleData[{"MachineLearning","Mushroom"},"TrainingData"];
testset = ExampleData[{"MachineLearning","Mushroom"},"TestData"];
samples=testset[[All,1]];
Do[
trainingData=RandomSample[trainingset,100];
c=Classify[trainingData];
Print@trainingData[[1]];
method=c[[1]]["Model","Method"];
probs1=c[samples,"Probabilities"];
probs2=c[#,"Probabilities"]&/@samples;
check=probs1[[All,1]]-probs2[[All,1]]//Chop[#,10^-5]&//Union;
Print[method,"=",check],{i,10}]

problem


The problem seems it was caused by some Processor and FeatureExtractor In some times, the result is extremely different, which means the model is useless.

If we remove Missing data and Boolean Data in dataset, the problem is still there.

trainingset = ExampleData[{"MachineLearning","Mushroom"},"TrainingData"];
testset = ExampleData[{"MachineLearning","Mushroom"},"TestData"];
samples=testset[[All,1]];
rules={Missing[]->"Missing"};
dataTrain=trainingset/.rules;
Do[
trainingData=RandomSample[dataTrain,100];
c=Classify[trainingData,Method->"GradientBoostedTrees",FeatureExtractor->"DimensionReducedVector"];
Print@trainingData[[1]];
method=c[[1]]["Model","Method"];
processor=c[[1]]["Model","Processor"];
Print[processor];
If[method!="GradientBoostedTrees",Continue,
probs1=c[samples/.rules,"Probabilities"];
probs2=c[#,"Probabilities"]&/@samples;
check=probs1[[All,1]]-probs2[[All,1]]//Chop[#,10^-5]&//Union;
Print[method,"=",check]],{i,10}]

enter image description here

The following result is funny The same sample in a batch will get two different results in predict.

My test notebook is here.

enter image description here

$\endgroup$
  • 1
    $\begingroup$ Please provide a reproducible code with a fixed random seed for the sample. $\endgroup$ – swish May 27 at 21:20
  • $\begingroup$ @swish Updated, $\endgroup$ – HyperGroups May 28 at 7:33
  • $\begingroup$ Can confirm some funky behavior in tree related Classify methods, probabilities may differ drastically whether one predicts samples in bulk or not. $\endgroup$ – swish May 28 at 9:27
3
$\begingroup$

Not an answer, but a more concrete example of when its happening

trainingset = 
  ExampleData[{"MachineLearning", "Mushroom"}, "TrainingData"];
testset = ExampleData[{"MachineLearning", "Mushroom"}, "TestData"];
samples = testset[[All, 1]];
BlockRandom[
 trainingData = RandomSample[trainingset, 1000];
 c = Classify[trainingData, Method -> "DecisionTree"],
 RandomSeeding -> 1
 ]
c[samples[[79]], "Probabilities"]
c[samples[[{1, 2, 79}]], "Probabilities"][[-1]]

At first run the last two probabilities are the same, but rut this code again, and it yields a complete opposite predictions!

$\endgroup$
  • $\begingroup$ Your example is designated the Tree method in Classify, that's a great found. $\endgroup$ – HyperGroups May 29 at 12:42
  • $\begingroup$ I found the problem is probably caused by preprocessing of Classify. If we encode the features manually, we can partly solve the problem. $\endgroup$ – HyperGroups Jun 4 at 11:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.