4
$\begingroup$
 trainingset = 
ExampleData[{"MachineLearning","Mushroom"},"TrainingData"];
testset = ExampleData[{"MachineLearning","Mushroom"},"TestData"];
Train a classifier on a part of the training set:
SeedRandom[1234];
trainingSample = RandomSample[trainingset, 100];
c=Classify[trainingSample,Method->"GradientBoostedTrees"]
samples = testset[[All, 1]];

probs1=c[samples,"Probabilities"];
probs2=c[#,"Probabilities"]&/@samples;
probs1==probs2
True

enter image description here

If we train the model without setting the Method option, and Mathematica happen choose the GradientBoostTrees Method, then in prediction process, model gives two different results.

Both two methods to predict are useful.

Is this a bug? Following code: Batch Run Until one method is GradientBoostTrees

trainingset = ExampleData[{"MachineLearning","Mushroom"},"TrainingData"];
testset = ExampleData[{"MachineLearning","Mushroom"},"TestData"];
samples=testset[[All,1]];
Do[
trainingData=RandomSample[trainingset,100];
c=Classify[trainingData];
Print@trainingData[[1]];
method=c[[1]]["Model","Method"];
probs1=c[samples,"Probabilities"];
probs2=c[#,"Probabilities"]&/@samples;
check=probs1[[All,1]]-probs2[[All,1]]//Chop[#,10^-5]&//Union;
Print[method,"=",check],{i,10}]

problem


The problem seems it was caused by some Processor and FeatureExtractor In some times, the result is extremely different, which means the model is useless.

If we remove Missing data and Boolean Data in dataset, the problem is still there.

trainingset = ExampleData[{"MachineLearning","Mushroom"},"TrainingData"];
testset = ExampleData[{"MachineLearning","Mushroom"},"TestData"];
samples=testset[[All,1]];
rules={Missing[]->"Missing"};
dataTrain=trainingset/.rules;
Do[
trainingData=RandomSample[dataTrain,100];
c=Classify[trainingData,Method->"GradientBoostedTrees",FeatureExtractor->"DimensionReducedVector"];
Print@trainingData[[1]];
method=c[[1]]["Model","Method"];
processor=c[[1]]["Model","Processor"];
Print[processor];
If[method!="GradientBoostedTrees",Continue,
probs1=c[samples/.rules,"Probabilities"];
probs2=c[#,"Probabilities"]&/@samples;
check=probs1[[All,1]]-probs2[[All,1]]//Chop[#,10^-5]&//Union;
Print[method,"=",check]],{i,10}]

enter image description here

The following result is funny The same sample in a batch will get two different results in predict.

My test notebook is here.

enter image description here

$\endgroup$
3
  • 1
    $\begingroup$ Please provide a reproducible code with a fixed random seed for the sample. $\endgroup$
    – swish
    May 27, 2019 at 21:20
  • $\begingroup$ @swish Updated, $\endgroup$ May 28, 2019 at 7:33
  • $\begingroup$ Can confirm some funky behavior in tree related Classify methods, probabilities may differ drastically whether one predicts samples in bulk or not. $\endgroup$
    – swish
    May 28, 2019 at 9:27

1 Answer 1

3
$\begingroup$

Not an answer, but a more concrete example of when its happening

trainingset = 
  ExampleData[{"MachineLearning", "Mushroom"}, "TrainingData"];
testset = ExampleData[{"MachineLearning", "Mushroom"}, "TestData"];
samples = testset[[All, 1]];
BlockRandom[
 trainingData = RandomSample[trainingset, 1000];
 c = Classify[trainingData, Method -> "DecisionTree"],
 RandomSeeding -> 1
 ]
c[samples[[79]], "Probabilities"]
c[samples[[{1, 2, 79}]], "Probabilities"][[-1]]

At first run the last two probabilities are the same, but rut this code again, and it yields a complete opposite predictions!

$\endgroup$
2
  • $\begingroup$ Your example is designated the Tree method in Classify, that's a great found. $\endgroup$ May 29, 2019 at 12:42
  • $\begingroup$ I found the problem is probably caused by preprocessing of Classify. If we encode the features manually, we can partly solve the problem. $\endgroup$ Jun 4, 2019 at 11:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.