# Two different results produced by Classify

 trainingset =
ExampleData[{"MachineLearning","Mushroom"},"TrainingData"];
testset = ExampleData[{"MachineLearning","Mushroom"},"TestData"];
Train a classifier on a part of the training set:
SeedRandom[1234];
trainingSample = RandomSample[trainingset, 100];
samples = testset[[All, 1]];

probs1=c[samples,"Probabilities"];
probs2=c[#,"Probabilities"]&/@samples;
probs1==probs2
True


If we train the model without setting the Method option, and Mathematica happen choose the GradientBoostTrees Method, then in prediction process, model gives two different results.

Both two methods to predict are useful.

Is this a bug? Following code: Batch Run Until one method is GradientBoostTrees

trainingset = ExampleData[{"MachineLearning","Mushroom"},"TrainingData"];
testset = ExampleData[{"MachineLearning","Mushroom"},"TestData"];
samples=testset[[All,1]];
Do[
trainingData=RandomSample[trainingset,100];
c=Classify[trainingData];
Print@trainingData[[1]];
method=c[[1]]["Model","Method"];
probs1=c[samples,"Probabilities"];
probs2=c[#,"Probabilities"]&/@samples;
check=probs1[[All,1]]-probs2[[All,1]]//Chop[#,10^-5]&//Union;
Print[method,"=",check],{i,10}]


The problem seems it was caused by some Processor and FeatureExtractor In some times, the result is extremely different, which means the model is useless.

If we remove Missing data and Boolean Data in dataset, the problem is still there.

trainingset = ExampleData[{"MachineLearning","Mushroom"},"TrainingData"];
testset = ExampleData[{"MachineLearning","Mushroom"},"TestData"];
samples=testset[[All,1]];
rules={Missing[]->"Missing"};
dataTrain=trainingset/.rules;
Do[
trainingData=RandomSample[dataTrain,100];
Print@trainingData[[1]];
method=c[[1]]["Model","Method"];
processor=c[[1]]["Model","Processor"];
Print[processor];
probs1=c[samples/.rules,"Probabilities"];
probs2=c[#,"Probabilities"]&/@samples;
check=probs1[[All,1]]-probs2[[All,1]]//Chop[#,10^-5]&//Union;
Print[method,"=",check]],{i,10}]


The following result is funny The same sample in a batch will get two different results in predict.

My test notebook is here.

• Please provide a reproducible code with a fixed random seed for the sample. May 27 '19 at 21:20
• @swish Updated, May 28 '19 at 7:33
• Can confirm some funky behavior in tree related Classify methods, probabilities may differ drastically whether one predicts samples in bulk or not. May 28 '19 at 9:27

Not an answer, but a more concrete example of when its happening

trainingset =
ExampleData[{"MachineLearning", "Mushroom"}, "TrainingData"];
testset = ExampleData[{"MachineLearning", "Mushroom"}, "TestData"];
samples = testset[[All, 1]];
BlockRandom[
trainingData = RandomSample[trainingset, 1000];
c = Classify[trainingData, Method -> "DecisionTree"],
RandomSeeding -> 1
]
c[samples[[79]], "Probabilities"]
c[samples[[{1, 2, 79}]], "Probabilities"][[-1]]


At first run the last two probabilities are the same, but rut this code again, and it yields a complete opposite predictions!

• Your example is designated the Tree method in Classify, that's a great found. May 29 '19 at 12:42
• I found the problem is probably caused by preprocessing of Classify. If we encode the features manually, we can partly solve the problem. Jun 4 '19 at 11:37