Classifier Random Forest Accuracy Not Consistent

Question

I tried running a random forest classifier over different training sample sizes. The results are very strange because I could not replicate my accuracy with the same sample size in the second iteration although nothing else changed. Moreover the accuracy does not improve with larger sample sizes.

My results:

TSS Method  Accuracy(1st Iteration) Accuracy(2nd Iteration)
5   RF      0,514       
20  RF      0,510       
50  RF      0,524       
100 RF      0,541                   0,507   
110 RF      0,496
200 RF      0,506       
300 RF      0,502       
400 RF      0,445

Is the implementation of the classifier or method wrong or are these results ok?

Answer 1

Is the implementation of the classifier or method wrong or are these results ok?

We cannot conclude from the posted table of classification accuracy results that "the classifier or method is wrong". I can come up with at least several explanations. (And it would be nice if data is provided.)

The simplest explanation is that none of your variables is decisive for the class labels in your data. Below is an example.

Example of no accuracy change

Consider the following steps:

1. Take the Titanic data.
2. Remove the variables "passenger class" and "passenger sex".
3. Build and a measure the accuracy of classifiers with training data samples with different sizes.

Here is the code for these steps:

testSetName = "Titanic";

trainingSet = 
  ExampleData[{"MachineLearning", testSetName}, "TrainingData"];
 testSet = ExampleData[{"MachineLearning", testSetName}, "TestData"];

trainingSet2 = trainingSet;
trainingSet2[[All, 1]] = trainingSet2[[All, 1]][[All, {2}]];
testSet2 = testSet;
testSet2[[All, 1]] = testSet2[[All, 1]][[All, {2}]];

clRes = Table[(
    clFunc2 = 
     Classify[RandomSample[trainingSet2, n], 
      Method -> "RandomForest"]; 
    cm2 = ClassifierMeasurements[clFunc2, testSet2]; {n, 
     cm2["Accuracy"]}), {n, {5, 20, 50, 100, 200, 300, 400}}];
TableForm[clRes, TableHeadings -> {None, {"Sample size", "Accuracy"}}]

We can explain the classification results by looking at the histograms of the age distributions wishing the class labels:

asPairs = 
  DeleteCases[
   Map[Flatten@Apply[List, #] &, trainingSet2], {___, _Missing, ___}];

Histogram[#[[All, 1]], 20, "Probability"] & /@ GroupBy[asPairs, Last]

We can see that the histograms are too much alike, hence "passenger age" is not a decisive variable.

Importance of variables investigation

If we apply the procedure described in my answer of "How can I determine the importance of variables from Classify?" we can see that "passenger age" has no decisiveness on the classification (and "passenger sex" is quite decisive):