1
$\begingroup$

I tried running a random forest classifier over different training sample sizes. The results are very strange because I could not replicate my accuracy with the same sample size in the second iteration although nothing else changed. Moreover the accuracy does not improve with larger sample sizes.

My results:

TSS Method  Accuracy(1st Iteration) Accuracy(2nd Iteration)
5   RF      0,514       
20  RF      0,510       
50  RF      0,524       
100 RF      0,541                   0,507   
110 RF      0,496
200 RF      0,506       
300 RF      0,502       
400 RF      0,445

Is the implementation of the classifier or method wrong or are these results ok?

$\endgroup$
2
$\begingroup$

Is the implementation of the classifier or method wrong or are these results ok?

We cannot conclude from the posted table of classification accuracy results that "the classifier or method is wrong". I can come up with at least several explanations. (And it would be nice if data is provided.)

The simplest explanation is that none of your variables is decisive for the class labels in your data. Below is an example.

Example of no accuracy change

Consider the following steps:

  1. Take the Titanic data.
  2. Remove the variables "passenger class" and "passenger sex".
  3. Build and a measure the accuracy of classifiers with training data samples with different sizes.

Here is the code for these steps:

testSetName = "Titanic";

trainingSet = 
  ExampleData[{"MachineLearning", testSetName}, "TrainingData"];
 testSet = ExampleData[{"MachineLearning", testSetName}, "TestData"];

trainingSet2 = trainingSet;
trainingSet2[[All, 1]] = trainingSet2[[All, 1]][[All, {2}]];
testSet2 = testSet;
testSet2[[All, 1]] = testSet2[[All, 1]][[All, {2}]];

clRes = Table[(
    clFunc2 = 
     Classify[RandomSample[trainingSet2, n], 
      Method -> "RandomForest"]; 
    cm2 = ClassifierMeasurements[clFunc2, testSet2]; {n, 
     cm2["Accuracy"]}), {n, {5, 20, 50, 100, 200, 300, 400}}];
TableForm[clRes, TableHeadings -> {None, {"Sample size", "Accuracy"}}]

enter image description here

We can explain the classification results by looking at the histograms of the age distributions wishing the class labels:

asPairs = 
  DeleteCases[
   Map[Flatten@Apply[List, #] &, trainingSet2], {___, _Missing, ___}];

Histogram[#[[All, 1]], 20, "Probability"] & /@ GroupBy[asPairs, Last]

enter image description here

We can see that the histograms are too much alike, hence "passenger age" is not a decisive variable.

Importance of variables investigation

If we apply the procedure described in my answer of "How can I determine the importance of variables from Classify?" we can see that "passenger age" has no decisiveness on the classification (and "passenger sex" is quite decisive):

enter image description here

$\endgroup$
  • $\begingroup$ Hi. Thank for the reply. I am trying to classify pictures of tables and chairs and wanted to compare sample sizes and different classifier methods. So there are no variables like in the titanic data set. $\endgroup$ – user3483676 May 25 '16 at 17:12
  • $\begingroup$ @user3483676 I have worked on classifiers with/for images, and I have seen classifiers with overall accuracies like the ones you show (close to 0.5). But for the most important class labels I was able to make the classifiers to achieve 0.7-0.8 precision and and 0.6-0.8 recall (while the overall accuracy is still being ~0.5). $\endgroup$ – Anton Antonov May 25 '16 at 17:22
  • $\begingroup$ How important is preprocessing of pictures? If you use ConformImages don't you lose a lot of information in the picture? It is faster of course and you run out of memory otherwise. $\endgroup$ – user3483676 May 26 '16 at 2:59
  • $\begingroup$ @user3483676 Yeah processing is important, I would say half the battle. You might also consider using image data like luminosity, levels, etc. which are found before using ConformImages. $\endgroup$ – Anton Antonov May 26 '16 at 3:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.