Error using accuracy and ConfusionMatrixPlot of ClassifierMeasurements

Question

EDIT: Here is the complete code that i have used:

class = 2;  
className = {"car", "dog"};
width = 224; height = 224; 
dir = SetDirectory["C:\\Users\\cnn\\Desktop\\images"];
loadFiles[dir_] := 
 Map[File[#] -> FileNameTake[#, {-2}] &, 
  FileNames["*.jpg", dir, Infinity]]; trainingData = 
 loadFiles[FileNameJoin[{dir, "train"}]];
testingData = loadFiles[FileNameJoin[{dir, "test"}]];
files = FileNames["*.JPG" | "*.JPEG", 
  "C:\\Users\\cnn\\Desktop\\images\\test", Infinity] 
testData = Import[#, ImageSize -> {224, 224}] & /@ files;
netEncoder = 
  NetEncoder[{"Image", {width, height}, ColorSpace -> "RGB"}];
netDecoder = NetDecoder[{"Class", className}];
conv1 = ConvolutionLayer[64, {3, 3}, "PaddingSize" -> 1, 
   "Stride" -> 1 ];
conv2 = ConvolutionLayer[64, {3, 3}, "PaddingSize" -> 1, 
   "Stride" -> 1];
conv3 = ConvolutionLayer[128, {3, 3}, "PaddingSize" -> 1, 
   "Stride" -> 1];
conv4 = ConvolutionLayer[128, {3, 3}, "PaddingSize" -> 1, 
   "Stride" -> 1];
conv5 = ConvolutionLayer[256, {3, 3}, "PaddingSize" -> 1, 
   "Stride" -> 1];
conv6 = ConvolutionLayer[256, {3, 3}, "PaddingSize" -> 1, 
   "Stride" -> 1];
conv7 = ConvolutionLayer[256, {3, 3}, "PaddingSize" -> 1, 
   "Stride" -> 1];
conv8 = ConvolutionLayer[256, {3, 3}, "PaddingSize" -> 1, 
   "Stride" -> 1];
conv9 = ConvolutionLayer[512, {3, 3}, "PaddingSize" -> 1, 
   "Stride" -> 1];
conv10 = ConvolutionLayer[512, {3, 3}, "PaddingSize" -> 1, 
   "Stride" -> 1];
conv11 = ConvolutionLayer[512, {3, 3}, "PaddingSize" -> 1, 
   "Stride" -> 1];
conv12 = ConvolutionLayer[512, {3, 3}, "PaddingSize" -> 1, 
   "Stride" -> 1];
conv13 = ConvolutionLayer[512, {3, 3}, "PaddingSize" -> 1, 
   "Stride" -> 1];
conv14 = ConvolutionLayer[512, {3, 3}, "PaddingSize" -> 1, 
   "Stride" -> 1];
conv15 = ConvolutionLayer[512, {3, 3}, "PaddingSize" -> 1, 
   "Stride" -> 1];
conv16 = ConvolutionLayer[512, {3, 3}, "PaddingSize" -> 1, 
   "Stride" -> 1];

block1 = {BatchNormalizationLayer[], conv1, Ramp, conv2, Ramp, 
   PoolingLayer[2, 2], conv3, Ramp, conv4, Ramp, PoolingLayer[2, 2], 
   conv5, Ramp, conv6, Ramp, conv7, Ramp, conv8, Ramp, 
   PoolingLayer[2, 2], conv9, Ramp, conv10, Ramp, conv11, Ramp, 
   conv12, Ramp, PoolingLayer[2, 2], conv13, Ramp, conv14, Ramp, 
   conv15, Ramp, conv16, Ramp, PoolingLayer[2, 2], FlattenLayer[], 
   4096, Ramp, DropoutLayer[0.50], 4096, Ramp, DropoutLayer[0.50], 
   class, SoftmaxLayer[]};

vggNet = NetChain[block1, "Output" -> netDecoder, 
  "Input" -> netEncoder]
vggNet2 = NetInitialize[vggNet];
trainedNet = 
 NetTrain[vggNet2, trainingData, ValidationSet -> testingData, 
  MaxTrainingRounds -> 1]
cm = ClassifierMeasurements[trainedNet, testData]
cm["Accuracy"]
cm["ConfusionMatrixPlot"]

c = Classify@trainedNet;
ClassifierMeasurements[c, testData, Accuracy]
ClassifierMeasurements[c, testData, F1Score]
ClassifierMeasurements[c, testData, ConfusionMatrixPlot]
c /@ {Accuracy, FScore, ConfusionMatrixPlot} // TableForm

Here, I have used a folder named images which contain two subfolders named train and test. Since i am doing all this on a CPU i just used two subfolders in train and test named car and dog which contained 70 images each. The test folder also has same subfolders with 30 images each. My goal is to just find out my network accuracy and plot the confusion matrix. END EDIT

I have come across yet another problem. I am trying to create VGGNet and with some help have overcome obstacles till training. I am facing problem with finding out the accuracy of my network and also plotting the confusion matrix. I had referred the documentation as well as the community discussions and found that my goals can be achieved by two methods:

1)

c = Classify@trainedNet;
ClassifierMeasurements[c, testingData, "Accuracy"]
ClassifierMeasurements[c, testingData, "F1Score"]
ClassifierMeasurements[c, testingData, "ConfusionMatrixPlot"]
c /@ {"Accuracy", "FScore", "ConfusionMatrixPlot"} // TableForm

when I use the above code, I get the following error:

ClassifierMeasurements::mlincfttp: Incompatible variable type (Image) and variable value
 (File[C:\Users\cnn\Desktop\images\test\car\00301.jpg]).

2) I believe another method is

cm = ClassifierMeasurements[trainedNet, testingData]
cm["Accuracy"]
cm["ConfusionMatrixPlot"]

But when I use the above code I get the following error:

ClassifierMeasurements::mlincfttp: Incompatible variable type (Image) and variable value
(File[C:\Users\cnn\Desktop\images\test\car\00301.jpg]).

Initial part of my code is:

dir = SetDirectory["C:\\Users\\cnn\\Desktop\\images"];
loadFiles[dir_] := 
 Map[File[#] -> FileNameTake[#, {-2}] &, 
  FileNames["*.jpg", dir, Infinity]];
 trainingData = 
     loadFiles[FileNameJoin[{dir, "train"}]]; 
testingData = 
     loadFiles[FileNameJoin[{dir, "test"}]];
    files = FileNames["*.JPG" | "*.JPEG", 
      "C:\\Users\\cnn\\Desktop\\images\\test", Infinity] 
    testData = Import[#, ImageSize -> {224, 224}] & /@ files;

No matter whether I use testData or testingData for classifiermeasurements, I am still getting errors and am unable to get the accuracy as well as plot the confusion matrix. Any help would be highly appreciated.

EDIT: I am also getting the following error: ClassifierMeasurements::bdfmt: Argument {,,,,,,<<40>>,,Image[RawArray[UnsignedInteger8,<75,50,3>],Byte,ColorSpace->ColorProfileData[RawArray[UnsignedInteger8,<560>],RGB,XYZ],ImageResolution->{240,240},Interleaving->True],Image[RawArray[UnsignedInteger8,<75,50,3>],Byte,ColorSpace->ColorProfileData[RawArray[UnsignedInteger8,<560>],RGB,XYZ],ImageResolution->{240,240},Interleaving->True],,<<10>>} should be a rule, a list of rules, or an association.

for the code:

cm = ClassifierMeasurements[trainedNet, testData]
cm["Accuracy"]
cm["ConfusionMatrixPlot"]

Thank you

Ashish

Answer 1

This is to address your question concerning incompatible data type message:

ClassifierMeasurements::mlincfttp: Incompatible variable type (Image) and variable value (File[C:\Users\cnn\Desktop\images\test\car\00301.jpg]).

I encountered a similar message while running ClassifierMeasurements. This function generated intermittent errors for some test datasets, but not for some others drawn from the same master dataset. What happened is as follows.

When training a classifier using Classify, Mathematica makes assumptions about the data types, and then it expects the same data types in the test (and new) data. For example, in my case, there was a relatively small master dataset from which training and test sets were drawn. One variable (out of some 400) happended to take values 0,1 and 3, however sometimes in the training data only instances with values 0 and 1 were present. ClassifyInformation showed that such variables were treated as Boolean. So if the value of 3 would be present for this variable in some tests, then the ClassifierMeasurements would generate a message complaint about an incompatible variable type. I fixed the problem by applying N to the corresponding variable to make it real and to force it to be recognized as numerical (instead of Boolean). It took significant time to figure that out, and I wish there was an explicit way of specifying variable types for the classifiers. If someone knows how to do this, it would be helpful.