Issue with Transfer Learning using VGG16 NetModel

Question

I am trying to use the VGG16 architecture available in the Wolfram Neural Network Repository to classify the Fashion MNIST dataset. The VGG 16 by default accepts images of size 3x224x224. However the images in the Fashion MNIST are grayscale of dimensions 1x28x28. I have tried to tune the model by using the command

newNet = NetReplacePart(vgg16, "Input"->NetEncoder[{"Image", {28, 28}, ColorSpace->"Grayscale"}]]

But this throws an error message that says

NetEncoder producing a 1x28x28 array of real numbers cannot be attached to a port Input which must be 3x224x224 array of real numbers.

This is highly frustrating and I cannot find a way through. Any help will be deeply appreciated. Thanks in advance.

Answer 1

It's not immediately obvious to me that transfer learning is appropriate here, considering the datasets are rather different in dimensions - but here goes since you asked:

1. You have to train (at-least) an input convolution layer to take your input images to the size the first trained convolution layer expects , here that's (64,224,224)

inputNet = NetChain[{
   ConvolutionLayer[64, {3, 3}, "PaddingSize" -> {1, 1}], Ramp},
  "Input" -> 
   NetEncoder[{"Image", {224, 224}, ColorSpace -> "Grayscale"}]]

2. That means we also have to discard the first trained conv layer from VGG-16

vgg = NetModel["VGG-16 Trained on ImageNet Competition Data"];
tempNet = Take[vgg, {3, -4}]

3. We can then attach some linear layers at the end together with a Decoder to predict our classes. I was too lazy to download the MNIST fashion data, so here's some dummy data for 10 classes

trainSet = Table[
   RandomImage[1, {224, 224}] -> f[RandomInteger[{1, 10}]], 100];

newNet = NetChain[<|"input" -> inputNet, "pretrainedNet" -> tempNet, 
   "linearNew" -> LinearLayer[10], "softmax" -> SoftmaxLayer[]|>, 
  "Output" -> NetDecoder[{"Class", Array[f, 10]}]]

4. Finally, we can re-train only the input convolutional net and the linear net

trainedNet = 
 NetTrain[newNet, trainSet, 
  LearningRateMultipliers -> {"input" -> 1, "linearNew" -> 1, _ -> 0}]