12
$\begingroup$

I was playing with the neural network in Mathematica and found that the same neural network gives different classification results to the same image, which is very strange.

Consider classifying this dog image using the "Wolfram ImageIdentify Net":

enter image description here

imgd = 
  ImageResize[
   ImageCrop@Import["https://i.stack.imgur.com/8HXqN.jpg"], {224, 224}];

net = NetModel["Wolfram ImageIdentify Net for WL 11.1"];

net[imgd]
(* Entity["Concept", "GoldenRetriever::t59tg"] *)

We see that the image is correctly classified.

Now I construct another neural network using part of the first network

array = List@Normal[SparseArray[{2241 -> 1}, {4315}]];
net2 = 
  NetChain[{Take[net, {1, "softmax"}], 
    LinearLayer[1, "Weights" -> array, "Biases" -> None], 
    SummationLayer[]}];

then do something irrelevant to the first neural network

imtest = imgd;
Table[
  gd = net2[imtest, NetPortGradient["Input"]];
  gdimg = ImageAdjust@NetDecoder["Image"][gd];
  imtest = ImageAdjust[imtest + 0.1*gdimg];
  , {20}];

And now the classification of the dog image using the first neural network is not correct anymore

net[imgd]
(* Entity["Concept", "Colander::73kyp"] *)

However, both the image and the neural network are identical as before

imgd == 
 ImageResize[
  ImageCrop@Import["https://i.stack.imgur.com/8HXqN.jpg"], {224, 224}]
(* True *)

net == NetModel["Wolfram ImageIdentify Net for WL 11.1"]
(* True *)

So why does this happen? Is this a bug?

I'm using Mathematica version 11.1.1.0 on macOS 10.12.4.

It looks like the problem is in the BatchNormalizationLayer. Although the two BatchNormalizationLayers have identical parameters, they produce different results:

net3 = NetModel["Wolfram ImageIdentify Net for WL 11.1"];
data = Take[net, {1, "conv_1"}][imgd];

InputForm[NetExtract[net, "bn_1"]] == InputForm[NetExtract[net3, "bn_1"]]
(* True *)

NetExtract[net, "bn_1"][data] == NetExtract[net3, "bn_1"][data]
(* False *)

I've also tested other models, it seems that only models contain the batch normalization layers have this problem. Networks don't have batch normalization such as VGG-16 behaves as what we expected.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Exploration of the problem. This is not an answer.

Problem may be related with NetPortGradient.

Quit[]

imgd = ImageResize[ImageCrop@Import["https://i.stack.imgur.com/8HXqN.jpg"], {224, 224}];
net = NetModel["Wolfram ImageIdentify Net for WL 11.1"];
net[imgd]

enter image description here

net2 = net;

SeedRandom[0];
Do[
 {
   i,
   img = imgd + RandomReal[{0, 1}];
   net2[img];
   net@imgd
   } // Print,
 {i, 20}
 ]

enter image description here

SeedRandom[0];
Do[
 {
   i,
   img = imgd + RandomReal[{0, 1}];

   (*-->*) net2[img, NetPortGradient["Input"]]; (*<--*)

   net@imgd
   } // Print,
 {i, 20}
 ]

enter image description here

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for exploring this. The strange thing about it is that the two nets give different results, even when all the weights in the nets are identical. $\endgroup$
    – xslittlegrass
    May 4, 2017 at 15:49
  • $\begingroup$ This problem is due to BatchNormalizationLayer: ``` b = NetInitialize@BatchNormalizationLayer["Input" -> {2}]; data = {0.3, 0.4}; out1 = b[data] b[data, NetPortGradient["Input"]]; out2 = b[data] Out[37]= {0.29985, 0.3998} Out[39]= {0.284447, 0.379263} ``` Our backend MXNet has not supported calculating grads when not in training mode. But in training mode, the moving mean and variance of BatchNorm get automatically updated. Support for calculating grads in test mode has recently been added: github.com/apache/incubator-mxnet/issues/7189. This will automatically solve this. $\endgroup$
    – Sebastian
    Aug 7, 2017 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.