# Generalized Backpropagation for Neural Networks (e.g. DeepDream)

Is there currently any way to abuse NetTrain to maximize an arbitrary cost/utility -function in some hidden layer and/or even modify the input to increase neuron activation? An application for this could be something like Google DeepDream from which the image below was taken (see this youtube video for a comprehensive explaination).

The image was (most likely) taken by training a convolutional neural network for recognizing buildings and then iteratively modifying a supplied input image to maximize the neuron activation in neurons associated with recognizing buildings.

Beside being fun to look at one interesting real world application could be to investigate earlier layers in a network and try to get a feeling of what features and increasingly abstract concepts a network is actually learning (edges, ovals, eyes, faces, cats etc.) by looking at what it "dreams". See the image below for a network that seems to learn edges and patterns.

• In his presentation of A Shallow Tour of Deep Learning, Sebastian Bodenstein made a similar plot using ImageIdentify, but he didn't talk about how he made it. – xslittlegrass Sep 8 '16 at 14:44
• @xslittlegrass Now I'd like to see A Deep Tour of Deep Learning – Sascha Sep 8 '16 at 15:47
• @Szabolcs Thanks for adding a bounty:) – Sascha Sep 13 '16 at 10:37
• I suppose this problem is impossible to solve in current version as it requires RNN but RNN is not avaliable yet. – Wjx Sep 13 '16 at 11:25
• @Wjx As far as I understand the theory behind DeepDream it is only a more general form of Backpropagation where one does not start from the output (minimizing some distance measure/cost-/utility-function between what the network outputs and the training set), but can use Backpropagation to minimize/maximize an arbitrary distance measure/cost-/utility-function e.g. the activation of a specific neuron or group of neurons (for instance neurons found to be related to recognizing e.g. cats). – Sascha Sep 13 '16 at 11:45

Sebastian mentioned in his answer that deepdream can be possible using NetDerivative. Here are my attempts following his outlines.

Instead of using the inception model, I'm using VGG-16 here for simplicity. Inception model allows arbitrary image size, but may need some extra normalization steps.

The MXNet version of the VGG model can be downloaded from MXnet's github page https://github.com/dmlc/mxnet-model-gallery

The VGG model can be loaded as

Needs["NeuralNetworks"]
VGG = ImportMXNetModel["vgg16-symbol.json", "vgg16-0000.params"]


The VGG is trained by 224 by 224 images, so we will load and crop our test image to this size

img = RemoveAlphaChannel@
ImageCrop[
Import["http://hplussummit.com/images/wolfram.jpg"], {224, 224},


We can then take the network up to the layer at which we want maximum activation, and then attach a SummationLayer

net = NetChain[{Take[VGG, {"conv1_1", "pool1"}], SummationLayer[]},
"Input" -> NetEncoder[{"Image", {224, 224}}]]


We then want to back propagate to the input, and find the gradient this respect to the value at the summation layer. This can be done with NetDerivative. This is what the gradient looks like

NetDecoder["Image"][
First[(NetDerivative[net]@<|"Input" -> img|>)[
NetPort["Inputs", "Input"]]]]


We can now add the gradient to the input image, and calculate the gradient with respect this new image, and so and so forth. This process can also be understood as gradient ascent. Here is the function that does one step of gradient ascent

applyGradient[net_, img_, stepsize_] :=
Module[{imgt, gdimg, gddata, max, dim},
gdimg =
NetDecoder["Image"][
First[(NetDerivative[net]@<|"Input" -> img|>)[
NetPort["Inputs", "Input"]]]];
gddata = ImageData[gdimg];
max = Max@Abs@gddata;
Image[ImageData[img] + stepsize*gddata/max]
]


We can then apply this repeatedly to our input image and get a deepdream like image

Nest[applyGradient[net, #, 0.1] &, img, 10]


Here are the dreamed image at different pooling layers. When we dream at early pooling layer, localized simple features show up. And as we get to later layers of the network, more complexity features emerges.

dream[img_, layer_, stepsize_, steps_] := Module[{net},
net = NetChain[{Take[VGG, {"conv1_1", layer}], SummationLayer[]},
"Input" -> NetEncoder[{"Image", {224, 224}}]];
Nest[applyGradient[net, #, stepsize] &, img, steps]
]

Show[ImagePad[dream[img, #1, #2, #3], 10, White],
ImageSize -> {224, 224}] & @@@ {{"pool2", 0.1, 10}, {"pool3", 0.1,
10}, {"pool4", 0.2, 20}, {"pool5", 0.5, 20}} // Row


The way to generate the deep-dream images can also be used to visualize the convolution layers in the model. To visualize on convolution kernel, we can attach a fully connected layer right after the convolution layer that keeps only one of the filter channel and zero all others.

weights[n_] := List@PadRight[Table[0., n - 1]~Join~{1.}, 1000]
maxAtLayer[img_, n_] := Module[{net, result},
net = NetChain[{Take[VGG, {"conv1_1", "conv5_3"}], FlattenLayer[],
DotPlusLayer[1, "Weights" -> weights[n], "Biases" -> {0}],
SummationLayer[]},
"Input" -> NetEncoder[{"Image", {224, 224}}]];
Nest[applyGradient[net, #, 0.01] &, img, 30];
result
]

img = Image[(20*RandomReal[{0, 1}, {224, 224, 3}] + 128)/255.];

maxAtLayer[img, RandomInteger[{1,512}]]


The same can be applied to the last layer of the network

• +1 for A New Kind of Answering my question – Sascha Dec 9 '16 at 8:09
• Why do you say Inception models allow arbitrary image size? I looked in the wolfram neural net repo and it has a 224x224 NetEncoder just like VGG-16 – Nico A Jul 3 '19 at 3:38

If you have an inception model, its mostly possible using hidden functionality (but without GPU training). The steps would look like this:

1. Cut the inception model at some level using Take. Then add a Ramp (or other) positive activation and SummationLayer: DeepDream simply wants to maximize the total amount of neuronal firing at the last layer (the original DeepDream implementation squares the output of the last layer, which isn't possible right now, but will be in 11.1). Using a SummationLayer produces a single output, so this can be used as a loss that can be differentiated.

2. NetDerivative allows you differentiate networks (buts its hidden functionality and might be modified/replaced). You need this to obtain the derivative of the output with respect to the input

<<NeuralNetworks
chain = NetChain[{ElementwiseLayer[LogisticSigmoid]}]
NetDerivative[chain]@<|"Input" -> {2, 3, 4}|>

3. Optimize + Jitter: do gradient descent on the input image to find the image that maximizes the total activation of the final layer. But this alone doesn't produce any interesting results (this is a standard problem generating images from discriminative models). The key is to add some regularization, which in the case of DeepDream is random pixel jittering per iteration (see here for more detail + details of the image upscaling and downscaling used to improve results).

I plan to add a DeepDream example to the Mathematica docs once NetDerivative is a system function. And obviously these are just the steps you might follow, not a solution. But I thought it would be interesting for people to know how they might implement this.

• Thanks for the answer! The bounty would have expired in less than an hour so I awarded it right away. – Szabolcs Sep 20 '16 at 8:59
• Not sure if I deserve it given that its not a fully working example... But thanks! – Sebastian Sep 20 '16 at 9:04
• Thank you for this interesting outlook! Could you perhaps include a quick sketch of building and taking apart the inception model in your answer? A small example network would suffice. – Sascha Sep 20 '16 at 9:23
• @Sebastian I allowed myself to add a link to a relevant part of Google's Udacity course on Deep Learning – Sascha Sep 20 '16 at 9:41
• Is NetDerivative gone? what did replace it? – Fortsaint Jan 8 '19 at 13:50

Time to revitalize this question thanks to Mr. Aster Ctor with DeepDreamBeta and DeepDreamAlpha