# Adding Noise in Neural Net via ElementwiseLayer

I am trying to add noise to a neural net to train a denoising autoencoder. I don't want to pre-calculate the noisy data. Rather, I'd like the noise be applied via a function using the ElementwiseLayer.

I can't get this to work using a pure function I specify. See below:

NoiseFunction[x_, noise_] :=
x + RandomVariate[
NormalDistribution[0, (x + 0.00000000001)/100*noise]]


This function works well when I map it over a preexisting dataset. However, when I try to include it in an ElementwiseLayer, I get the following error:

ElementwiseLayer[NoiseFunction[#, 1] &]


"NoiseFunction[#1,1]& could not be symbolically evaluated as a unary scalar function"

Simpler functions work, such as:

ElementwiseLayer[Exp[#] &]


However whenever the function is called with an option or second variable I get the error:

ElementwiseLayer[Mod[#,2] &]

"Mod[#1,2]& could not be symbolically evaluated as a unary scalar function."

Any ideas how to get more complex functions to work in ElementwiseLayer? Thank you, E

The first point in the documentation for ElementwiseLayer contains a list of functions that can be used with it. Unfortunately it's a pretty limited list. You might have better luck writing a generator for noisy images and using that as a second input for your network, then doing an elementwise add of the noise and the image at some point in the network.

Let's use LeNet and MNIST as an example. Here we'll add noise to images of handwritten images.

First we'll get LeNet, and then we're going to build our new network that takes a noisy image as extra input.

lenet = NetModel["LeNet Trained on MNIST Data"]

net = NetGraph[{
lenet
}, {
{NetPort@"Input", NetPort@"Noise"} -> 1,
1 -> 2
}, "Input" -> NetEncoder[{"Image", 28, ColorSpace -> "Grayscale"}],
"Noise" -> NetEncoder[{"Image", 28, ColorSpace -> "Grayscale"}]]


We can get an idea of what this looks like from the output:

It's pretty simple: we are just adding the input noise to the input image.

Now let's get the data and write a generator function that will give input to our network training.

data = ResourceData["MNIST"];

generator = Function[
Table[Module[{d = RandomChoice[data]}, <|"Input" -> First@d,
"Output" -> Last@d,
"Noise" -> RandomImage[10, 28]|>], #BatchSize]];


If we run this generator, we can see that it outputs an Association with three pairs: "Input", "Noise", and "Output".

Now we can train the network by simply passing the generator to NetTrain.

NetTrain[net, generator]


Now, for instance, if we change the first argument to RandomImage in our generator to add more noise, for instance by changing it to .5, we can see that the network has a much harder time learning, because the descent of the loss is significantly less quick.

I added the noise right at the start of the network, so we don't actually even need to do this inside the network - we could just have written our generator function to give an "Input" with the noise pre-added. However, with this method, we could add the noise at any point in the network, like after the first two ConvolutionLayers in LeNet, or at the bottom of a U-Net. Our only problem would be making sure the noise is of the correct dimensions.

• Hope this makes some sense, I've got the flu 🤒 and might not be thinking too clearly. – Carl Lange Jan 28 '20 at 21:29
• This makes perfect sense. Already adapted your solution to my problem and have it training now. The generator function as input for NetTrain is a neat idea which I will definitely keep using in the future. I used to make endless lists of training examples...Thank you! – Edward Jan 28 '20 at 22:59
• That's great, happy to help! There's some decent tutorial pages about using generator functions that are worth checking out, such as this one: reference.wolfram.com/language/tutorial/… – Carl Lange Jan 28 '20 at 23:03