How can I add noise to a neural network input?

Question

A common "trick" in neural network training is to train an autoencoder using a "degraded" sample as input and the original sample as target output. The layer type DropoutLayer[] is made for this purpose. But I would like to play with different types of noise, e.g. Gaussian noise, salt&pepper noise, to train a denoising autoencoder.

I can think of a few ways to implement this, but I'm not happy with any of them:

• I can add noise offline and pass the noisy and original samples to NetTrain as input and target, respectively. But then exactly the same noise trained in every round, which probably leads to overfitting.
• I can pass noisy samples to NetTrain as above, but only train a few rounds, then restart with different noisy samples. (i.e. call NetTrain in a loop). But that seems to break a lot of the internal learning-rate heuristics in NetTrain (e.g. the momentum term is reset in every cycle)
• I can link together a constant input, a several dropout layers and a TotalLayer to get a poor man's Gaussian noise approximation, (each DropoutLayer produces a Bernoulli random variable per pixel, and the sum of a number of Bernoulli random variables is approximately Gaussian). This is my best idea so far, but I need a lot of DropoutLayers to get close to a continuous distribution, and it's not very efficient to include an input that's always constant

Answer 1

Use NeuralNetworks`NoiseLayer

<<NeuralNetworks`
noisy = NetGraph[{
    "noise" -> NoiseLayer[{3, 90, 160}, "Distribution" -> NormalDistribution[0, 0.1]],
    "add" -> ThreadingLayer[Plus]
},
    {{"noise", NetPort["Input"]} -> "add"},
    "Input" -> NetEncoder[{"Image", {160, 90}}],
    "Output" -> NetDecoder["Image"]
]

You can find more undocumented definitions with

NeuralNetworks`$LayerData["Noise"]//Dataset

Answer 2

For what it's worth, this is my "add binomial random variate layer":

Needs["NeuralNetworks`"] (* for ScalarTimesLayer, ScalarPlusLayer *)
With[{σ = 0.1, n = 5},
 addNoiseLayer = NetGraph[
   Flatten@{
     TotalLayer[], ScalarTimesLayer[σ*1./5], 
     ScalarPlusLayer[-1*σ], TotalLayer[],
     Table[DropoutLayer[], {n}]
     },
   Flatten@{
     1 -> 2 -> 3 -> 4,
     NetPort["Input"] -> 4,

     Table[NetPort["const"] -> 4 + i -> 1, {i, n}]
     }, "Input" -> inputDim, "const" -> inputDim]]

Answer 3

In Mathematica 12.1, DropoutLayer exposes his mask, so it's possible to train using a random binary mask, with DropoutLayer["OutputPorts" -> "BinaryMask"] (or DropoutLayer["OutputPorts" -> {"Output", "BinaryMask"}])

In Mathematica 12.2, it will be possible to simulate many types of noises (including once where you give the parameters -like mean and variance- through outputs of other subparts of a NetGraph). The novelty is already documented in the current 12.2 builds under the name of RandomArrayLayer.

Answer 4

You can also do this by using a generator function for generating network inputs during NetTrain. Here's a minimal (if somewhat nonsensical) example:

net = NetInitialize @ NetGraph[
   <|
    "chain" -> NetChain[{5, Ramp, 1}],
    "plus" -> ThreadingLayer[Plus]
    |>,
   {
    {NetPort["Input"], NetPort["Noise"]} -> "plus" -> "chain"
    },
   "Input" -> {1}
];

data = List /@ N[Range[10]];
genFun = Function[
   With[{dat = RandomChoice[data, #BatchSize]},
    <|
     "Input" -> dat,
     "Output" -> dat,
     "Noise" -> RandomVariate[NormalDistribution[], {#BatchSize, 1}]
     |>
    ]
];

Test the generator:

genFun[<|"BatchSize" -> 5|>]

<|"Input" -> {{9.}, {3.}, {10.}, {5.}, {5.}}, "Output" -> {{9.}, {3.}, {10.}, {5.}, {5.}}, "Noise" -> {{0.605894}, {-1.57657}, {0.586584}, {-1.8153}, {0.443682}}|>

Train using the generator:

NetTrain[net, genFun]

Each round, genFun will be evaluated to create a new batch of training examples, so each round you'll get fresh random numbers.