A common "trick" in neural network training is to train an autoencoder using a "degraded" sample as input and the original sample as target output. The layer type DropoutLayer[] is made for this purpose. But I would like to play with different types of noise, e.g. Gaussian noise, salt&pepper noise, to train a denoising autoencoder.

I can think of a few ways to implement this, but I'm not happy with any of them:

  • I can add noise offline and pass the noisy and original samples to NetTrain as input and target, respectively. But then exactly the same noise trained in every round, which probably leads to overfitting.
  • I can pass noisy samples to NetTrain as above, but only train a few rounds, then restart with different noisy samples. (i.e. call NetTrain in a loop). But that seems to break a lot of the internal learning-rate heuristics in NetTrain (e.g. the momentum term is reset in every cycle)
  • I can link together a constant input, a several dropout layers and a TotalLayer to get a poor man's Gaussian noise approximation, (each DropoutLayer produces a Bernoulli random variable per pixel, and the sum of a number of Bernoulli random variables is approximately Gaussian). This is my best idea so far, but I need a lot of DropoutLayers to get close to a continuous distribution, and it's not very efficient to include an input that's always constant
  • $\begingroup$ May be you can use "$AugmentationFunction" -> f for image NetEncoder... you can check the answers from this link as an idea but I don't think it's exactly what you need... mathematica.stackexchange.com/questions/126120/… $\endgroup$
    – s.s.o
    Dec 8, 2016 at 21:17
  • $\begingroup$ @s.s.o: If this is called repeatedly for every round, it would be perfect, but I'll have to come up with some way to test this... Thanks for the link, that question is very similar and it's good to hear this will be addressed in 11.1 $\endgroup$ Dec 9, 2016 at 7:28
  • $\begingroup$ If I understand correctly you want every round different noise... In fact NN can accept file names (in order to keep the original images) you can add the argumentation function to this file name get random noised images and feed it to NN. Similar to Alexey Golyshev multi round training with Do[ ... {numberOfRounds}] you can do multi stage training. Not best solution but may help. $\endgroup$
    – s.s.o
    Dec 9, 2016 at 12:04
  • 4
    $\begingroup$ Has anyone figured out the most efficient way to do this? I tried using the new generator function in 11.1, which allows NetTrain to create each batch of training data according to any given function. However, this slows down the training by an order of magnitude. $\endgroup$
    – dan7geo
    Apr 3, 2017 at 10:17

4 Answers 4


Use NeuralNetworks`NoiseLayer

noisy = NetGraph[{
    "noise" -> NoiseLayer[{3, 90, 160}, "Distribution" -> NormalDistribution[0, 0.1]],
    "add" -> ThreadingLayer[Plus]
    {{"noise", NetPort["Input"]} -> "add"},
    "Input" -> NetEncoder[{"Image", {160, 90}}],
    "Output" -> NetDecoder["Image"]

You can find more undocumented definitions with


  • $\begingroup$ Hi, it seems like NoiseLayer is outdated. Do you know how to add the random noise for Mathematica v 11.3? $\endgroup$ Jul 14, 2020 at 9:55

For what it's worth, this is my "add binomial random variate layer":

Needs["NeuralNetworks`"] (* for ScalarTimesLayer, ScalarPlusLayer *)
With[{σ = 0.1, n = 5},
 addNoiseLayer = NetGraph[
     TotalLayer[], ScalarTimesLayer[σ*1./5], 
     ScalarPlusLayer[-1*σ], TotalLayer[],
     Table[DropoutLayer[], {n}]
     1 -> 2 -> 3 -> 4,
     NetPort["Input"] -> 4,

     Table[NetPort["const"] -> 4 + i -> 1, {i, n}]
     }, "Input" -> inputDim, "const" -> inputDim]]

enter image description here

  • $\begingroup$ mma says "ScalarTimesLayer[0.02] is not a net function.",and environment is ""11.1.0 for Microsoft Windows (64-bit) (March 13, 2017)"" $\endgroup$
    – partida
    Aug 31, 2017 at 3:02
  • $\begingroup$ @partida: not tested, but I think you can use ElementwiseLayer[#*0.02 &] in version 11 $\endgroup$ Aug 31, 2017 at 6:34
  • $\begingroup$ yes,they do the same thing,thanks $\endgroup$
    – partida
    Aug 31, 2017 at 7:20

In Mathematica 12.1, DropoutLayer exposes his mask, so it's possible to train using a random binary mask, with DropoutLayer["OutputPorts" -> "BinaryMask"] (or DropoutLayer["OutputPorts" -> {"Output", "BinaryMask"}]) Documentation of DropoutLayer (new in 12.1)

In Mathematica 12.2, it will be possible to simulate many types of noises (including once where you give the parameters -like mean and variance- through outputs of other subparts of a NetGraph). The novelty is already documented in the current 12.2 builds under the name of RandomArrayLayer. Documentation of RandomArrayLayer (introduced 12.2) Application of RandomArrayLayer (introduced 12.2) Application of RandomArrayLayer (introduced 12.2)


You can also do this by using a generator function for generating network inputs during NetTrain. Here's a minimal (if somewhat nonsensical) example:

net = NetInitialize @ NetGraph[
    "chain" -> NetChain[{5, Ramp, 1}],
    "plus" -> ThreadingLayer[Plus]
    {NetPort["Input"], NetPort["Noise"]} -> "plus" -> "chain"
   "Input" -> {1}

data = List /@ N[Range[10]];
genFun = Function[
   With[{dat = RandomChoice[data, #BatchSize]},
     "Input" -> dat,
     "Output" -> dat,
     "Noise" -> RandomVariate[NormalDistribution[], {#BatchSize, 1}]

Test the generator:

genFun[<|"BatchSize" -> 5|>]

<|"Input" -> {{9.}, {3.}, {10.}, {5.}, {5.}}, "Output" -> {{9.}, {3.}, {10.}, {5.}, {5.}}, "Noise" -> {{0.605894}, {-1.57657}, {0.586584}, {-1.8153}, {0.443682}}|>

Train using the generator:

NetTrain[net, genFun]

Each round, genFun will be evaluated to create a new batch of training examples, so each round you'll get fresh random numbers.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.