15
$\begingroup$

A common "trick" in neural network training is to train an autoencoder using a "degraded" sample as input and the original sample as target output. The layer type DropoutLayer[] is made for this purpose. But I would like to play with different types of noise, e.g. Gaussian noise, salt&pepper noise, to train a denoising autoencoder.

I can think of a few ways to implement this, but I'm not happy with any of them:

  • I can add noise offline and pass the noisy and original samples to NetTrain as input and target, respectively. But then exactly the same noise trained in every round, which probably leads to overfitting.
  • I can pass noisy samples to NetTrain as above, but only train a few rounds, then restart with different noisy samples. (i.e. call NetTrain in a loop). But that seems to break a lot of the internal learning-rate heuristics in NetTrain (e.g. the momentum term is reset in every cycle)
  • I can link together a constant input, a several dropout layers and a TotalLayer to get a poor man's Gaussian noise approximation, (each DropoutLayer produces a Bernoulli random variable per pixel, and the sum of a number of Bernoulli random variables is approximately Gaussian). This is my best idea so far, but I need a lot of DropoutLayers to get close to a continuous distribution, and it's not very efficient to include an input that's always constant
$\endgroup$
  • $\begingroup$ May be you can use "$AugmentationFunction" -> f for image NetEncoder... you can check the answers from this link as an idea but I don't think it's exactly what you need... mathematica.stackexchange.com/questions/126120/… $\endgroup$ – s.s.o Dec 8 '16 at 21:17
  • $\begingroup$ @s.s.o: If this is called repeatedly for every round, it would be perfect, but I'll have to come up with some way to test this... Thanks for the link, that question is very similar and it's good to hear this will be addressed in 11.1 $\endgroup$ – Niki Estner Dec 9 '16 at 7:28
  • $\begingroup$ If I understand correctly you want every round different noise... In fact NN can accept file names (in order to keep the original images) you can add the argumentation function to this file name get random noised images and feed it to NN. Similar to Alexey Golyshev multi round training with Do[ ... {numberOfRounds}] you can do multi stage training. Not best solution but may help. $\endgroup$ – s.s.o Dec 9 '16 at 12:04
  • 3
    $\begingroup$ Has anyone figured out the most efficient way to do this? I tried using the new generator function in 11.1, which allows NetTrain to create each batch of training data according to any given function. However, this slows down the training by an order of magnitude. $\endgroup$ – dan7geo Apr 3 '17 at 10:17
4
$\begingroup$

Use NeuralNetworks`NoiseLayer

<<NeuralNetworks`
noisy = NetGraph[{
    "noise" -> NoiseLayer[{3, 90, 160}, "Distribution" -> NormalDistribution[0, 0.1]],
    "add" -> ThreadingLayer[Plus]
},
    {{"noise", NetPort["Input"]} -> "add"},
    "Input" -> NetEncoder[{"Image", {160, 90}}],
    "Output" -> NetDecoder["Image"]
]

You can find more undocumented definitions with

NeuralNetworks`$LayerData["Noise"]//Dataset

$\endgroup$
10
$\begingroup$

For what it's worth, this is my "add binomial random variate layer":

Needs["NeuralNetworks`"] (* for ScalarTimesLayer, ScalarPlusLayer *)
With[{σ = 0.1, n = 5},
 addNoiseLayer = NetGraph[
   Flatten@{
     TotalLayer[], ScalarTimesLayer[σ*1./5], 
     ScalarPlusLayer[-1*σ], TotalLayer[],
     Table[DropoutLayer[], {n}]
     },
   Flatten@{
     1 -> 2 -> 3 -> 4,
     NetPort["Input"] -> 4,

     Table[NetPort["const"] -> 4 + i -> 1, {i, n}]
     }, "Input" -> inputDim, "const" -> inputDim]]

enter image description here

$\endgroup$
  • $\begingroup$ mma says "ScalarTimesLayer[0.02] is not a net function.",and environment is ""11.1.0 for Microsoft Windows (64-bit) (March 13, 2017)"" $\endgroup$ – partida Aug 31 '17 at 3:02
  • $\begingroup$ @partida: not tested, but I think you can use ElementwiseLayer[#*0.02 &] in version 11 $\endgroup$ – Niki Estner Aug 31 '17 at 6:34
  • $\begingroup$ yes,they do the same thing,thanks $\endgroup$ – partida Aug 31 '17 at 7:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.