11
$\begingroup$

How can I control the randomness of DropoutLayer in order to get the same result after each execution?

For example:

drop = DropoutLayer[];
drop[{1, 2, 3, 4, 5}, NetEvaluationMode -> "Train"]

As it is written under Possible Issues (in the Documentation Center) for DropoutLayer:

Currently, any randomness invoked by NetEvaluationMode->"Train" is not affected by SeedRandom and BlockRandom

Thanks!

$\endgroup$

2 Answers 2

7
$\begingroup$
net = NetGraph[
  {
   ThreadingLayer[Times]
   },
  {
   {NetPort["Input"], NetPort["Dropout"]} -> 1
   }
  ]

enter image description here

dropout[size_, p_Real: 0.5] := RandomChoice[{p, 1 - p} -> {0, 1}, size]

Table[net[<|"Input" -> Range[6], "Dropout" -> dropout[{6}]|>], {10}] // MatrixForm

enter image description here

Table[SeedRandom[0]; net[<|"Input" -> Range[6], "Dropout" -> dropout[{6}]|>], {10}] // MatrixForm

enter image description here

EXAMPLE

net = NetGraph[
  <|
   "dropout" -> NetGraph[{ThreadingLayer[Times]}, {{NetPort["Input"], NetPort["Dropout"]} -> 1}],
   "classify" -> {2, SoftmaxLayer[]}
   |>,
  {"dropout" -> "classify"},
  "Input" -> 3,
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]

enter image description here

SeedRandom[0];
n = 10;
data = Thread[RandomReal[{-1, 1}, {n, 3}] -> RandomInteger[{0, 1}, n]];

generator = Function[
   With[
    {x = RandomSample[data, #BatchSize]},
    <|
     "Input" -> x[[;; , 1]],
     "Output" -> x[[;; , 2]],
     "Dropout" -> dropout[{#BatchSize, 3(*same size as Input*)}]
     |>
    ]
   ];

SeedRandom[0];
netT = NetTrain[net, generator, MaxTrainingRounds -> 1, BatchSize -> 2]

enter image description here

For evaluation we need to replace custom dropout layer with DropoutLayer which has the same probability.

netT = RightComposition[
   NetExtract[#, "classify"] &,
   NetPrepend[#, DropoutLayer[0.5]] &,
   NetReplacePart[#, "Output" -> NetDecoder[{"Class", {0, 1}}]] &
   ]@netT

enter image description here

netT[{1, 2, 3}, "Probabilities"]

<|0 -> 0.0449557, 1 -> 0.955044|>

$\endgroup$
3
  • $\begingroup$ Thanks a lot Alexey! You define your own dropout layer in order to get the control of randomness, which is fine. But my question was specifically for the built-in Mathematica command DropoutLayer. Is it possible to fix the SeedRandom as it is done in TensorFlow and PyTorch (for example)? $\endgroup$
    – demm
    Commented Jan 7, 2019 at 8:21
  • 2
    $\begingroup$ @demm No, it's impossible. Will likely be fixed in Mathematica 12. $\endgroup$ Commented Jan 7, 2019 at 8:55
  • $\begingroup$ Interestingly, within LSTMs, GNNs etc. the internal dropout option (Dropout, for the gate contributions) is fixed by setting a SeedRandom before NetInitialize. $\endgroup$
    – demm
    Commented Jan 8, 2019 at 10:25
5
$\begingroup$

If you find that there is no equivalent operation in NeuralNetwoks like TensorFlow or PyTorch, you should ask <<MXNetLink` for help.

drop=DropoutLayer[];
do:=(
    MXNetLink`MXSeedRandom[42];
    drop[Range@5,NetEvaluationMode->"Train"]
);
SameQ@Table[do,100]

do should return array 0 forever.


PS:

NeuralNetwoks is something like Keras || FastAI.

MXNet is at the same level as TensorFlow & PyTorch.

$\endgroup$
2
  • 1
    $\begingroup$ TargetDevice -> "CPU": {0., 0., 0., 0., 0.} TargetDevice -> "GPU": {2., 4., 6., 0., 0.} $\endgroup$ Commented Jan 13, 2019 at 7:48
  • 5
    $\begingroup$ github.com/apache/incubator-mxnet/pull/12091 "...on the CPU and GPU the random number generator works slightly differently. On the CPU the numbers are generated in the range [0,1) and on the GPU they are (0,1]." $\endgroup$ Commented Jan 13, 2019 at 7:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.