9
$\begingroup$

How can I control the randomness of DropoutLayer in order to get the same result after each execution?

For example:

drop = DropoutLayer[];
drop[{1, 2, 3, 4, 5}, NetEvaluationMode -> "Train"]

As it is written under Possible Issues (in the Documentation Center) for DropoutLayer:

Currently, any randomness invoked by NetEvaluationMode->"Train" is not affected by SeedRandom and BlockRandom

Thanks!

$\endgroup$
7
$\begingroup$
net = NetGraph[
  {
   ThreadingLayer[Times]
   },
  {
   {NetPort["Input"], NetPort["Dropout"]} -> 1
   }
  ]

enter image description here

dropout[size_, p_Real: 0.5] := RandomChoice[{p, 1 - p} -> {0, 1}, size]

Table[net[<|"Input" -> Range[6], "Dropout" -> dropout[{6}]|>], {10}] // MatrixForm

enter image description here

Table[SeedRandom[0]; net[<|"Input" -> Range[6], "Dropout" -> dropout[{6}]|>], {10}] // MatrixForm

enter image description here

EXAMPLE

net = NetGraph[
  <|
   "dropout" -> NetGraph[{ThreadingLayer[Times]}, {{NetPort["Input"], NetPort["Dropout"]} -> 1}],
   "classify" -> {2, SoftmaxLayer[]}
   |>,
  {"dropout" -> "classify"},
  "Input" -> 3,
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]

enter image description here

SeedRandom[0];
n = 10;
data = Thread[RandomReal[{-1, 1}, {n, 3}] -> RandomInteger[{0, 1}, n]];

generator = Function[
   With[
    {x = RandomSample[data, #BatchSize]},
    <|
     "Input" -> x[[;; , 1]],
     "Output" -> x[[;; , 2]],
     "Dropout" -> dropout[{#BatchSize, 3(*same size as Input*)}]
     |>
    ]
   ];

SeedRandom[0];
netT = NetTrain[net, generator, MaxTrainingRounds -> 1, BatchSize -> 2]

enter image description here

For evaluation we need to replace custom dropout layer with DropoutLayer which has the same probability.

netT = RightComposition[
   NetExtract[#, "classify"] &,
   NetPrepend[#, DropoutLayer[0.5]] &,
   NetReplacePart[#, "Output" -> NetDecoder[{"Class", {0, 1}}]] &
   ]@netT

enter image description here

netT[{1, 2, 3}, "Probabilities"]

<|0 -> 0.0449557, 1 -> 0.955044|>

$\endgroup$
  • $\begingroup$ Thanks a lot Alexey! You define your own dropout layer in order to get the control of randomness, which is fine. But my question was specifically for the built-in Mathematica command DropoutLayer. Is it possible to fix the SeedRandom as it is done in TensorFlow and PyTorch (for example)? $\endgroup$ – demm Jan 7 at 8:21
  • 2
    $\begingroup$ @demm No, it's impossible. Will likely be fixed in Mathematica 12. $\endgroup$ – Alexey Golyshev Jan 7 at 8:55
  • $\begingroup$ Interestingly, within LSTMs, GNNs etc. the internal dropout option (Dropout, for the gate contributions) is fixed by setting a SeedRandom before NetInitialize. $\endgroup$ – demm Jan 8 at 10:25
4
$\begingroup$

If you find that there is no equivalent operation in NeuralNetwoks like TensorFlow or PyTorch, you should ask <<MXNetLink` for help.

drop=DropoutLayer[];
do:=(
    MXNetLink`MXSeedRandom[42];
    drop[Range@5,NetEvaluationMode->"Train"]
);
SameQ@Table[do,100]

do should return array 0 forever.


PS:

NeuralNetwoks is something like Keras || FastAI.

MXNet is at the same level as TensorFlow & PyTorch.

$\endgroup$
  • 1
    $\begingroup$ TargetDevice -> "CPU": {0., 0., 0., 0., 0.} TargetDevice -> "GPU": {2., 4., 6., 0., 0.} $\endgroup$ – Alexey Golyshev Jan 13 at 7:48
  • 2
    $\begingroup$ github.com/apache/incubator-mxnet/pull/12091 "...on the CPU and GPU the random number generator works slightly differently. On the CPU the numbers are generated in the range [0,1) and on the GPU they are (0,1]." $\endgroup$ – Alexey Golyshev Jan 13 at 7:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.