DropoutLayer
randomly deactivates a fixed fraction of the outputs of the layer before. My understanding of dropout regularization is that instead of training a single network, we're training a statistical mixture of networks, where every combination of active/inactive neurons is an element of the mixture. For testing, we "average" over the whole mixture, and get better generalization.
So for a convolutional layer it would make sense to me to deactivate complete channels, instead of single pixels. This would also make training much faster, as inactive channels wouldn't have to be calculated.
Is there any way to achieve this with Mathematica? (Or alternatively, is there research the shows that applying dropout to full channels is a bad idea?)
DropoutLayer[1]
orDropoutLayer[.9999]
not working for you? $\endgroup$DropoutLayer[1]
since in training mode, there is a division by (1-p) where p is the probability of dropout (= 1 in this case) which results in overflow. $\endgroup$