Can `DropoutLayer` drop complete channels of a tensor?

Question

DropoutLayer randomly deactivates a fixed fraction of the outputs of the layer before. My understanding of dropout regularization is that instead of training a single network, we're training a statistical mixture of networks, where every combination of active/inactive neurons is an element of the mixture. For testing, we "average" over the whole mixture, and get better generalization.

So for a convolutional layer it would make sense to me to deactivate complete channels, instead of single pixels. This would also make training much faster, as inactive channels wouldn't have to be calculated.

Is there any way to achieve this with Mathematica? (Or alternatively, is there research the shows that applying dropout to full channels is a bad idea?)

Answer 1

One way to do this is by feeding a constant array (with length equal to the number of channels) to a dropout layer, then replicating its output to create a tensor with the correct dimensions and finally multiplying this tensor with the output of your convolution layer.

However, I'm not sure if this is going to make the training much faster since it will be doing a bunch of redundant multiplications with zeroes.

Example:

Code shown above:

conv = NetChain[{ConvolutionLayer[2, {2, 2}]}, "Input" -> {1, 4, 4}];

drop = NetChain[{ConstantArrayLayer["Array" -> ConstantArray[1, 2]], 
  DropoutLayer[], ReplicateLayer[{3, 3}], TransposeLayer[1 -> 3]}];

net = NetGraph[{conv, drop, ThreadingLayer[Times]}, {{1, 2} -> 3}] // NetInitialize;

net[RandomReal[1, {1, 4, 4}], NetEvaluationMode -> "Train"] // Column

Related research:

Dropping channels: Channel-Drop

Dropping weights: DropConnect