# How to make a switch in Neural Networks?

In this paper,they design a network like this.

The left(green) is a normal network that mapping one feature to another feature,the right(red) side is an autoencoder(using RNN).But a switch is inserted.

During training the switch is set randomly for each network.

So the embedding vector can learn multiform information.

Using MNIST to make a example. In order to learn image features,I define this network(for simplify,only use DNN).

resource = ResourceObject["MNIST"];
trainingData = ResourceData[resource, "TestData"];
trainingData = {List @@ # -> First@#} & /@ trainingData;


NetGraph[{FlattenLayer[], 8, 28*28, ReshapeLayer[{1, 28, 28}],
UnitVectorLayer[10, "Input" -> "Integer"]}, {1 -> 2 -> 3 -> 4,NetPort["digit"] -> 5},
"Input" -> NetEncoder[{"Image", {28, 28}, ColorSpace -> "Grayscale"}],
"Output1" -> NetDecoder[{"Image", ColorSpace -> "Grayscale"}]]


But I want the net like this:

So the image features not only can learn information from image pixels,but only learn information about digits.

How to do it?

If I understood correctly, you want that the node 3 either sees the output of node 2 or of node 5 (that both have the same dimensions) depending on a randomly chosen "switch" value.

You could add the "switch" variable as an extra input (with the same dimensions as the output of 2/5), that you set randomly to 0 or 1 for every sample (e.g. sample generator function), and then use ThreadingLayer[#1*#2&] and ThreadingLayer[#1*(1-#2)&] to set either node 2 or 5 to 0, then add the results, like this (untested):

NetGraph[{net1, net2, ThreadingLayer[#1*#2 &],

• I don‘t know how ThreadingLayer[#1*(1 - #2) &] determine which is #1,which is #2.If #1 means switch,it will leads problem Commented Sep 1, 2017 at 11:11
• I think the function arguments are ordered like the inputs in the graph, i.e. for{1, 2} -> 3 #1 will be the output of node 1 and #2 will be the output of node 2 Commented Sep 1, 2017 at 11:35