Layers
BatchNormalizationLayer
There are several layers introduced in v.11 that can not be used uninitialized, this is one of them. Input must be either a rank 1 or rank 3 tensor. To be honest, I do not think we can see its true effect on one input as demonstrated by these two examples below. I believe the effect of Batch normalization is only implemented by the function NetTrain. If someone has a simple example of this please let me know and I will update this.
batchNet = NetInitialize[BatchNormalizationLayer["Input"->{1}]];
ListLinePlot[Flatten[Table[{i - batchNet[{i}]}, {i, -20, 50}]]]
tensor334 = {{{1, 2, 10, 20}, {3, 4, 30, 40}, {5, 6, 50, 60}}, {{5, 6,
50, 60}, {7, 8, 70, 80}, {9, 10, 90, 100}}, {{1000, 10000, 1,
5}, {500, 5000, 12, 215}, {21312, 325, 6234, 412}}};
batchNet = NetInitialize[BatchNormalizationLayer["Input" -> {3, 3, 4}]];
batchNet[tensor]
CatenateLayer
The CatenateLayer is pretty simple. It is the Net akin to Flatten[] (if you appended two lists together). If you aren't exactly impressed by this yet, it has more intresting abilities when it comes to network architecture (see example 2). Because Mathematica designed each Net's nodes / neurons / "ports" to recieve only one input with the exception of loss layers that take an association of two rules, it makes it unintuitive how to project two seperate nodes to a third different node. CatenateLayer (and others like it) will allow for this. Running example 2 in your notebook will show how the typically linear NetGraphs (in most Mathematica's examples) can become quite more intricate. To demonstrate how intricate it can become (should you have the patience to code it) see example 3.
Correction above from Sebastian
JHM gives a clear distinction about the purposes of the uses of CatenateLayer and FlattenLayer. See FlattenLayer.
Example 1
a = {{{1},{2},{3}}};
a//Dimensions
b = CatenateLayer[][a]
b//Dimensions
c = CatenateLayer[][b]
c//Dimensions
Example 2
NetGraph[{BatchNormalizationLayer[], Tanh, LogisticSigmoid, Tanh,TotalLayer[], TotalLayer[], TotalLayer[], CatenateLayer[],DotPlusLayer[50]}, {1 -> 2, 1 -> 3, 1 -> 4, 2 -> 5, 3 -> 5, 3 -> 6, 4 -> 6, 2 -> 7, 4 -> 7, 5 -> 8, 6 -> 8, 7 -> 8, 8 -> 9}, "Input" -> 2]
Example 3
NetInitialize[
NetGraph[{BatchNormalizationLayer[], Tanh, LogisticSigmoid, Tanh,
TotalLayer[], TotalLayer[], TotalLayer[], CatenateLayer[],
DotPlusLayer[50], BatchNormalizationLayer[], Tanh, LogisticSigmoid,
Tanh, TotalLayer[], TotalLayer[], TotalLayer[], CatenateLayer[],
DotPlusLayer[50], DropoutLayer[], DropoutLayer[], TotalLayer[],
LogisticSigmoid, BatchNormalizationLayer[], Tanh, LogisticSigmoid,
Tanh, TotalLayer[], TotalLayer[], TotalLayer[], CatenateLayer[],
DotPlusLayer[50], DropoutLayer[], TotalLayer[], TotalLayer[],
TotalLayer[], TotalLayer[], TotalLayer[], TotalLayer[],
DotPlusLayer[50], DotPlusLayer[1]}, {1 -> 2, 1 -> 3, 1 -> 4,
2 -> 5, 3 -> 5, 3 -> 6, 4 -> 6, 2 -> 7, 4 -> 7, 5 -> 8, 6 -> 8,
7 -> 8, 8 -> 9, 10 -> 11, 10 -> 12, 10 -> 13, 11 -> 14, 12 -> 14,
11 -> 15, 13 -> 15, 13 -> 16, 12 -> 16, 16 -> 17, 15 -> 17,
14 -> 17, 17 -> 18, 18 -> 19, 9 -> 20, 20 -> 21, 19 -> 21,
21 -> 22, 23 -> 24, 23 -> 25, 23 -> 26, 24 -> 27, 25 -> 27,
24 -> 28, 25 -> 29, 26 -> 28, 26 -> 29, 27 -> 30, 28 -> 30,
29 -> 30, 30 -> 31, 31 -> 32, 32 -> 21, 30 -> 33, 8 -> 33, 8 -> 34,
17 -> 34, 30 -> 35, 17 -> 35, 33 -> 36, 34 -> 36, 34 -> 37,
35 -> 37, 37 -> 38, 36 -> 38, 38 -> 39, 39 -> 21, 22 -> 40},
"Input" -> 85]]
CrossEntropyLossLayer
Depending on your familiarity with information theory this layer may or may not make much sense to you. I recommend Information Theory: a tutorial introduction if you are new to this concept and want to learn more (PDF download from the author's ResearchGate account).
For a brief and unformal description, Entropy (information) is defined somewhat backwards to most people's intuition e.g. unlike probability where if we are certain of an event occuring we give it the value 1, here if we know something we give it the value 0. Why? Because if we know something happens / will happen, then if that event occurs we do not gain any extra knowledge. So along those lines you can think of Entropy as the surprise or amount of information we gain if something happen.
For example, this gives an output 0
CrossEntropyLossLayer[][<|"Input" -> {1}, "Target" -> 1|>]
And this does not. If we increase the input compared to the target, you can see that the value's magnitude increases.
CrossEntropyLossLayer[][<|"Input" -> {2}, "Target" -> 1|>]
CrossEntropyLossLayer[][<|"Input" -> {20}, "Target" -> 1|>]
CrossEntropyLossLayer[][<|"Input" -> {200}, "Target" -> 1|>]
ListLinePlot[Table[{i, CrossEntropyLossLayer[][<|"Input" -> {i}, "Target" -> 1|>]}, {i, 0,100}]]
Here we have been apply our data to the index of the target class. There is also the ability to use the option "Probabilities" to pass your data to a vector of class probabilities.
ConvolutionLayer
The convolution layer is similar to the padding layer, with the exception of being able to specify the number of outputs.
This layer, unlike others which can either take an arbitrary rank tensor or a rank 1 or rank 3 tesnors, can only take a rank three numerical tensor.
The example below shows how different kernel sizes of {h,w} affect this {3,3,3} tensor input. Here the output channels are limited to 1 for clarity. The first list in this output is the dimensions of the output, followed by the output.
If it seems too confusing, in short, lets say you provide a tensor with dimensions {a,b,c} to ConvolutionLayer[n, {h,w}], the resulting output would be (most likely$*$) a tensor with dimensions {n,b-h+1,c-w+1}. It should be clear that your kernel can't be larger than second and third dimensions of your input tensor.
$*$ we will talk about this formula in the poolying layer.
Table[{Dimensions[
NetInitialize[ConvolutionLayer[1, {i, j}, "Input" -> {3, 3, 3}]][
{
{{1, 2, 3}, {3, 2, 1}, {7, 8, 9}},
{{4, 5, 6}, {6, 5, 4}, {1, 2, 3}},
{{7, 8, 9}, {9, 8, 7}, {4, 5, 6}}
}
]], NetInitialize[
ConvolutionLayer[1, {i, j}, "Input" -> {3, 3, 3}]][
{
{{1, 2, 3}, {3, 2, 1}, {7, 8, 9}},
{{4, 5, 6}, {6, 5, 4}, {1, 2, 3}},
{{7, 8, 9}, {9, 8, 7}, {4, 5, 6}}
}
]}, {i, 1, 3}, {j, 1, 3}] // MatrixForm
DeconvolutionLayer
It basically "undoes" the ConvolutionLayer. However do not be mistaken, if you feed the output of convolution to deconvolution you will not recieved the same result.
a={
{{1, 2, 3}, {3, 2, 1}, {7, 8, 9}},
{{4, 5, 6}, {6, 5, 4}, {1, 2, 3}},
{{7, 8, 9}, {9, 8, 7}, {4, 5, 6}}
};
a//Dimensions
b=NetInitialize[ConvolutionLayer[1, {1, 1}, "Input" -> {3, 3, 3}]][a];
b//Dimensions
c=NetInitialize[DeconvolutionLayer[3, {1, 1}, "Input" -> {1, 3, 3}]][b];
c//Dimensions
DropoutLayer
This is a pretty important layer (in my opinion), it is similar to a drop out method used my neural network enthusiasts to make neural networks more akin to other ensemble methods like random forest.
In essence it sets it takes one argument, p, which is the probability that the its input elements are set to zero during training, and increases the remainder by $\frac{1}{p}$. This is similar to the BatchNormalizationLayer I guess... in the sense that you can not see its effect without training. e.g.
DropoutLayer[.5][Range[-3, 3]]
Will give you:
Range[-3, 3]
Unless you are using NetTrain. So this makes trying to adapt this for other purposes a bit more tricky. If you know of a way to invoke this without NetTrain, please let me know.
EmbeddingLayer
This is the one of several net layers (including ConvolutionLayer, DotPlusLayer,etc) that I am aware of, that the documentation specifically calls "trainable" (i.e. parameters can be modified during training). It too must be initalized before use. It has two arguments, n and size. It takes integers in [1,n] and puts them into a vector of length size.
NetInitialize[EmbeddingLayer[2, 3]][{1}]
NetInitialize[EmbeddingLayer[5, 3]][{1,3,2}]
There exists other ways to get this functionality using the other layers. I think this layer really ownly exists because of classes. The documentation gives the following example:
NetInitialize[EmbeddingLayer[2, 3, "Input" -> NetEncoder[{"Class",{True, False}}]]]
This to me is odd, as NetEncoder has a specific option for Booleans; However that would not work here as the first argument n, can not be zero (by definition).
From Sebastian
Suppose you are trying to create a vector representation of a very high-dimensional categorical input (like words). NetEncoder will produce one-hot encoded vectors of the same dimension as the number of categories. This can be absolutely massive, and make training impossible. EmbeddingLayer solves this: it maps integers directly to a low-dimensional vector subspace, and this embedding is trained. This allows things like Word2Vec to be implemented (for example). The docs should make this use-case clearer.
FlattenLayer
I forgot this in my original post. Thank you JHM for providing this.
Catenate[{{1, 2, 3}, {4, 5, 6}}]
(* {1, 2, 3, 4, 5, 6) *)
CatenateLayer[][{{1, 2, 3}, {4, 5, 6}}]
(* {1, 2, 3, 4, 5, 6} *)
CatenateLayer[][{{{1, 2}, {3, 4}}, {{5, 6}, {7, 8}}}]
(* {{1, 2}, {3, 4}, {5, 6}, {7, 8}} *)
Flatten[{{1, 2, 3}, {4, 5, 6}}]
(* {1, 2, 3, 4, 5, 6} *)
FlattenLayer[][{{1, 2, 3}, {4, 5, 6}}]
(* {1., 2., 3., 4., 5., 6.} *)
FlattenLayer[][{{{1, 2}, {3, 4}}, {{5, 6}, {7, 8}}}]
(* {1., 2., 3., 4., 5., 6., 7., 8.} *)
CatenateLayer and FlattenLayer have different usage. If you want to connect your lists (i.e. flatten only the outermost level), then CatenateLayer would be the correct choice. If you want to make your rank-n tensor into a vector, then FlattenLayer would be appropriate.
Note: FlattenLayer automatically converts Integers to Reals. I do not know whether that is intended.
MeanAbsoluteLossLayer
This layer does exactly what you think it would, tehrefore I am only putting two lines of code which should make it pretty aparent.
MeanAbsoluteLossLayer[][<|"Input" -> {1, 1, 1, 4}, "Target" -> {1, 1, 1, 4}|>]
MeanAbsoluteLossLayer[][<|"Input" -> {1, 1, 1, 4}, "Target" -> {1.1, 0.9, 1, 4}|>]
If one specifies the Input to be an integer n, e.g.
MeanAbsoluteLossLayer["Input"->n];
The one can cycle over a tensor where the inner most layer is of length n.
e.g.
MeanAbsoluteLossLayer["Input"->3][<|"Input"->{{1,1,1},{1,.9,1.1}},"Target"->{{1,1,1},{1,1,1}|>];
MeanSquaredLossLayer
Similar to that of above, but, you know, squares first...
MeanSquaredLossLayer[][<|"Input" -> {1, 1, 1, 4}, "Target" -> {1, 1, 1, 4}|>]
MeanSquaredLossLayer[][<|"Input" -> {1, 1, 1, 4}, "Target" -> {1.1, 0.9, 1, 4}|>]
PoolingLayer
This is similar to ConvolutionLayer, at least it takes the same inputs, the difference being on the function of the kernel. You can specify the kernel to use the functions Max, Mean or Total. How this works will be clear in the example.
As promised, however, lets talk about the tranformation of a tensor with dimensions {a,b,c} via this function. Foremost, unlike ConvolutionLayer, you can not change the output, i.e. the first dimension of the output will always be the same as a Dimension[output] will yield {a,x,y}. So what are the values of x and y? Let us assume you do not mess with the PaddingSize or Stride options. Then x will be the minimial value of either x-k or x-1, where k is the kernel size {h}. Similarly, y would be the minimum of either y-k or y-1, where kernel size is {w}.
If you want to mess with PaddingSize (how many zeros you add to your input) and Stride (the step size of your kernel), then the function becomes $Floor[\frac{Min[{x+2p-k+s-1,x+2p-1}]}{s}]$. Same for y.
PoolingLayer[{1, 2}][
{
{{1, 2, 3}, {3, 2, 1}, {7, 8, 9}},
{{4, 5, 6}, {6, 5, 4}, {1, 2, 3}},
{{7, 8, 9}, {9, 8, 7}, {4, 5, 6}}
}
]
PoolingLayer[{2, 2}][
{
{{1, 2, 3}, {3, 2, 1}, {7, 8, 9}},
{{4, 5, 6}, {6, 5, 4}, {1, 2, 3}},
{{7, 8, 9}, {9, 8, 7}, {4, 5, 6}}
}
]
PoolingLayer[{3, 2}][
{
{{1, 2, 3}, {3, 2, 1}, {7, 8, 9}},
{{4, 5, 6}, {6, 5, 4}, {1, 2, 3}},
{{7, 8, 9}, {9, 8, 7}, {4, 5, 6}}
}
]
As the default kernel function is Max, this output shouldn't be too hard to understand.
ReshapeLayer
This layer's name says it all. There are some limitations of course. Whatever input you try to reshape needs to be able to fit nicely in that shape. It will not automatically pad or return an untransformed input.
ReshapeLayer[{2, 2, 2}][Range[8]]
ReshapeLayer[{1, 8, 1}][Range[8]]
SummationLayer
This really doesn't need any explaination. It is basically the same as Total[], only you should probably specify your input size.
SummationLayer[]
SummationLayer["Input"->{3}][{1,2,3}]
Total[{1,2,3}]
From Sebastian
To be more precise, its more like Total[array, Infinity]. TotalLayer acts more like Total[{array1, array2, ...}]
TotalLayer
This is like list addition. It adds elementwise whatever lists you through at it
TotalLayer[][{{1,2,3},{1,2,3}}]
{1,2,3} + {1,2,3}
