EmbeddingLayer\[size, n\] trains a representation of a list of integers between 1 and n into a real vector of dimension size. However, it is a common practice to represent stop characters as 0; for example the default setting in Keras's embedding treats zeros as padding.
Mathematica's EmbeddingLayer supports an option "Input"->Varying that would seem to support this, but it doesn't play well with LinearLayer. Consider the following example (in Mathematica 12.3.1):
(*some examples*)
good = {5, 4, 3, 1};
bad = {5, 4, 3, 0};
ugly = {5, 4, 3};
(*suppose you specify the input dimension...only the first example will work*)
em = NetInitialize@NetChain[{EmbeddingLayer[2, 5, "Input" -> 4], LinearLayer[1]}];
FailureQ[em[#]] & /@ {good, bad, ugly} (* {False, True, True} *)
(*suppose you specify "Input"->"Varying", all examples fail*)
em = NetInitialize@NetChain[{EmbeddingLayer[2, 5, "Input"->"Varying"],LinearLayer[1]}];
FailureQ[em[#]] & /@ {good, bad, ugly}
(*with error: NetInitialize: Validation failed for second layer of net (LinearLayer):
input port of layer has dimensions RowWithSeparators[" ","\" \"","n","2"],
but dynamic dimensions are not currently supported by LinearLayer "*)
Question: What's the correct way to use EmbeddingLayer for varying length input (with or without zero padding)?
FWIW, MXNet uses zero indexing (as discussed previously on Mathematica stack exchange), and the MXNet mx.nd.Embedding documentation doesn't seem to describe anything like Keras' mask_zero option. So this may be a limitation in the underlying MXNet engine, rather than in Mathematica per se.
EmbeddingLayerwill cause an output with variable length, which meansLinearLayerwill have to have a variable size weight array which I don't think it is nor should be possible. One possible solution could be setting a maximal input length toEmbeddingLayer, padding the variable input with explicit net structure before theEmbeddingLayer. $\endgroup$