Bug in EmbeddingLayer?

Question

cnn = NetChain[
   {
    EmbeddingLayer[16, 100],
    ConvolutionLayer[64, 3, "Interleaving" -> True],
    AggregationLayer[Mean, 1],
    LinearLayer[2],
    SoftmaxLayer[]
    },
   "Input" -> 10
   ] // NetInitialize

Export["Python\\cnn.json", cnn, "MXNet"]

import mxnet as mx
import numpy as np

cnn = mx.gluon.SymbolBlock.imports('cnn.json', ['Input'], 'cnn.params')

cnn(mx.nd.array([[1,2,3,4,5,6,7,8,9,10]]))

[[0.52003264 0.47996736]]

cnn@Range[1, 10]

{0.520033,0.479967}

Mathematica embeds integers between 1 and n.

And MXNet embeds integers between 0 and n-1. Integers equal or greater than n are for unknown tokens.

cnn(mx.nd.array([[0,1,2,3,4,5,6,7,8,9]]))

[[0.49799052 0.50200945]]

cnn(mx.nd.array([[90,91,92,93,94,95,96,97,98,99]]))

[[0.48555845 0.51444155]]

cnn(mx.nd.array([[91,92,93,94,95,96,97,98,99,100]]))

[[0.4880071 0.51199293]]

cnn(mx.nd.array([[91,92,93,94,95,96,97,98,99,101]]))

[[0.4880071 0.51199293]]

No difference in output between 100 and 101 at the end.

And Mathematica has the same output as for unknown token at the end.

cnn@Range[91, 100]

{0.488007, 0.511993}

Answer 1

Hi from a developer of the Machine Learning team at Wolfram Research!

The behaviour you point out is the expected one. Let's take a look at how our EmbeddingLayer is represented in the exported mxnet model.

cnn = NetInitialize @ NetChain[
   {
    EmbeddingLayer[16, 100],
    ConvolutionLayer[64, 3, "Interleaving" -> True],
    AggregationLayer[Mean, 1],
    LinearLayer[2],
    SoftmaxLayer[]
    },
   "Input" -> 10
   ];
Export["cnn.json", cnn, "MXNet"];

It is associated with nodes {0, 1, 2, 3} in the graph below:

You can hover over the graph nodes to inspect their details, or import as a dataset (only displaying the relevant nodes for simplicity):

You can see that the input is fed to a PlusScalar node subtracting one, thus converting our 1-indexing convention to the 0-indexing used by mxnet. Since this conversion is made inside the mxnet symbol it is carried over in python, so you will get integers between 1 and n there as well. In general, our mxnet exporter always produces the exact same mxnet symbol we run in the backend, without making any attempt of translating between conventions.

As for the out-of-bounds inputs, mxnet behaviour is properly documented here: https://mxnet.incubator.apache.org/api/python/symbol/symbol.html#mxnet.symbol.Embedding

By default, if any index mentioned is too large, it is replaced by the index that addresses the last vector in an embedding matrix.

This is not true for our framework instead:

In[27]:= cnn[{91, 92, 93, 94, 95, 96, 97, 98, 99, 101}]

During evaluation of In[27]:= NetChain::invindata3: Data supplied to port "Input" could not be encoded; input is not a length-10 vector of integers between 1 and 100.

Out[27]= $Failed

Again, in this case we don't follow mxnet convention and just fail with an appropriate message. We have our own, completely separated input validation and error reporting system. In above example, the system is rejecting the input and the computation returns with a $Failed before even reaching mxnet.

Example

emb = NetChain[{EmbeddingLayer[2, 3, "Weights" -> Table[{i, i}, {i, 1, 3}]]}, "Input" -> 1]

Export["Python\\emb.json", emb, "MXNet"]

import mxnet as mx

emb = mx.gluon.SymbolBlock.imports('emb.json', ['Input'], 'emb.params')

emb(mx.nd.array([[0,1,2,3,4]]))

[[[1. 1.] [1. 1.] [2. 2.] [3. 3.] [3. 3.]]]