Specify NetEncode's tokenization model?

Question

NetEncoder[{"Tokens"}] uses WordData indices to tokenize an input string. But is there a way to tell NetEncode to use a specific model instead, e.g. glove or word2vec?

Answer 1

You should use GloVe as the weights in EmbeddingLayer, not in the NetEncoder.

GloVe = {{1, 2}, {3, 4}, {0, 0}};

net = NetChain[{
   EmbeddingLayer["Weights" -> GloVe],
   LongShortTermMemoryLayer[10],
   SequenceLastLayer[],
   2,
   SoftmaxLayer[]
   },
  "Input" -> NetEncoder[{"Tokens", {"a", "b"}}],
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]

If you don't want to fine-tune GloVe with your data, use LearningRateMultipliers.

data = {"a a" -> 0, "b b" -> 1, "a b" -> 0, "b a" -> 1, "a b c" -> 0};

netT = NetTrain[net, data]

NetExtract[netT, {1, "Weights"}]

{{0.652085, 2.91778}, {3.59744, 3.59821}, {0.252089, -0.200977}}

netT = NetTrain[net, data, LearningRateMultipliers -> {1 -> 0, _ -> 1}]

NetExtract[netT, {1, "Weights"}]

{{1., 2.}, {3., 4.}, {0., 0.}}

Note that we have 2 tokens in NetEncoder and 3 vectors in GloVe. It's because:

You should make sure that your pre-trained GloVe has size d+1 or you can append zero vector of according size.

Example with the real GloVe

https://resources.wolframcloud.com/NeuralNetRepository/resources/GloVe-50-Dimensional-Word-Vectors-Trained-on-Tweets

https://nlp.stanford.edu/projects/glove/

GloVe = NetModel["GloVe 50-Dimensional Word Vectors Trained on Tweets"]

We got EmbeddingLayer with GloVe as the weights and NetEncoder on the input. This neural networks are the same:

net1 = NetChain[{
   GloVe,
   LongShortTermMemoryLayer[10],
   SequenceLastLayer[],
   2,
   SoftmaxLayer[]
   },
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]

net2 = NetChain[{
   EmbeddingLayer["Weights" -> NetExtract[GloVe, "Weights"]],
   LongShortTermMemoryLayer[10],
   SequenceLastLayer[],
   2,
   SoftmaxLayer[]
   },
  "Input" -> NetExtract[GloVe, "Input"],
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]