1
$\begingroup$

NetEncoder[{"Tokens"}] uses WordData indices to tokenize an input string. But is there a way to tell NetEncode to use a specific model instead, e.g. glove or word2vec?

$\endgroup$
2
$\begingroup$

You should use GloVe as the weights in EmbeddingLayer, not in the NetEncoder.

GloVe = {{1, 2}, {3, 4}, {0, 0}};

net = NetChain[{
   EmbeddingLayer["Weights" -> GloVe],
   LongShortTermMemoryLayer[10],
   SequenceLastLayer[],
   2,
   SoftmaxLayer[]
   },
  "Input" -> NetEncoder[{"Tokens", {"a", "b"}}],
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]

If you don't want to fine-tune GloVe with your data, use LearningRateMultipliers.

data = {"a a" -> 0, "b b" -> 1, "a b" -> 0, "b a" -> 1, "a b c" -> 0};

netT = NetTrain[net, data]

NetExtract[netT, {1, "Weights"}]

{{0.652085, 2.91778}, {3.59744, 3.59821}, {0.252089, -0.200977}}

netT = NetTrain[net, data, LearningRateMultipliers -> {1 -> 0, _ -> 1}]

NetExtract[netT, {1, "Weights"}]

{{1., 2.}, {3., 4.}, {0., 0.}}

Note that we have 2 tokens in NetEncoder and 3 vectors in GloVe. It's because:

enter image description here

You should make sure that your pre-trained GloVe has size d+1 or you can append zero vector of according size.

Example with the real GloVe

https://resources.wolframcloud.com/NeuralNetRepository/resources/GloVe-50-Dimensional-Word-Vectors-Trained-on-Tweets

https://nlp.stanford.edu/projects/glove/

GloVe = NetModel["GloVe 50-Dimensional Word Vectors Trained on Tweets"]

enter image description here

We got EmbeddingLayer with GloVe as the weights and NetEncoder on the input. This neural networks are the same:

net1 = NetChain[{
   GloVe,
   LongShortTermMemoryLayer[10],
   SequenceLastLayer[],
   2,
   SoftmaxLayer[]
   },
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]

net2 = NetChain[{
   EmbeddingLayer["Weights" -> NetExtract[GloVe, "Weights"]],
   LongShortTermMemoryLayer[10],
   SequenceLastLayer[],
   2,
   SoftmaxLayer[]
   },
  "Input" -> NetExtract[GloVe, "Input"],
  "Output" -> NetDecoder[{"Class", {0, 1}}]
  ]
$\endgroup$
  • $\begingroup$ Can you show how to use the real gloVe embedding dictionary from WordData? $\endgroup$ – M.R. Apr 20 '18 at 15:38
  • $\begingroup$ @M.R. See my edits. $\endgroup$ – Alexey Golyshev Apr 20 '18 at 16:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.