Neural machine translation model using deep learning

Question

In Mathematica version 11.1, SequenceAttentionLayer is introduced to model attentions in the recurrent neural network. It has been shown that attention mechanism can greatly improve the quality of a translation. It seems that we now have all the building blocks (RNN, attention,etc.) for an attention-based neural machine translation. So can we build a simple example of that using the neural network framework in Mathematica?

Answer 1

Its definitely possible, and I want to add an example of doing this to the 11.2 docs. In the meantime, I can give some pointers. First, I'll assume you are doing machine translation via the sequence-to-sequence approach ("Sequence to Sequence Learning with Neural Networks", Sutskever et al), which uses special tokens to indicate the beginning and end of a sentence (without these tokens, there is no way to know the length of the predicted target sequence, as its length is not known from the input sequence). There is a complete example of this in the docs (NetTrain -> Applications -> Integer Addition -> last example in this subsection). This shows exactly how to do the training with teacher forcing and special end tokens, and the example can be reused for machine translation.

The example didn't use attention. Here is how to build the full teacher-forcing net with attention. First, start with the encoder that encodes the source language to a vector (needs to return both the vector representation, and the states to feed into the attention layer):

encoderNetAttend = 
 NetGraph[{UnitVectorLayer[], GatedRecurrentLayer[150], 
   SequenceLastLayer[]}, {1 -> 2 -> 3, 2 -> NetPort["States"]}]

Then define the decoder (takes the vector representation of the sentence and produces a sequence:

decoderNet = NetGraph[{
   UnitVectorLayer[11],
   SequenceMostLayer[],
   GatedRecurrentLayer[150]},
  {NetPort["Input"] -> 1 -> 2 -> 3,
   NetPort["State"] -> NetPort[3, "State"]}
  ]

Finally, define the full teacher-forcing training net:

trainingNetAttention = NetGraph[<|
   "encoder" -> encoderNetAttend,
   "decoder" -> decoderNet,
   "attention" -> SequenceAttentionLayer[],
   "catenate" -> CatenateLayer[2],
   "classifier" -> {NetMapOperator[LinearLayer[]],
     SoftmaxLayer[]},
   "loss" -> CrossEntropyLossLayer["Index"],
   "rest" -> SequenceRestLayer[]|>,
  {NetPort["Input"] -> "encoder", 
   NetPort["encoder", "Output"] -> NetPort["decoder", "State"],
   NetPort["encoder", "States"] -> NetPort["attention", "Input"],
   NetPort["Target"] -> NetPort["decoder", "Input"],
   "decoder" -> NetPort["attention", "Query"],
   {"decoder", "attention"} -> 
    "catenate" -> "classifier" -> NetPort["loss", "Input"],
   NetPort["Target"] -> "rest" -> NetPort["loss", "Target"]}]

You will also need to attach appropriate Encoders for the languages you wish to train on (or preprocess the text yourself).