In many Sequence to Sequence machine learning tasks, an Attention Mechanism is incorporated.
Before Transformer Networks, introduced in the paper: Attention Is All You Need, mainly RNNs were used to model seq2seq models.
An attention score was generated along with a new hidden state (this blog illustrates the process very well). So, basically, the attention scores were dependent both on time and on an input sequence element. I am interested in whether it is possible to model this relationship by Mathematica LongShortTermMemoryLayer and AttentionLayer.
More specifically, I am interested in recreating the famous handwritten synthesis network by Alex Graves.
As you can see from the architecture, after the first LSTM layer,
there is an Attention mechanism (the original name mentioned in the paper is "local window layer"): which for each timepoint and character models the belief of writing that character. So, is it possible to model the "local window layer " by an AttentionLayer?
And if yes, then how it should be combined with an LSTM layer, what
should be the keys, values, and queries in this case?
I would be grateful for any help.
