0
$\begingroup$

Could anyone elaborate on the new Neural Network layer in Mathematica Verison 12: https://reference.wolfram.com/language/ref/AttentionLayer.html

AttentionLayer

The documentation is rather sparse on this.

Any insight would be greatly appreciated.

Thank You

Michel

$\endgroup$
2
$\begingroup$
att = AttentionLayer["Dot"]

enter image description here

is the same as this:

net = NetGraph[
  {
   TransposeLayer[],
   DotLayer[],
   SoftmaxLayer[],
   DotLayer[]
   },
  {
   NetPort["Query"] -> 2,
   NetPort["Key"] -> 1 -> 2,
   2 -> 3,
   {3, NetPort["Value"]} -> 4
   }
  ]

enter image description here

SeedRandom[0];
input = RandomReal[{-1, 1}, {4, 2}];
query = RandomReal[{-1, 1}, {3, 2}];
att@<|"Key" -> input, "Value" -> input, "Query" -> query|>

{{0.0650637,0.358659},{0.130756,0.397385},{0.467243,0.533}}

net@<|"Key" -> input, "Value" -> input, "Query" -> query|>

{{0.0650637,0.358659},{0.130756,0.397385},{0.467243,0.533}}

Attention matrix:

SoftmaxLayer[]@(query.Transpose[input]) // TableForm

enter image description here

MatrixPlot[%, FrameLabel -> {{"Input", ""}, {"Query", ""}}]

enter image description here

Useful links:

How to visualize attention?

http://wac.36f4.edgecastcdn.net/0036F4/pub/www.wolfram.com/technology-conference/2018/Wednesday/2018-10-17_WTC_nn_framework.nb (slide 9)

http://www.wolfram.com/language/12/neural-network-framework/use-transformer-neural-nets.html

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.