I have 2 nets, i.e. net1 (i.e. LinearLayer) and net2 (i.e. EmbeddingLayer) as below, while the input are just integers from 1 to 10:-

net1 = NetChain[{LinearLayer[2, "Input" -> 1]}];
net1 = BlockRandom[SeedRandom[10]; NetInitialize[net1]];
input = {#} & /@ Range[10];
output = net1[input];

enter image description here

net2 = NetChain[{EmbeddingLayer[2, 100]}];
net2 = BlockRandom[SeedRandom[10]; NetInitialize[net2]];
input = Range[10];
output = net2[input];

enter image description here

As you can see, net1 is linear, while net2 is obviously non-linear. But why net2 is not linear? What is the mathematics behind EmbeddingLayer? Is EmbeddingLayer also using the linear operation $y=A W + B$ or not?

Many thanks!

  • 1
    $\begingroup$ My guess is that it encodes the integers with one-hot encoding and given random initial weight matrix it should behave like this. $\endgroup$ – swish Jul 28 '18 at 18:42
  • $\begingroup$ Have a look in the 'Properties and Relations' section of EmbeddingLayer and LinearLayer to understand what they are computing. It should be clear then. $\endgroup$ – Sebastian Jul 29 '18 at 21:07
  • $\begingroup$ Thanks, that of EmbeddingLayer make it clear. $\endgroup$ – H42 Jul 31 '18 at 4:59

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.