I have 2 nets, i.e. net1 (i.e. LinearLayer) and net2 (i.e. EmbeddingLayer) as below, while the input are just integers from 1 to 10:-

net1 = NetChain[{LinearLayer[2, "Input" -> 1]}];
net1 = BlockRandom[SeedRandom[10]; NetInitialize[net1]];
input = {#} & /@ Range[10];
output = net1[input];

enter image description here

net2 = NetChain[{EmbeddingLayer[2, 100]}];
net2 = BlockRandom[SeedRandom[10]; NetInitialize[net2]];
input = Range[10];
output = net2[input];

enter image description here

As you can see, net1 is linear, while net2 is obviously non-linear. But why net2 is not linear? What is the mathematics behind EmbeddingLayer? Is EmbeddingLayer also using the linear operation $y=A W + B$ or not?

Many thanks!

  • 1
    $\begingroup$ My guess is that it encodes the integers with one-hot encoding and given random initial weight matrix it should behave like this. $\endgroup$ – swish Jul 28 '18 at 18:42
  • $\begingroup$ Have a look in the 'Properties and Relations' section of EmbeddingLayer and LinearLayer to understand what they are computing. It should be clear then. $\endgroup$ – Sebastian Jul 29 '18 at 21:07
  • $\begingroup$ Thanks, that of EmbeddingLayer make it clear. $\endgroup$ – H42 Jul 31 '18 at 4:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.