7
$\begingroup$

I would like to train a word-embedding type model (specifically "Walklets" from the paper by Bryan Perozzi et al).

In the simplest form (ignoring stuff like negative sampling), we want to maximize $\Pr(\phi(i) | \phi(j))$ given observed input pairs, where $\phi(x)$ gives the embedding vector. This is modeled by

$$\frac{1}{1 + e^{-\phi(i) \cdot \phi(j)}}$$.

So essentially I need to maximize this value (perhaps after log transformations or whatever else is convenient).

So far I have the following:

walkletNetwork[g_?GraphQ, d_Integer] := Module[
  {embed = NetInitialize[EmbeddingLayer[Length[VertexList[g]], d]]},
  NetGraph[
   {embed, embed},
   {NetPort["Input1"] -> 1, NetPort["Input2"] -> 2}]]

Now I want to take those two outputs and simply compute the dot product, then apply the LogisticSigmoid transformation on the result and use it as my loss function. Is there a way to do this within the current framework? If not, does anyone know of a workaround? Am I barking up the wrong tree?

$\endgroup$
1
  • $\begingroup$ with new operators in the prerelease 11.1 there are built-in operators for computing the distance between the embeddings of two points, and for using contrastive divergence. i'm not sure it makes sense to post them right now, because they could change at any time. however, if you need to do something like this and can get access to the prerelease, I recommend it. $\endgroup$ – Michael Curry Feb 21 '17 at 21:04
4
$\begingroup$

This is much easier now that 11.1 is out with ContrastiveLossLayer and NetPairEmbeddingOperator. The solution would be something like:

ClearAll[walkletNetwork];
walkletNetwork[vertices_List, d_Integer] :=
 NetInitialize[
  NetChain[
   {NetPairEmbeddingOperator[EmbeddingLayer[d]]},
   "Input" -> NetEncoder[{"Class", vertices}]]]

And then, when training, provide real and fake input pairs and specify ContrastiveLossLayer[] as the loss function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.