13
$\begingroup$

In v11.3, NetSharedArray[] and NetInsertSharedArray[] give us a mechanism of sharing weights and biases across neural networks, and this is exciting because now we can train siamese networks and do deep image ranking, and other related applications.

I don't see any applications section for either of these symbols in the documentation center, but I'd like to see how one might implement a basic Triplet Network, for example FaceNet. The problem is that I'm not sure how to write the triplet hinge loss in Mathematica.

References:

$\endgroup$
10
+50
$\begingroup$

The problem is that I'm not sure how to write the triplet hinge loss in Mathematica.

Suppose we have three images A,B,C. For each image, we have a feature vector, $\textbf{a}=f(A)$ etc. The formula for the triplet loss is then: $$loss({\textbf{a}, \textbf{p}, \textbf{n}}) = max(\Vert\textbf{a} - \textbf{p}\Vert^{2} - \Vert\textbf{a} - \textbf{n}\Vert^2 +\alpha, 0)$$

First, define a NetGraph that computes the L2-Norm $\Vert\ldots\Vert$:

l2norm = NetGraph[{ThreadingLayer[(#1 - #2)^2 &], SummationLayer[]}, {1 -> 2}]

Now the triplet loss is simply (with an $\alpha=0.2$):

alpha = 0.2;
tripletloss = NetGraph[{l2norm, l2norm, ThreadingLayer[Max[#1 - #2 + alpha, 0]&]}, 
  {{NetPort["a"], NetPort["p"]} -> 1 -> 3, {NetPort["a"], NetPort["n"]} -> 2 -> 3}]

We can evaluate this loss for three feature vectors:

tripletloss[<|"a" -> {3, 2}, "n" -> {3, 2}, "p" -> {1.2, 2.1}|>]
Out[145]= 3.45

This can be used as a loss in your triplet net.

$\endgroup$
  • $\begingroup$ Are there any built-in image datasets that have (anchor, positive,negative) labels to test this on? $\endgroup$ – user5601 Mar 16 '18 at 1:21
  • $\begingroup$ @user5601 you can always check the Neural Net repo or its public face to see $\endgroup$ – b3m2a1 Mar 17 '18 at 22:03
  • $\begingroup$ @b3m2a1 I did check there first, nothing there that's relevant to triplet embeddings. $\endgroup$ – user5601 Mar 18 '18 at 1:44
2
$\begingroup$

I'm adding this post becuase it is the complete answer to the original question (I accepted Sebastian's parital answer because it was very helpful).

This code implements a MWE of what I had in mind, and adds a few follow-up questions/remarks:

(* Define the loss from Sebastian's post above *)
l2norm = NetGraph[{ThreadingLayer[(#1 - #2)^2 &], 
   SummationLayer[]}, {1 -> 2}]; alpha = 0.2;
tripletloss = NetGraph[{l2norm, l2norm, 
   ThreadingLayer[Max[#1 - #2 + alpha, 0] &]}, {{NetPort["a"], NetPort["p"]} -> 
    1 -> 3, {NetPort["a"], NetPort["n"]} -> 2 -> 3}]

(* get triplet training data e.g. (anchor, pos, neg) *)
resource = ResourceObject["MNIST"];
trainingData = ResourceData[resource, "TrainingData"];
tripleBatchGenerator[assn_Association] := Module[{pi, ni, a, p, n},
  Table[pi = RandomInteger[{0, 9}];
   ni = RandomChoice[Complement[Range[0, 9], {pi}]];
   pos = Position[trainingData, x_ -> pi];
   {a, p} = Extract[trainingData, RandomSample[pos, 2]][[All, 1]];
   n = Part[trainingData, 
      RandomChoice[
       Complement[Range[Length@trainingData], Flatten[pos]]]][[1]];
   <|"a" -> a, "p" -> p, "n" -> n|>, assn["BatchSize"]]
  ]

(* Question: is this a correct/good way to make the siamese portion? *)
evalnet = NetTake[NetModel["LeNet"], {1, -3}]
aevalnet = NetInsertSharedArrays[evalnet];
pevalnet = NetInsertSharedArrays[evalnet];
nevalnet = NetInsertSharedArrays[evalnet];

(* Sow pieces into a netgraph for training *)
enc = NetEncoder[{"Image", {28, 28}, "Grayscale"}];
net = NetGraph[<|
   "aevalnet" -> aevalnet,
   "pevalnet" -> pevalnet,
   "nevalnet" -> nevalnet,
   "loss" -> tripletloss|>, {
   NetPort["a"] -> "aevalnet" -> NetPort["loss", "a"], 
   NetPort["p"] -> "pevalnet" -> NetPort["loss", "p"], 
   NetPort["n"] -> "nevalnet" -> NetPort["loss", "n"],
   "loss" -> NetPort["Loss"]}, "a" -> enc, "p" -> enc, "n" -> enc]

 (* Train it! *)
 trainResult = NetTrain[net, tripleBatchGenerator, All, MaxTrainingRounds -> 500]

It starts training correctly:

enter image description here

However, here are some follow-up questions/remarks:

  1. When using a generator function, specifying the ValidationSet doesn't seem to work (e.g. ValidationSet->Scaled[.1]).
  2. Is there any good way to automate a search for the hyperparameter alpha (which was randomly set to .2)?
  3. Are the 3 calls to NetInsertSharedArrays right, is using NetMapOperator better?

Perhaps @Sebastian can address these to help put the finishing touches on it?

$\endgroup$
  • 1
    $\begingroup$ @Sebastian I'm not sure how to apply NetInsertSharedArrays correctly... $\endgroup$ – M.R. Oct 11 '18 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.