1
$\begingroup$

I'm trying to learn an embedding on Cifar-10 by finetuning ResNet-50 with a few added layers:

cifar10 = ResourceData["CIFAR-10"]
toRule[{img1_ -> l1_, img2_ -> l2_}] := {img1, img2} -> l1 != l2;
trainingData = Table[toRule@RandomSample[cifar10, 2], 30000];

res = NetModel["ResNet-50 Trained on ImageNet Competition Data"]
f = NetTake[res, {1, "flatten_0"}];
net = NetChain[{f, 50, DropoutLayer[0.3], Ramp, 2}]
neo = NetPairEmbeddingOperator[net]

trained = NetTrain[neo, trainingData, 
                LearningRateMultipliers -> {-4 -> 1, -1 -> 1, _ -> 0.01}]
embedding = NetExtract[trained, "Net"]

groups = KeySort@GroupBy[RandomSample[cifar10, 1000], Last -> First];
points = Map[embedding, groups];
ListPlot[KeyValueMap[Legended[#2, #1] &, points], Frame -> True, 
 Axes -> False, AspectRatio -> 1, FrameTicks -> False, 
 PlotStyle -> PointSize[0.012]]

However, the loss looks really weird (it should not be 0 when the training starts):

enter image description here

And so does the embedding:

enter image description here

This should be a simple task, what am I doing wrong here?

$\endgroup$
1
$\begingroup$

Ah, simple mistake, since the net is inside an operator, it should be:

LearningRateMultipliers -> {{"Net", -4;;-1;;2 -> 1}, {"Net", _ -> 0.01}}

I'll forgive myself since there is no example of using LearningRateMultipliers inside a NetPairEmbeddingOperator in the documentation (perhaps that would be nice to add).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.