How does mathematica compute loss with multi-losses?

Question

I want to make a DNN that it can learn from text feature(1534-dims) and old audio features(256*4(four old frames)-dims) to predict audio feature(256-dims) at current frame.

So it has two inputs,one is text feature(1534-dims),the other is old audio features(1024 dims).And one output is current audio feature(256-dims).

First I want net can learn how to reduce dimensions from 1534-dims to 128-dims,so I define encodeNet and decodeNet.

encodeNet = NetChain[{512, Ramp, 128}, "Input" -> 1534(*text feature*)];
decodeNet = NetChain[{512, Ramp, 512, Ramp, 256}, 
                     "Input" -> 128(*low rank representation of text feature*)];

Second I want to combine this reduced dimensions 128-dims and old audio features 1024-dims,then use it to predict the current audio features.

net = NetGraph[{encodeNet, decodeNet, CatenateLayer[], 512, Ramp, 512,Ramp, 256}, 
           {NetPort["Input1"] -> 1 -> 2 -> NetPort["Output1"], 
           NetPort["Input2"] -> 3, 1 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> NetPort["Output2"]}, 
           "Input1" -> 1534, "Input2" -> 256*4, "Output1" -> 256, "Output2" -> 256]

So the data into output1 is same as the data into output2.

input1 = N@RandomInteger[1, {100, 1534}];
input2 =   RandomReal[1, {100, 256*4}];
output =   RandomReal[1, {100, 256}];

data = <|"Input1" -> input1, "Input2" -> input2, "Output1" -> output,"Output2" -> output|>;

Then train the net.

net = NetTrain[net, data, MaxTrainingRounds -> 100];

So how does mathematica compute loss with multi-losses? is it loss=loss1+loss2?

And is there better net than can deal with this problem? I want to use EmbeddingLayer to replace encodeNet, but have no idea.

And can it have one ouput port and how to use it?

Answer 1

The default behavior for multiple loss is that they are summed together to give a single total loss as the loss of the network. We can verify that by comparing networks:

• A network with two losses
• A network with a single loss that is the sum of the two losses

First, prepare some data

input1 = RandomReal[1, {10, 2}];
input2 = RandomReal[1, {10, 2}];
output1 = RandomReal[1, {10, 10}];
output2 = RandomReal[1, {10, 10}];

This is the network with two losses

SeedRandom[1234];

net1 = NetInitialize[
  NetGraph[{10, Ramp, 10, Ramp, CatenateLayer[], 
    10}, {NetPort["Input1"] -> 1 -> 2 -> 5, 
    NetPort["Input2"] -> 3 -> 4 -> 5 -> 6 -> NetPort["Output2"], 
    2 -> NetPort["Output1"]}, "Input2" -> 2, "Input1" -> 2]]

When training, two MeanSquaredLossLayer's are attached automatically. We can print the loss in training

data1 = <|"Input1" -> input1, "Input2" -> input2, "Output1" -> output1,
    "Output2" -> output2|>;
NetTrain[net1, data1, MaxTrainingRounds -> 4, 
  TrainingProgressFunction -> (Print[#RoundLoss] &)];

(*
0.756067
0.752072
0.748098
0.74413
*)

Now let's compare a network with a single loss that is the sum of the two losses in the first network

SeedRandom[1234];

net2 = NetInitialize[
  NetGraph[{10, Ramp, 10, Ramp, CatenateLayer[], 10, 
    MeanSquaredLossLayer[], MeanSquaredLossLayer[], ReshapeLayer[{1}],
     ReshapeLayer[{1}], 
    TotalLayer[]}, {NetPort["Input1"] -> 1 -> 2 -> 5, 
    NetPort["Input2"] -> 3 -> 4 -> 5 -> 6 -> 8, 2 -> 7, 
    NetPort[7, "Loss"] -> 9 -> 11, 
    NetPort[8, "Loss"] -> 10 -> 11 -> NetPort["Loss"], 
    NetPort["Target1"] -> NetPort[7, "Target"], 
    NetPort["Target2"] -> NetPort[8, "Target"]}, "Input2" -> 2, 
   "Input1" -> 2]]

l to the first network

data2 = <|"Input1" -> input1, "Input2" -> input2, 
   "Target1" -> output1, "Target2" -> output2|>;

NetTrain[net2, data2, MaxTrainingRounds -> 4, 
  TrainingProgressFunction -> (Print[#RoundLoss] &)];

(*
0.756067
0.752072
0.748098
0.74413
*)

The default behavior can be overwritten by setting the third argument of NetTrain.