# How does mathematica compute loss with multi-losses?

I want to make a DNN that it can learn from text feature(1534-dims) and old audio features(256*4(four old frames)-dims) to predict audio feature(256-dims) at current frame.

So it has two inputs,one is text feature(1534-dims),the other is old audio features(1024 dims).And one output is current audio feature(256-dims).

First I want net can learn how to reduce dimensions from 1534-dims to 128-dims,so I define encodeNet and decodeNet.

encodeNet = NetChain[{512, Ramp, 128}, "Input" -> 1534(*text feature*)];
decodeNet = NetChain[{512, Ramp, 512, Ramp, 256},
"Input" -> 128(*low rank representation of text feature*)];


Second I want to combine this reduced dimensions 128-dims and old audio features 1024-dims,then use it to predict the current audio features.

net = NetGraph[{encodeNet, decodeNet, CatenateLayer[], 512, Ramp, 512,Ramp, 256},
{NetPort["Input1"] -> 1 -> 2 -> NetPort["Output1"],
NetPort["Input2"] -> 3, 1 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> NetPort["Output2"]},
"Input1" -> 1534, "Input2" -> 256*4, "Output1" -> 256, "Output2" -> 256] So the data into output1 is same as the data into output2.

input1 = N@RandomInteger[1, {100, 1534}];
input2 =   RandomReal[1, {100, 256*4}];
output =   RandomReal[1, {100, 256}];

data = <|"Input1" -> input1, "Input2" -> input2, "Output1" -> output,"Output2" -> output|>;


Then train the net.

net = NetTrain[net, data, MaxTrainingRounds -> 100]; So how does mathematica compute loss with multi-losses? is it loss=loss1+loss2?

And is there better net than can deal with this problem? I want to use EmbeddingLayer to replace encodeNet, but have no idea.

And can it have one ouput port and how to use it?

• There's no support for multiple losses yet – M.R. May 22 '17 at 1:24
• @M.R. But it really compute loss...And it also can train the net... – partida May 22 '17 at 3:16
• I think the total loss is the sum of all losses in default. You can find an example of the multiple-loss training in the last example of NetTrain->Loss Specifications. You can also specify the total loss as a weighted sum of the individual loss using the third parameter to NetTrain. You can find an example of that in the style transfer example in NetTrain. – xslittlegrass May 22 '17 at 4:10

## 1 Answer

The default behavior for multiple loss is that they are summed together to give a single total loss as the loss of the network. We can verify that by comparing networks:

• A network with two losses
• A network with a single loss that is the sum of the two losses

First, prepare some data

input1 = RandomReal[1, {10, 2}];
input2 = RandomReal[1, {10, 2}];
output1 = RandomReal[1, {10, 10}];
output2 = RandomReal[1, {10, 10}];


This is the network with two losses

SeedRandom;

net1 = NetInitialize[
NetGraph[{10, Ramp, 10, Ramp, CatenateLayer[],
10}, {NetPort["Input1"] -> 1 -> 2 -> 5,
NetPort["Input2"] -> 3 -> 4 -> 5 -> 6 -> NetPort["Output2"],
2 -> NetPort["Output1"]}, "Input2" -> 2, "Input1" -> 2]] When training, two MeanSquaredLossLayer's are attached automatically. We can print the loss in training

data1 = <|"Input1" -> input1, "Input2" -> input2, "Output1" -> output1,
"Output2" -> output2|>;
NetTrain[net1, data1, MaxTrainingRounds -> 4,
TrainingProgressFunction -> (Print[#RoundLoss] &)];

(*
0.756067
0.752072
0.748098
0.74413
*)


Now let's compare a network with a single loss that is the sum of the two losses in the first network

SeedRandom;

net2 = NetInitialize[
NetGraph[{10, Ramp, 10, Ramp, CatenateLayer[], 10,
MeanSquaredLossLayer[], MeanSquaredLossLayer[], ReshapeLayer[{1}],
ReshapeLayer[{1}],
TotalLayer[]}, {NetPort["Input1"] -> 1 -> 2 -> 5,
NetPort["Input2"] -> 3 -> 4 -> 5 -> 6 -> 8, 2 -> 7,
NetPort[7, "Loss"] -> 9 -> 11,
NetPort[8, "Loss"] -> 10 -> 11 -> NetPort["Loss"],
NetPort["Target1"] -> NetPort[7, "Target"],
NetPort["Target2"] -> NetPort[8, "Target"]}, "Input2" -> 2,
"Input1" -> 2]] l to the first network

data2 = <|"Input1" -> input1, "Input2" -> input2,
"Target1" -> output1, "Target2" -> output2|>;

NetTrain[net2, data2, MaxTrainingRounds -> 4,
TrainingProgressFunction -> (Print[#RoundLoss] &)];

(*
0.756067
0.752072
0.748098
0.74413
*)


The default behavior can be overwritten by setting the third argument of NetTrain.

• why use CatenateLayer[] other than TotalLayer[] in the last layer of net2? – partida May 24 '17 at 2:13
• @partida You are right, using TotalLayer layer is more reasonable. – xslittlegrass May 24 '17 at 15:19