# How does mathematica compute loss with multi-losses?

I want to make a DNN that it can learn from text feature(1534-dims) and old audio features(256*4(four old frames)-dims) to predict audio feature(256-dims) at current frame.

So it has two inputs,one is text feature(1534-dims),the other is old audio features(1024 dims).And one output is current audio feature(256-dims).

First I want net can learn how to reduce dimensions from 1534-dims to 128-dims,so I define encodeNet and decodeNet.

encodeNet = NetChain[{512, Ramp, 128}, "Input" -> 1534(*text feature*)];
decodeNet = NetChain[{512, Ramp, 512, Ramp, 256},
"Input" -> 128(*low rank representation of text feature*)];


Second I want to combine this reduced dimensions 128-dims and old audio features 1024-dims,then use it to predict the current audio features.

net = NetGraph[{encodeNet, decodeNet, CatenateLayer[], 512, Ramp, 512,Ramp, 256},
{NetPort["Input1"] -> 1 -> 2 -> NetPort["Output1"],
NetPort["Input2"] -> 3, 1 -> 3 -> 4 -> 5 -> 6 -> 7 -> 8 -> NetPort["Output2"]},
"Input1" -> 1534, "Input2" -> 256*4, "Output1" -> 256, "Output2" -> 256]


So the data into output1 is same as the data into output2.

input1 = N@RandomInteger[1, {100, 1534}];
input2 =   RandomReal[1, {100, 256*4}];
output =   RandomReal[1, {100, 256}];

data = <|"Input1" -> input1, "Input2" -> input2, "Output1" -> output,"Output2" -> output|>;


Then train the net.

net = NetTrain[net, data, MaxTrainingRounds -> 100];


So how does mathematica compute loss with multi-losses? is it loss=loss1+loss2?

And is there better net than can deal with this problem? I want to use EmbeddingLayer to replace encodeNet, but have no idea.

And can it have one ouput port and how to use it?

• There's no support for multiple losses yet
– M.R.
Commented May 22, 2017 at 1:24
• @M.R. But it really compute loss...And it also can train the net... Commented May 22, 2017 at 3:16
• I think the total loss is the sum of all losses in default. You can find an example of the multiple-loss training in the last example of NetTrain->Loss Specifications. You can also specify the total loss as a weighted sum of the individual loss using the third parameter to NetTrain. You can find an example of that in the style transfer example in NetTrain. Commented May 22, 2017 at 4:10

The default behavior for multiple loss is that they are summed together to give a single total loss as the loss of the network. We can verify that by comparing networks:

• A network with two losses
• A network with a single loss that is the sum of the two losses

First, prepare some data

input1 = RandomReal[1, {10, 2}];
input2 = RandomReal[1, {10, 2}];
output1 = RandomReal[1, {10, 10}];
output2 = RandomReal[1, {10, 10}];


This is the network with two losses

SeedRandom[1234];

net1 = NetInitialize[
NetGraph[{10, Ramp, 10, Ramp, CatenateLayer[],
10}, {NetPort["Input1"] -> 1 -> 2 -> 5,
NetPort["Input2"] -> 3 -> 4 -> 5 -> 6 -> NetPort["Output2"],
2 -> NetPort["Output1"]}, "Input2" -> 2, "Input1" -> 2]]


When training, two MeanSquaredLossLayer's are attached automatically. We can print the loss in training

data1 = <|"Input1" -> input1, "Input2" -> input2, "Output1" -> output1,
"Output2" -> output2|>;
NetTrain[net1, data1, MaxTrainingRounds -> 4,
TrainingProgressFunction -> (Print[#RoundLoss] &)];

(*
0.756067
0.752072
0.748098
0.74413
*)


Now let's compare a network with a single loss that is the sum of the two losses in the first network

SeedRandom[1234];

net2 = NetInitialize[
NetGraph[{10, Ramp, 10, Ramp, CatenateLayer[], 10,
MeanSquaredLossLayer[], MeanSquaredLossLayer[], ReshapeLayer[{1}],
ReshapeLayer[{1}],
TotalLayer[]}, {NetPort["Input1"] -> 1 -> 2 -> 5,
NetPort["Input2"] -> 3 -> 4 -> 5 -> 6 -> 8, 2 -> 7,
NetPort[7, "Loss"] -> 9 -> 11,
NetPort[8, "Loss"] -> 10 -> 11 -> NetPort["Loss"],
NetPort["Target1"] -> NetPort[7, "Target"],
NetPort["Target2"] -> NetPort[8, "Target"]}, "Input2" -> 2,
"Input1" -> 2]]


l to the first network

data2 = <|"Input1" -> input1, "Input2" -> input2,
"Target1" -> output1, "Target2" -> output2|>;

NetTrain[net2, data2, MaxTrainingRounds -> 4,
TrainingProgressFunction -> (Print[#RoundLoss] &)];

(*
0.756067
0.752072
0.748098
0.74413
*)


The default behavior can be overwritten by setting the third argument of NetTrain.

• why use CatenateLayer[] other than TotalLayer[] in the last layer of net2? Commented May 24, 2017 at 2:13
• @partida You are right, using TotalLayer layer is more reasonable. Commented May 24, 2017 at 15:19