# Custom Loss Layer to ignore ordering

I am trying to train a net that takes a sequence of numbers and gives a sequence of numbers. For example:

net = NetInitialize@NetChain[{LongShortTermMemoryLayer[1]}, "Input" -> {"Varying", 1}]


However, during training I do not care about the ordering of the numbers in the result so that I want a loss function loss that has the following property:

loss[<|"Input" -> {{2},{1}}, "Target" -> {{2},{1}}|>]
==loss[<|"Input" -> {{1},{2}}, "Target" -> {{2},{1}}|>]


Is there a way to define, for example, a MeanSquaredLossLayer that ignores order?

Best, Max

• You're going to have to find some way to sort the elements of your sequence. I'm not quite sure how to achieve that, but you might be able to train a post-processing net that sorts sequences or something. – Sjoerd Smit Apr 7 '18 at 19:37
• yes, if I could use SortBy in the loss layer it would be amazing but training a whole other net to sort sequences would be a bit of an overkill. – MaxJ Apr 7 '18 at 21:34
• The thing is: I'm not aware of any build-in functionality that will do the sorting for you. Training a network to do the sorting for you should be a one-time effort and once you've done that, you can keep re-using the trained sorting net. It should be fairly efficient overall and I really don't know of another way to do it. – Sjoerd Smit Apr 8 '18 at 11:48

We must do this by sorting the input before giving it to the loss layer.

In version 11.3, you can use the undocumented MXLayer functionality to sort your data before feeding it to MeanSquaredLossLayer or similar. In version 12 and up, you can use OrderingLayer.

Version 11.3 only: Here we define an MXLayer (undocumented) that will sort some data.

sortLayer = NeuralNetworksMXLayer[
<|"Input" -> {10}|>, <|"Output" -> {10}|>,
Function[
in = NeuralNetworksPrivateGetInput["Input"];
sorted = NeuralNetworksPrivateSowNode["sort", in];
NeuralNetworksPrivateSetOutput["Output", sorted]
]
]


Be aware that you need to modify the Input and Output dimensions to work with your data. Leaving it as {10} like this allows for lists of length 10. "Varying" will work for "Input" but doesn't appear to work for "Output", so this may not fully answer your question.

What we're doing there is using the sort functionality in MXNet on the Input port and Outputting the sorted data.

We can see that this works:

sortLayer[{3, 5, 1, 3, 2}]


{1., 2., 3., 3., 5.}

Version 12+: We can use the built-in OrderingLayer to sort data for us, as the previous MXLayer functionality has been removed. However, I am having a difficult time actually making this happen. This is approximately what I would have expected:

NetGraph[{OrderingLayer[], ReshapeLayer[{Inherited, 1}],
ExtractLayer[]}, {
NetPort@"Input" -> 1 -> 2,
NetPort@"Input" -> NetPort[{3, "Input"}],
2 -> NetPort[{3, "Position"}]
}]


but sadly it returns \$Failed.

Now we can use your sortLayer, however it's defined, on top of a MeanSquaredLossLayer like so:

sortedMeanSquaredLossLayer = NetGraph[{
sortLayer,
sortLayer,
MeanSquaredLossLayer[]
},
{NetPort@"Input" -> 1,
NetPort@"Target" -> 2,
{1, 2} -> 3}
]


We're sorting the Input and Target data and feeding them to the loss layer.

sortedMeanSquaredLossLayer[<|"Input" -> {1, 3, 2, 4, 5},
"Target" -> {1, 2, 4, 5, 6}|>]
`

0.6