How to modify NetTrain's error calculation

Question

I'm training a neural network at the moment on an image segmentation problem (that is, I have an images and masks, and I am trying to train the network to generate its own masks from other images.

I'd like the network to use Intersection over Union as its error metric, rather than whatever it otherwise uses ("error" in the training progress box).

However, I can't find anything in the documentation about this error metric, how it's calculated, or how to change it.

An IOU function is available in a comment on this post, in case it's useful. Other details about a segmentation network (UNET) are also available there.

However, my question is more arbitrary - how does one change this error metric at all, for any NetTrain invocation, rather than how do I change it to IOU.

Thanks.

Answer 1

Unfortunately there is currently no way to override the default metrics (loss, and, where a cross entropy layer is used, error rate).

I'll describe our current plans for the next version, which may change of course.

In broad terms, we plan to make clarify and generalize the way NetTrain produces training networks and metrics like error rate.

In the general case, the net you provide can compute any combination of losses and metrics. You'll just have to tell NetTrain which output ports to drive down via SGD, and which output ports to report as metrics via a new TrainingMetrics option (these can overlap in the case of ordinary loss of course). So, for example, you could compute IOU and output it as a port called "IOU", and then specify NetTrain[..., TrainingMetrics -> "IOU"] to see it plotted during training.

We'll make it possible to compute F1 score, precision, and recall easily using special layers. Let us know if you have other suggestions for built-in metric layers.

These metrics will then be plotted nicely. They will also be available as properties of NetTrain and as keys in the various callback functions.

We'll retain the current "magic" that automatically makes a training network using an appropriate loss function when LossFunction is Automatic. In this special case, it might make sense to allow you to specify e.g. TrainingMetrics -> "F1" and have NetTrain automatically create the corresponding metric layers.