3
$\begingroup$

I'm training a neural network at the moment on an image segmentation problem (that is, I have an images and masks, and I am trying to train the network to generate its own masks from other images.

I'd like the network to use Intersection over Union as its error metric, rather than whatever it otherwise uses ("error" in the training progress box).

training progress reporting box

However, I can't find anything in the documentation about this error metric, how it's calculated, or how to change it.

An IOU function is available in a comment on this post, in case it's useful. Other details about a segmentation network (UNET) are also available there.

However, my question is more arbitrary - how does one change this error metric at all, for any NetTrain invocation, rather than how do I change it to IOU.

Thanks.

$\endgroup$
  • $\begingroup$ When I change the LossFunction (for example to SoftIOU), the error goes away. I wonder if the error comes from a NetPort used by the default LossFunction. $\endgroup$ – Carl Lange Aug 26 '18 at 23:47
6
$\begingroup$

Unfortunately there is currently no way to override the default metrics (loss, and, where a cross entropy layer is used, error rate).

I'll describe our current plans for the next version, which may change of course.

In broad terms, we plan to make clarify and generalize the way NetTrain produces training networks and metrics like error rate.

In the general case, the net you provide can compute any combination of losses and metrics. You'll just have to tell NetTrain which output ports to drive down via SGD, and which output ports to report as metrics via a new TrainingMetrics option (these can overlap in the case of ordinary loss of course). So, for example, you could compute IOU and output it as a port called "IOU", and then specify NetTrain[..., TrainingMetrics -> "IOU"] to see it plotted during training.

We'll make it possible to compute F1 score, precision, and recall easily using special layers. Let us know if you have other suggestions for built-in metric layers.

These metrics will then be plotted nicely. They will also be available as properties of NetTrain and as keys in the various callback functions.

We'll retain the current "magic" that automatically makes a training network using an appropriate loss function when LossFunction is Automatic. In this special case, it might make sense to allow you to specify e.g. TrainingMetrics -> "F1" and have NetTrain automatically create the corresponding metric layers.

$\endgroup$
  • 1
    $\begingroup$ Thanks, that's very helpful. I look forward to the next version, these changes wuld be very beneficial. :) Just to clarify the first line of your comment, it is possible to modify the default LossFunction in 11.3 - I have currrently implemented a SoftIOU in a NetGraph that acts as a loss function. $\endgroup$ – Carl Lange Aug 27 '18 at 12:17
  • $\begingroup$ @CarlLange yes you can specify LossFunction -> "nameofoutputport" to specify that an output port should be used as the loss. This is documented. $\endgroup$ – Taliesin Beynon Sep 2 '18 at 19:16

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.