So in short my issue is that I'am training a neural network and while the training loss reduces over time, the loss of validation set remains constant.

The progress of training looks like this:

Training progress

I am using the modified architecure of Single-Image depth perception which is changed to use the 64x64 images and I've added the linear layer at the end which has a 50 channels thus the net output is 50 values, these values are the PCA coefficients of the depth images which I use as the dataset. So basically I took a bunch of depth images, performed a PCA analythis and converted all images to the first 50 coefficients of the PCA model with the highest eigen-values. There is no issue with the PCA model or vectors, it was verified multiple times, but I can't get the net to train on the coefficients properly.

I've also tried to train on the depth images themselves which produced the expected training progress of gradual decline of both training loss and validation loss, but my goal is to produce the PCA values since the depth images produce by the net have too much noise which is really bad in my case. While the PCA coefficients produced by the net a pretty far off on the validation loss, the resulting depth image produced when converting PCA -> depth image is great in terms of low noise hence why I am training on the PCA coefficients.

I've also tried to reduce the number of channels of the convolution layers and barely managed to reduce the validation loss from 0.13 down to 0.11, while the training loss went from 0.001 to 0.09 and the size of the net went from 24Mb down to 0.5Mb. My overall goal is to get 0.05-0.01 validation loss.

Adding the L2Regularization either increases the trainig time or makes it so that loss never leaves the value of 1.

My NetTrain call looks like this:

Net = NetTrain[Net, trainingData, ValidationSet -> validationData, TargetDevice -> "GPU", TimeGoal -> 50*3600, Method -> {"ADAM"}, TrainingProgressCheckpointing -> {"Directory", NotebookDirectory[] <> "trained/PCA", "Interval" -> Quantity[1, "Hours"]}]

So my question is what can I do to the net architecture or the NetTrain parameters to achieve lower validation loss?

  • $\begingroup$ How large is your training set? If the net can approximate correct results on the training set and still get the validation set wrong, it might mean that the training set is too small (or the network's capacity to large) $\endgroup$ – Niki Estner Aug 2 at 9:41
  • $\begingroup$ The dataset size is 10k for training and 500 for validation. But even when I reduced the number of channels for convlolution layers in 8 times thust that reduced the number of trainable parameters down to about 100k, I was still getting about the same validation loss, even though the number of total output values in the dataset is about 500k (10k * 50). And I can't understand how can it overfit the training data and keep the validation loss constant at all time (next to zero change in validation loss) $\endgroup$ – Helseth Hlaalu Aug 2 at 9:53
  • 3
    $\begingroup$ I hope someone manages to give you some good hints, but you have to keep in mind that machine learning is all about tweaking. There are no universal answers here that I'm aware of. Maybe your data simple doesn't allow for better generalization then what you're getting. Maybe the architecture is completely unsuited for the task. Maybe you're stuck in the completely wrong part of optimization space. Maybe the regularization settings are wrong. There are a lot of possible issues that could be at play. $\endgroup$ – Sjoerd Smit Aug 2 at 10:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.