I'm very curious why the loss of validation set is even lower than the loss of training set when I use NetTrain.

Say, on this page, https://reference.wolfram.com/language/tutorial/NeuralNetworksSequenceLearning.html#1094728277 ,

for the Q&A RNN trained on the bAbI QA Dataset, the loss of validation set shouldn't be lower than the loss of training set, according to Goodfellow's DL book. Right?

Is it possible that these 2 sets are mistakenly labeled in the NetTrain function when it tries to plot the learning curve? enter image description here

  • $\begingroup$ Welcome! If you could please add the code to reproduce this, we can try to help. $\endgroup$
    – M.R.
    Apr 13, 2020 at 18:14
  • $\begingroup$ Crossposted here. $\endgroup$ Apr 13, 2020 at 18:27

1 Answer 1


It is not a bug, this behaviour is explained by the dropout. The training loss is computed with dropout enabled (as in DropoutLayer[...][..., NetEvaluationMode -> "Train"]). The validation loss is computed with dropout disabled (as in DropoutLayer[...][..., NetEvaluationMode -> "Test"]).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.