The NetTrain function to train a neural network often takes several hours to complete. But if there's an error during that time, it simply aborts and returns $Failed, so hours of training progress are lost.

I'd prefer if NetTrain would just print a message for recoverable errors and continue training. Is there a way to achieve that?

For example, after several hours of training NetTrain just aborted with this error message:

RandomVariate::array: The array dimensions {104,2143} given in position 2 of RandomVariate[UniformDistribution[{-0.02,0.02}],{104,2143}] should be a list of non-negative machine-sized integers giving the dimensions for the result.

Since {104,2143} is a list of non-negative machine-sized integers, and the same line of code with the same arguments worked a million times before, I'd say this is a rare bug in RandomVariate. I don't see how I could fix or even reproduce it, so I'd rather just ignore it and try again.

Alternatively: Is there a way to catch all errors that might happen in my generator function, and simply try the same code again?

  • 1
    $\begingroup$ Have you seen TrainingProgessCheckpointing? $\endgroup$ – Sascha Apr 5 '17 at 12:08
  • $\begingroup$ @Sascha: I hadn't seen this, thanks. I'll definitely use that for future overnight training sessions. Then I wouldn't have lost the training progress until the error happened. Still would have wasted the PC time after the error. $\endgroup$ – Niki Estner Apr 5 '17 at 12:53

Yeah that's kinda annoying.

For situations where your data is a generator, and you expect a message to be issued when the failure happens (e.g. your RandomVariate example) this should robustify your training:

$generator = Function[...]; 
$robustGenerator = Function[$fallback = Check[$generator[#], $fallback]];
NetTrain[net, $robustGenerator, ...] 

This will just re-use the last batch if a message occurs during the call to the actual generator.

Ideally we'd handle this more directly in NetTrain, of course.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.