7
$\begingroup$

The NetTrain function to train a neural network often takes several hours to complete. But if there's an error during that time, it simply aborts and returns $Failed, so hours of training progress are lost.

I'd prefer if NetTrain would just print a message for recoverable errors and continue training. Is there a way to achieve that?

For example, after several hours of training NetTrain just aborted with this error message:

RandomVariate::array: The array dimensions {104,2143} given in position 2 of RandomVariate[UniformDistribution[{-0.02,0.02}],{104,2143}] should be a list of non-negative machine-sized integers giving the dimensions for the result.

Since {104,2143} is a list of non-negative machine-sized integers, and the same line of code with the same arguments worked a million times before, I'd say this is a rare bug in RandomVariate. I don't see how I could fix or even reproduce it, so I'd rather just ignore it and try again.

Alternatively: Is there a way to catch all errors that might happen in my generator function, and simply try the same code again?

$\endgroup$
2
  • 1
    $\begingroup$ Have you seen TrainingProgessCheckpointing? $\endgroup$
    – Sascha
    Apr 5, 2017 at 12:08
  • $\begingroup$ @Sascha: I hadn't seen this, thanks. I'll definitely use that for future overnight training sessions. Then I wouldn't have lost the training progress until the error happened. Still would have wasted the PC time after the error. $\endgroup$ Apr 5, 2017 at 12:53

1 Answer 1

6
$\begingroup$

Yeah that's kinda annoying.

For situations where your data is a generator, and you expect a message to be issued when the failure happens (e.g. your RandomVariate example) this should robustify your training:

$generator = Function[...]; 
$robustGenerator = Function[$fallback = Check[$generator[#], $fallback]];
NetTrain[net, $robustGenerator, ...] 

This will just re-use the last batch if a message occurs during the call to the actual generator.

Ideally we'd handle this more directly in NetTrain, of course.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.