# Can I change NetTrain's error handling?

The NetTrain function to train a neural network often takes several hours to complete. But if there's an error during that time, it simply aborts and returns $Failed, so hours of training progress are lost. I'd prefer if NetTrain would just print a message for recoverable errors and continue training. Is there a way to achieve that? For example, after several hours of training NetTrain just aborted with this error message: RandomVariate::array: The array dimensions {104,2143} given in position 2 of RandomVariate[UniformDistribution[{-0.02,0.02}],{104,2143}] should be a list of non-negative machine-sized integers giving the dimensions for the result. Since {104,2143} is a list of non-negative machine-sized integers, and the same line of code with the same arguments worked a million times before, I'd say this is a rare bug in RandomVariate. I don't see how I could fix or even reproduce it, so I'd rather just ignore it and try again. Alternatively: Is there a way to catch all errors that might happen in my generator function, and simply try the same code again? • Have you seen TrainingProgessCheckpointing? – Sascha Apr 5 '17 at 12:08 • @Sascha: I hadn't seen this, thanks. I'll definitely use that for future overnight training sessions. Then I wouldn't have lost the training progress until the error happened. Still would have wasted the PC time after the error. – Niki Estner Apr 5 '17 at 12:53 ## 1 Answer Yeah that's kinda annoying. For situations where your data is a generator, and you expect a message to be issued when the failure happens (e.g. your RandomVariate example) this should robustify your training: $generator = Function[...];
$robustGenerator = Function[$fallback = Check[$generator[#],$fallback]];
NetTrain[net, \$robustGenerator, ...]


This will just re-use the last batch if a message occurs during the call to the actual generator.

Ideally we'd handle this more directly in NetTrain, of course.