Can I change NetTrain's error handling?

Question

The NetTrain function to train a neural network often takes several hours to complete. But if there's an error during that time, it simply aborts and returns $Failed, so hours of training progress are lost.

I'd prefer if NetTrain would just print a message for recoverable errors and continue training. Is there a way to achieve that?

For example, after several hours of training NetTrain just aborted with this error message:

RandomVariate::array: The array dimensions {104,2143} given in position 2 of RandomVariate[UniformDistribution[{-0.02,0.02}],{104,2143}] should be a list of non-negative machine-sized integers giving the dimensions for the result.

Since {104,2143} is a list of non-negative machine-sized integers, and the same line of code with the same arguments worked a million times before, I'd say this is a rare bug in RandomVariate. I don't see how I could fix or even reproduce it, so I'd rather just ignore it and try again.

Alternatively: Is there a way to catch all errors that might happen in my generator function, and simply try the same code again?

Answer 1

Yeah that's kinda annoying.

For situations where your data is a generator, and you expect a message to be issued when the failure happens (e.g. your RandomVariate example) this should robustify your training:

$generator = Function[...]; 
$robustGenerator = Function[$fallback = Check[$generator[#], $fallback]];
NetTrain[net, $robustGenerator, ...] 

This will just re-use the last batch if a message occurs during the call to the actual generator.

Ideally we'd handle this more directly in NetTrain, of course.