The NetTrain
function to train a neural network often takes several hours to complete. But if there's an error during that time, it simply aborts and returns $Failed
, so hours of training progress are lost.
I'd prefer if NetTrain
would just print a message for recoverable errors and continue training. Is there a way to achieve that?
For example, after several hours of training NetTrain
just aborted with this error message:
RandomVariate::array: The array dimensions {104,2143} given in position 2 of RandomVariate[UniformDistribution[{-0.02,0.02}],{104,2143}] should be a list of non-negative machine-sized integers giving the dimensions for the result.
Since {104,2143}
is a list of non-negative machine-sized integers, and the same line of code with the same arguments worked a million times before, I'd say this is a rare bug in RandomVariate
. I don't see how I could fix or even reproduce it, so I'd rather just ignore it and try again.
Alternatively: Is there a way to catch all errors that might happen in my generator function, and simply try the same code again?
TrainingProgessCheckpointing
? $\endgroup$