8
$\begingroup$

I have a neural network I am trying to train:

net=NetChain[{UnitVectorLayer[92], LongShortTermMemoryLayer[128], 
  DropoutLayer[0.2], LongShortTermMemoryLayer[128], LinearLayer[256], 
  DropoutLayer[0.3], LinearLayer[92], SoftmaxLayer[]}, 
 "Input" -> 100]
trainedNet=NetTrain[net, trainingData, BatchSize -> 64, MaxTrainingRounds -> 200,
  ValidationSet -> Scaled[0.1],TargetDevice->”GPU”]

With about 1000 training examples. The training goes well, except after a while, my computer can't handle it, and freezes completely. It becomes completely unresponsive to anything but the power button. This happens at varying times into the training - sometimes, it gets more than 50% through, and sometimes it barely gets through 2%. I’ve tried this on an admittedly bad laptop, but also on a fresh install of Windows and Mathematica 12 on a very powerful GPU with 16Gb of RAM.

Is there any way to find out why this is happening? Is there any documented resolutions to Mathematica freezing due to neural net training? Any way, perhaps, to pause and save the training state to train in small chunks?

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ You could try TrainingProgressCheckpointing to save the state. Implementing a TrainingProgressFunction might help with understanding why it freezes. $\endgroup$
    – Rohit Namjoshi
    Commented Jun 15, 2019 at 1:36
  • $\begingroup$ @RohitNamjoshi Thanks, it’s oddly difficult to find that documentation and I should be able to cobble something together with it. Any idea what I should try to monitor with TrainingProgessFunction? $\endgroup$
    – Nico A
    Commented Jun 15, 2019 at 1:40
  • 1
    $\begingroup$ In addition to the good advice from @RohitNamjoshi, you might try contacting tech support about this. For what it's worth, I have had similar problems, at least from the description. The underlying cause is running out of RAM and getting frozen by an OS that does not take kindly to being asked to read/write to/from storage. ("Thrash my disk? Why no, sir, I'd sooner thrash you.") $\endgroup$
    – Daniel Lichtblau
    Commented Jun 15, 2019 at 14:46
  • $\begingroup$ @DanielLichtblau I was thinking it was something ram related, but also it seems surprising that 16gb is not enough? Then again, I don’t know very much and many things are surprising to me! $\endgroup$
    – CA Trevillian
    Commented Jun 15, 2019 at 20:26
  • $\begingroup$ in this kind of case.. the first thing I'd try is to run it on the online programming lab. | Since you already have a valid license.. contacting tech support is not a bad option.. ( : $\endgroup$
    – p._phidot_
    Commented Jun 18, 2019 at 1:20

1 Answer 1

Reset to default
4
$\begingroup$

I've managed to find a workaround, though I know I'll be crucified for it. First, train the network for a safely short amount of time, and save an image of it:

trainedNet = 
 NetTrain[net, training, BatchSize -> 64, MaxTrainingRounds -> 2, 
  ValidationSet -> Scaled[0.1], 
  TrainingProgressCheckpointing -> {"Directory", 
    "C:\\Users\\TheMi\\Documents\\NeuralTemp"}]

And then continuously train and pause the training, allowing the computer some time to not implode:

Do[(NetTrain[
   Import[Last@
     FileNames[All, "C:\\Users\\TheMi\\Documents\\NeuralTemp"]], 
   training, BatchSize -> 64, MaxTrainingRounds -> 2, 
   ValidationSet -> Scaled[0.1], 
   TrainingProgressCheckpointing -> {"Directory", 
     "C:\\Users\\TheMi\\Documents\\NeuralTemp"}]; Pause[10];), 20]

This is possibly the worst code I've ever written, and is really a band-aid solution to the issue. However, seeing as I have now tested the freezing behavior on three machines, this seems to be an issue with Mathematica itself and until tech support gets back to me I doubt there is a better solution.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.