I have been experimenting with GPU NetTrain on AWS now when Mma 12.2 supports remote batch jobs (I don't have an Nvidia GPU to try these things out otherwise). In particular, I'm puzzled by the example on Mathematica documentation tutorial: Sequence Learning and NLP with Neural Networks, Language Modeling, teacherForcingNet variant. After evaluating requisites, this example does training like this:

result = NetTrain[teacherForcingNet, <|"Input" -> Keys[trainingData]|>,
   All, BatchSize -> 64, MaxTrainingRounds -> 5, 
  TargetDevice -> "CPU", ValidationSet -> Scaled[0.1]]

A CPU-based run, also if run on AWS, results a network with around 40% loss.

I have minimally modified the NetTrain step to perform this on GPU, and to run it on AWS:

job = RemoteBatchSubmit[env, 
  NetTrain[teacherForcingNet, <|"Input" -> Keys[trainingData]|>, All, 
   BatchSize -> 64, MaxTrainingRounds -> 5, TargetDevice -> "GPU", 
   ValidationSet -> Scaled[0.1]], 
  TimeConstraint -> Quantity[30, "Minutes"], 
  RemoteProviderSettings -> <|"GPUCount" -> 1|>]

When the training job completes, the resulting training object is available as job["EvaluationResult"] (and progress can be actually observed on runtime through job["JobLog"]). The problem is that when CPU-based training results error rate of around 41%, GPU-based run gets stuck at about 82% (effectively without learning anything).

What gives? Is this common behaviour for some networks (LeNet on MNIST dataset works just fine on GPU, for instance), a bug that needs fixing, and/or is a workaround available? Neither Method nor WorkingPrecision changes give difference in results.

  • $\begingroup$ Executing the first example at reference.wolfram.com/language/tutorial/… with Round@trained[{"5+2", "10+25", "9+11", "44+44"}], I obtain {8,35,18,88}. $\endgroup$
    – user64494
    Dec 30 '20 at 7:41
  • $\begingroup$ @user64494 That example is on the sequence regression section, I'm referring to language modelling section. I should probably try other examples with AWS cloud GPUs too, but my current interest happens to lie in language modelling... $\endgroup$
    – kirma
    Dec 30 '20 at 9:36
  • 1
    $\begingroup$ Strangely I get exactly the same result on my local machine - on GPU, the error rate doesn't descend at all. $\endgroup$
    – Carl Lange
    Dec 31 '20 at 20:23
  • 1
    $\begingroup$ I'm afraid I can't come up with any reason for this behaviour at all. I've tried everything I can think of to get some different result out of the GPU-trained version with no luck - though GPU training generally works fine. I would guess there's some bug in the GPU library triggered by this network - I'd say file a bug. $\endgroup$
    – Carl Lange
    Dec 31 '20 at 20:41
  • 1
    $\begingroup$ Could this relate to either 1) version of the MXNetResources paclet or 2) version of local CUDA runtime, drivers, etc.? $\endgroup$ Jan 15 at 20:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.