3
$\begingroup$

I am experimenting with Mathematica's Classify/NetTrain functions for the first time. When I don't set a TargetDevice option, Mathematica defaults to the CPU and everything works as expected. When I specify "TargetDevice"->"GPU" and run the same code, the kernel runs the code seemingly indefinitely and I can't interrupt the process without quitting the kernel.

I do get this error whenever I use function using "NeuralNetwork" for the first time:

General::newmxnet: An unexpected condition happened: a neural network runtime of the wrong version was loaded. This might lead to unexpected problems. Please contact Wolfram Research.

afaik, all my GPU drivers are up to date and I have tried running functions with and without loading the CudaLink package.

The code I am using is pretty straight forward, I don't think the problem is there, but I'll post it anyway in case it is:

This works:

Classify[
    trainingset,
    Method -> "NeuralNetwork"
 ]

Classify[
     trainingset,
     Method -> "NeuralNetwork",
     TargetDevice -> "CPU"
     ]

This does not:

Classify[
     trainingset,
     Method -> "NeuralNetwork",
     TargetDevice -> "GPU"
     ]

I hope someone can help me out.

EDIT: After letting the NetTrain function run for multiple hours, the process finally exits with the following error code:

NetTrain::arrdiv: Training was stopped early because one or more trainable parameters of the net diverged. The net with the lowest validation loss will be returned. To avoid divergence, ensure that the training data has been normalized to have zero mean and unit variance. You can also try specifying a lower learning rate, or use a different optimization method; the (possibly automatic) values used were Method->{ADAM,Beta1->0.9,Beta2->0.999,Epsilon->1/100000,GradientClipping->None,L2Regularization->None,LearningRate->Automatic,LearningRateSchedule->None,WeightClipping->None}, LearningRate->0.001. Alternatively, you can use the "GradientClipping" option to Method to bound the magnitude of gradients during training.
$\endgroup$
7
  • 1
    $\begingroup$ There is example in Tutorials n = 10000; trainingData = RandomReal[1, {n, 4}] -> RandomChoice[{1, 2, 3}, n]; AbsoluteTiming[classifier = Classify[trainingData, Method -> "NeuralNetwork", TargetDevice -> "GPU"]] . I run it and it works in v12.0. But first time we need to restart session since it upload package from the server. $\endgroup$ Feb 10 at 22:17
  • $\begingroup$ Also this example works with v.12.2 but in both versions Classify[] with option TargetDevice ->"CPU" runs faster. $\endgroup$ Feb 10 at 22:32
  • $\begingroup$ I get the same behaviour with your example. Restarting the session does not work for my own code though. I don't think reloading the session every time I want to train is a viable option either way.. $\endgroup$
    – Ruben
    Feb 11 at 14:08
  • $\begingroup$ Did you seen this post mathematica.stackexchange.com/questions/132022/… ? $\endgroup$ Feb 12 at 22:29
  • $\begingroup$ I saw that question. I don't think it's related. $\endgroup$
    – Ruben
    Feb 13 at 1:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.