1
$\begingroup$

I'm trying to run Mathematica on Azure instance with Tesla K80.
Setting TargetDevice->"GPU" produces the following set of errors for documentation page of NetTrain: errors 1-3

First error is intermittent, and so is second. But third one persists, and I'm sure that this is a bug since I've trained all the nets on this GPU!

I'm unsure what exactly is happening here. Steps I've tried, but have failed:

  1. Reinstalling Mathematica - doesn't work
  2. Installing Mathematica on a new machine - this works till I open my other notebook (containing custom nets) and run NetTrain on those nets.
  3. [Szabolcs's suggestion] Removing the Paclet/folder NeuralNetworks from C:\Program Files\Wolfram Research\Wolfram Desktop\12.1\SystemFiles\Components - this throws a different error:
    error4
  4. Installing Mathematica again on a new instance having no influence of my notebooks - this still throws the wrong runtime loaded error.

Update 1 [problem persists]

I downgraded from 12.1.1 to 12.0.0, and saw that above simple NetTrain works on GPU! Since nets created using newer versions can not be imported, I tried running NetTrain on my nets to train them again, but was greeted by following error: enter image description here

And now I have this new error for the above (first image) simple net from documentation page of NetTrain: enter image description here

More on Update 1's error:

trainingData is just a list of rules where an image maps to string.

On restarting the kernel, loading trainingData and running NetTrain
enter image description here
training proceeds (at least till 1 round).

So naturally I go ahead and add validationData, but I get the following error: enter image description here

First error claims that trainingData contains non-images, however that's not true:
enter image description here

enter image description here

enter image description here

And looking at $LastInternalError gives the same output: enter image description here

Now, if I run NetTrain again - I get the exact error shown in Update 1.

Update 2 [problem persists]

This is the error I get on a new Ubuntu 18.04 instance (K80) with Mathematica 12.1.1 installed (with none of my notebooks present on the machine to influence): enter image description here

Update 3 [problem persists]

This is the error I get on a new Windows instance (K80) with Mathematica 12.1.1 installed (with none of my notebooks present on the machine to influence): enter image description here

(Notebook was created using same version - 12.1.1)

Update 4 [problem persists]

I got access to a local machine with GTX 1070, the errors of Update 2 are still shown but just like Update 2 it also returns a trained net.

Conclusion

It turns out that Windows neural network runtime 10600 on Mathematica 12.1.1 doesn't support K80 (actually - has dropped support) at the time of writing.

$\endgroup$
6
  • 2
    $\begingroup$ As a last resort, you can try to delete the directory ~/.Mathematica/Paclets when Mathematica is not running. This will remove all versions of it. Unfortunately, I don't know which paclet it is, otherwise I would suggest just deleting the relevant one. $\endgroup$
    – Szabolcs
    Dec 2, 2020 at 13:26
  • $\begingroup$ Thanks, apparently Mathematica doesn't download/fix removed paclets and throws a new error Get::noopen: Cannot open NeuralNetworks $\endgroup$ Dec 3, 2020 at 5:51
  • $\begingroup$ Are you sure you're using the latest drivers and firmware for your GPU card? $\endgroup$
    – Lou
    Dec 3, 2020 at 7:21
  • $\begingroup$ Yes, Azure "Data Science Virtual Machine - Windows 2019" instance comes with the drivers installed. Plus no change (that I've noticed) has happened within Windows pre and post error. $\endgroup$ Dec 3, 2020 at 7:37
  • $\begingroup$ Did you quit Mathematica before deleting the Paclets directory? Did you remove the entire directory? "Cannot open NeuralNetworks" makes no sense as this is part of the installation ... There may be paclet updates, but there is also a built-in version. $\endgroup$
    – Szabolcs
    Dec 3, 2020 at 10:29

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.