8
$\begingroup$

It seems that GPU training in Mathematica has some additional requirements on the software. For example, when I run this example on a Linux system, I get the following errors

trainingData = 
  RandomReal[1, {10000, 4}] -> RandomReal[1, {10000, 4}];
net = NetChain[{8, 4}];
NetTrain[net, trainingData, TargetDevice -> "GPU"]

[13:35:27] /home/dszeto/Desktop/mxnet0932_64/dmlc-core/include/dmlc/./logging.h:300: [13:35:27] /home/dszeto/Desktop/mxnet0932_64/src/storage/storage.cc:38: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading CUDA: CUDA driver version is insufficient for CUDA runtime version

The system has a K40 GPU and CUDA 7.5 installed and has no problem running GPU version of tensorflow. Here is the detailed information:

$ uname -m && cat /etc/*release x86_64
LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Red Hat Enterprise Linux Server release 6.8 (Santiago) Red Hat
Enterprise Linux Server release 6.8 (Santiago) 
$ gcc --version gcc
(GCC) 4.9.0 
$ uname -r
2.6.32-642.11.1.el6.x86_64
In[1]:= $Version
Out[1]:= 11.1.0 for Linux x86 (64-bit) (March 13, 2017)

From the error message, it seems that Mathematica is using the CUDA library that is newer than the driver on the system. However, the CUDA library on the system should be compatible with the GPU driver, since the GPU training in tensorflow works well. Does Mathematica use its own CUDA library rather than the CUDA library on the system? What are the software requirements such as:

  1. GCC
  2. graphics driver
  3. cuDNN
  4. CUDA
  5. MXNet

It seems that Mathematica comes with an MXNet library (in folder SystemFiles/Components/MXNetLink/LibraryResources/) but the others?

Moreover, what are the hierarchy structures of these software components that the Mathematica neural network framework builds on?

Improve this question
$\endgroup$
13
  • 1
    $\begingroup$ Error says CUDA driver version is insufficient for CUDA runtime version so start by updating your drivers to the latest version. Check directly on Nvidia's site for the latest version because sometime the OS check does not give the latest version. Also check that your GPU has compute capability 3 or higher with CUDAInformation. $\endgroup$
    – Edmund
    Commented Apr 17, 2017 at 21:12
  • 1
    $\begingroup$ Possible duplicate of TargetDevice->"GPU" fails even though a CUDA GPU exists $\endgroup$
    – Edmund
    Commented Apr 17, 2017 at 21:13
  • 1
    $\begingroup$ @Edmund The GPU in tensorflow works well, so that means the CUDA and the GPU driver on the system are compatible with each other. I'm wondering whether it is the case that Mathematica uses its own CUDA library instead of the one on the system. $\endgroup$
    – xslittlegrass
    Commented Apr 17, 2017 at 21:40
  • 1
    $\begingroup$ @xslittlegrass I believe it does. It sounds like Mathematica may be trying to use the wrong CUDA version. I've asked other people in the company to take a look at this thread and comment. $\endgroup$
    – Taliesin Beynon
    Commented Apr 18, 2017 at 12:17
  • 2
    $\begingroup$ @xslittlegrass: this should definitely work, as we ship the CUDA drivers + appropriate libraries. Can you try on 11.1.1, and see whether its still broken? (we rebuilt the libraries for 11.1.1) If it is, then I will contact you next week to try and resolve this together (if you have time). Its hard to debug this without a machine where this is failing... $\endgroup$
    – Sebastian
    Commented Apr 22, 2017 at 16:55

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.