12
$\begingroup$

Context

I am trying to run CUDALink with a

Centos 6.10 Mathematica 12.0 [cuda 10.1]. Host has NVIDIA Corporation GV100GL [Tesla V100 PCIe 32GB] (rev a1) card

Attempt

Needs["CUDALink`"]

fails, so I updated the drivers (following this thread)

CUDAResourcesInstall["/tmp/CUDAResources-Lin64 12.0.303.paclet", Update->True]

Updating from Wolfram Research server ... Updating from Wolfram Research server ... {Paclet[CUDAResources, 12.0.303, <>]}

Unfortunately

CUDAQ[]

still returns

False

while consistently

 CUDAInformation[]

CUDAInformation::invdevnm: CUDA is not supported on device . Refer to CUDALink System Requirements for system requirements.

fails, (which is rather unfortunate given that it is a rather powerful card with 5120 core).

Question

Any idea what the problem is? Doe Mathematica not support Tesla V100 cards?

$\endgroup$
  • 1
    $\begingroup$ I wish mma could install driver automatically $\endgroup$ – user5601 Jan 15 at 16:34
  • $\begingroup$ SystemInstall[“NvidiaDriverAndCUDA”, Version-> Autodetect] $\endgroup$ – user5601 Jan 15 at 16:37
  • $\begingroup$ Lol, no I was saying it would be nice if that command existed $\endgroup$ – user5601 Jan 15 at 16:39
  • $\begingroup$ Do you need to run actual CUDA code or just train neural nets? Training on that GPU works on Linux even though CUDALink does not. Check nvidia-smi while `NetTrain[ .. , Target -> "GPU"] and you'll see the wolfram kernel using the GPU $\endgroup$ – Fortsaint Jan 17 at 11:34
  • $\begingroup$ Of course. You confirm it works then? $\endgroup$ – Fortsaint Jan 20 at 13:52
16
$\begingroup$

The CUDALink package doesn't work right on Linux. For years and years I've been reporting it, they don't fix it. You have to go through some hoops, but when you get it working, it's really sweet! My recommendation:

  • first ensure that your CUDA system itself works, i. e. without M. Compile/run the samples, write your own toy examples, etc.
  • I've been successful adding

export PATH=/usr/local/cuda-10.1/bin:/usr/local/cuda-10.1/NsightCompute-2019.1${PATH:+:${PATH}}

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.1/lib64

export NVIDIA_DRIVER_LIBRARY_PATH=/usr/bin/

export CUDA_LIBRARY_PATH=/usr/lib64/libcuda.so

at the top of my /etc/profile file. I think you can also add that to your user's .bashrc, but I didn't try it, I prefer to have this in the profile.

  • you may need to make some modifications, you're on CentOS, and I'm on Fedora, but it should in principle be the same, because they're compatible .rpm distros. I believe changes to the above shouldn't be necessary on Fedora, CentOS, and RHEL.

  • ensure you have the correct driver installed. There is one that ships with the CUDA installation, but I found this doesn't always update properly through rpmfusion (not NVidia's fault, it's the rpmfusion peeps that don't get this right). So download and install the latest version directly (I prefer to use the .run file directly. If on #rpmfusion they tell you differently, ignore. Use the .run file).

  • now the CUDALink package should be available. But: you still have to keep two things in mind:

    • you need to make a one-time dummy evaluation before you can use CUDAFunctionLoad. Calling it twice or something else (look at the installed compilers, or compiler information) does the trick. You need to do this only once after you loaded the package. But you always have to do this dummy evaluation after you reload the package!

    • for CUDAFunctionLoad you need to specify "CompilerInstallation" -> "/usr/local/cuda-10.1/bin/" and "XCompilerInstallation" -> "/usr/bin/"

in the options.

Now it works like a charm. The possibilities with CUDAFunctionLoad / CUDAFunction are just mind-boggling. This is 99% or more of what I use from that package. I'm thoroughly impressed, it's even easier than using your CUDA code in a file and then compiling it outside of M. That needs tons of other files and dependencies and the proper compilation line and the linker ... CUDAFunctionLoad takes care of all that. I find it the simplest way to work with CUDA, and it directly integrates with your workflow.

If you do it the way I described above you can also ignore all that paclet stuff that M installs, the old version that M installs there ... if you make these links in the profile correctly, you can use the latest and greatest version that you have installed.

Give it a try, let me know where you get stuck.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thank you very much for a detailed answer. Will give it a try soon. $\endgroup$ – chris Jan 16 at 7:41
  • $\begingroup$ a comment to my own answer: you can make it 10.2 for the CUDA version. The question was about 10.1, so I answered for 10.1. But once you install 10.2, just change the links above, and it will work with 10.2. $\endgroup$ – Andreas Lauschke Jan 22 at 17:22
9
$\begingroup$

In the end, in our case, the following allowed us to get CUDA to work.

export NVIDIA_DRIVER_LIBRARY_PATH /usr/lib64/libnvidia-tls.so.418.116.00

(adapt to your usage).
Then in mathematica

Needs["CUDALink`"]

and

CUDAQ[] 

launches the download of about 4.3 GBytes (!!) of data in .Mathematica (most of it in Paclets/Repository/CUDAResources-Lin64-12.0.346/CUDAToolkit), so be patient, but eventually it returns

(* True *)

Note incidentally that the machine obviously needs to be able to access the internet.

Then the sky is the limit :0)

width = 1024; height = 768; 
iconfig = {width, height, 1, 0,   1, 6}; 
config = {0.001, 0.0, 0.0, 0.0, 8.0, 15.0, 10.0, 5.0};
camera = {{2.0, 2.0, 2.0}, {0.0, 0.0, 0.0}}; AppendTo[camera,
  Normalize[camera[[2]] - camera[[1]]]]; AppendTo[camera, 
 0.75*Normalize[
   Cross[camera[[3]], {0.0, 1.0, 0.0}]]]; AppendTo[camera, 
 0.75*Normalize[Cross[camera[[4]], camera[[3]]]]];
config = Join[{config, Flatten[camera]}];
 pixelsMem = CUDAMemoryAllocate["Float", {height, width, 3}];
srcf = 
 FileNameJoin[{$CUDALinkPath, "SupportFiles", "mandelbulb.cu"}];
mandelbulb = 
 CUDAFunctionLoad[{srcf}, 
  "MandelbulbGPU", {{"Float", _, "Output"}, {"Float", _, 
    "Input"}, {"Integer32", _, "Input"}, "Integer32", "Float", 
   "Float"}, {16}, "UnmangleCode" -> False];
mandelbulb[pixelsMem, 
 Flatten[config], iconfig, 0, 0.0, 0.0, {width*height*3}];
pixels = CUDAMemoryGet[pixelsMem];Image[pixels]

Mathematica graphics

And, following @Fortsaint 's comment:

 data = Flatten@Table[{x, y} -> Exp[-Norm[{x, y}]], {x,-3,3,.005}, {y,-3,3,.005}];
 net = NetChain[{32, Tanh, 1}];
 trained = NetTrain[net, data, BatchSize -> 1024, "TargetDevice" -> "GPU"]

Starting training. Optimization Method: ADAM Device: GPU Batch Size: 1024 Batches Per Round: 1409 Training Examples: 1442401 .... Training complete.

which is 16 times faster with the GPU.

|improve this answer|||||
$\endgroup$
  • 1
    $\begingroup$ a) have a look at what paclet was d/led, check the version number. Current CUDA version is 10.2, and the CUDALink package doesn't always d/l the most recent. You have a Tesla V100, that's Volta generation, so would support the "new" tensor cores. Make sure you use a CUDA version that has everything you need. b) Teslas usually run on driver version numbers that are a tad behind. Your driver version is 418.116.00, the most recent driver supporting Tesla cards is 440.33.01 (most recent GTX driver is 440.44). As you are on a Tesla, you may want to upgrade to 440.33.01 and test with that. $\endgroup$ – Andreas Lauschke Jan 20 at 15:04
  • 1
    $\begingroup$ for example, in my paclet dir for M12 I find CUDALink-12.0.0, and in the file version.txt I find it's version 8.0.61. So everything in CUDA after 8.0.61 is missing (again, current version is CUDA 10.2). CUDA 9, 10, 10.1, 10.2, cooperative thread groups, ray-tracing, compiler improvements, ... $\endgroup$ – Andreas Lauschke Jan 20 at 15:30
  • $\begingroup$ @AndreasLauschke thanks for the advice. Will look into performance now. $\endgroup$ – chris Jan 20 at 18:28
7
$\begingroup$

This is a reply to a question from chris, not a "new" answer.

You mean a working example? How about 250 million normally-distributed random numbers, displayed in a histo:

Needs["CUDALink`"]

srcf = FileNameJoin[{$CUDALinkPath, "SupportFiles", "random.cu"}]

CUDAInverseCND = 
 CUDAFunctionLoad[{srcf}, 
  "InverseCND", {{_Real, _, "InputOutput"}, _Integer, _Integer}, 256, 
  "CompilerInstallation" -> "/usr/local/cuda-10.1/bin/", 
  "XCompilerInstallation" -> "/usr/bin/"]

(*evaluate the cell above twice*)

sampleCount = 250000000;
mem = CUDAMemoryAllocate[Real, sampleCount];
CUDAInverseCND[mem, sampleCount, 0];
samples = CUDAMemoryGet[mem];
Histogram[samples, 1000, "ProbabilityDensity"]

this should take two seconds for the CUDA part, the histo will take longer, as it now has to deal with 250 million data points.

you can now analyse the memory with

CUDAMemoryInformation@mem

the important parts are:

HostStatus->Synchronized,DeviceStatus->Synchronized,Residence->DeviceHost,Sharing->Shared,Type->Double,ByteCount->2000000000,Dimensions->{250000000}

when HostStatus and DeviceStatus are both synchronized, all is good. Before the CUDAMemoryGet CUDAMemoryInformation@mem will say HostStatus->Unsynchronized -- no memcopy to the host happened yet.

when done you should

CUDAMemoryUnload@mem

to free the memory on your GPU (this takes about 2 GB of GPU memory)

HTH

|improve this answer|||||
$\endgroup$
5
$\begingroup$

The automatic detection has been improved with a paclet update for version 12.0, so after doing

Map[PacletSiteUpdate, PacletSites[]];

PacletUpdate["GPUTools"]

Quit 

it will no longer be necessary to set NVIDIA_DRIVER_LIBRARY_PATH by hand.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.