How to use an external GPU to train neural networks in Mathematica?

Question

Taking steps in the uncharted territories of eGPUs, I have installed an NVIDIA GeForce Titan Xp GPU on an eGPU enclosure (Mantiz MZ-02) and have connected it to my laptop (HP Spectre x360 - 15-bl075nr running Windows 10) using Thunderbolt 3. It seems my machine recognizes the device (since the name of the GPU is correctly listed?!). However, when I want to train a neural network using the external GPU, it gives me the following error (the code is from Mathematica Documentation and is self-contained).

Input:

n = 1000;
trainingData = RandomReal[1, {n, 4}] -> RandomReal[1, {n, 4}];
net = NetChain[{8, Tanh, 2048, Tanh, 2048, Tanh, 4}];
AbsoluteTiming[
 trained = 
  NetTrain[net, trainingData, TargetDevice -> {"GPU", 2}, 
   MaxTrainingRounds -> 10]
 ] 

Output Error:

NetTrain::badtrgdev: TargetDevice -> {GPU,2} could not be used, please ensure that you have a compatible NVIDIA graphics card and have installed the latest operating system drivers.

My laptop does have an internal GPU (GeForce 940MX) and when I change TargetDevice -> {"GPU", 2} to TargetDevice -> {"GPU", 1}, everything works fine. How can I address this error?

Update

Good news, I was able to run the above code on a MacBook Pro (with Touch Bar) using my eGPU configuration. But when I run Needs["CUDALink`"] followed by CUDAQ[] to list all of my GPUs using SystemInformation[] the Mathematica kernel crashes on my laptop, so I am not able to ensure that it recognizes the GPU or not. But when I unplug the eGPU and restart Mathematica, QUDAQ[] returns True.

Answer 1

Go to the NVIDIA Drivers Download page and download and install the correct driver for your GPU and system.

Open a new Mathematica notebook and ensure that the GPUs can be detected. To do so, run Needs["CUDALink`"] and check if it works properly by executing CUDAQ[]. This might take about 10 minutes if you are executing it for the first time. If it collapses ONLY when your eGPU is plugged in, then it's because you don't have the right driver for your GPU. If it collapses with and without a GPU, then your paclets are messed up (easiest fix for me was FULLY uninstalling and reinstalling Mathematica by deleting the correct directories after uninstalling it (see this page)). These steps are not necessary to run neural network related function, but so you can list ALL of your GPUs.

Now, run SystemInformation[] and navigate to Links and then to CUDA to make sure that Mathematica sees your GPU. The GPU number that you must use in TargetDevice -> {"GPU", n} in replacement for n is listed there. If your GPU shows up, you are good to go.

Extra: For those of you who might be interested the eGPU runs 350% times faster than the internal GPU and the eGPU is 1620% faster than the Core i7 CPU. These were calculated using the code in the question above.