Setting up CUDA on WSL2 in 2023

Setting up CUDA on WSL2 probably costs you a little bit of performance but gives you a lot of flexibility - you're essentially running a VM with (some) Microsoft support. Here's the best source: Enable NVIDIA CUDA in WSL.

Luckily in the past year or so NVIDIA has started supporting this path, so you're (kind of) in good hands. A lot of advice will be mixed between Ubuntu and WSL, though, so I'm writing down my experience with installing CUDA on WSL2 today.

The below is a bit of a bloated install, so you might be able to get away with fewer things (for example TensorRT is pretty optional).

Install drivers

First, you'll need to get an NVIDIA driver (in Windows) for your card. I got the "NVIDIA Game Ready" driver from NVIDIA's website directly. Note that this is not installing an NVIDIA driver for your WSL (i.e. Linux) installation. As far as I can tell, this process uses your Windows driver directly (which makes sense, having two drivers for the same physical device running seems like a recipe for disaster).

I chose:

• No GEForce Experience
• HD Audio Driver
• PhysX

though of course you can make other choices. Once this is finished, you will see a bunch of files in C:\Windows\System32\lxss\lib\:

~ ❯ ls -l /mnt/c/Windows/System32/lxss/lib/
-r-xr-xr-x 1 singhrac singhrac 10188744 Aug 15 00:23 libcudadebugger.so.1
-r-xr-xr-x 1 singhrac singhrac   154088 Aug 15 00:23 libcuda.so
-r-xr-xr-x 1 singhrac singhrac   154088 Aug 15 00:23 libcuda.so.1
-r-xr-xr-x 1 singhrac singhrac   154088 Aug 15 00:23 libcuda.so.1.1
-r-xr-xr-x 3 singhrac singhrac  5401440 Aug 11 11:05 libd3d12core.so
-r-xr-xr-x 3 singhrac singhrac   800296 Aug 11 11:05 libd3d12.so
-r-xr-xr-x 2 singhrac singhrac   827904 Jun  5  2021 libdxcore.so
-r-xr-xr-x 1 singhrac singhrac 10820792 Aug 15 00:23 libnvcuvid.so
-r-xr-xr-x 1 singhrac singhrac 10820792 Aug 15 00:23 libnvcuvid.so.1
-r-xr-xr-x 1 singhrac singhrac 37086552 Aug 15 00:23 libnvdxdlkernels.so
-r-xr-xr-x 1 singhrac singhrac   551528 Aug 15 00:23 libnvidia-encode.so
-r-xr-xr-x 1 singhrac singhrac   551528 Aug 15 00:23 libnvidia-encode.so.1
-r-xr-xr-x 1 singhrac singhrac   233832 Aug 15 00:23 libnvidia-ml.so.1
-r-xr-xr-x 1 singhrac singhrac   362960 Aug 15 00:23 libnvidia-opticalflow.so
-r-xr-xr-x 1 singhrac singhrac   362960 Aug 15 00:23 libnvidia-opticalflow.so.1
-r-xr-xr-x 1 singhrac singhrac    68560 Aug 15 00:23 libnvoptix.so.1
-r-xr-xr-x 1 singhrac singhrac 83265296 Aug 15 00:23 libnvwgf2umx.so
-r-xr-xr-x 1 singhrac singhrac   678064 Aug 15 00:23 nvidia-smi

and the same set of files in /usr/lib/wsl/lib/. NOTE! This is the location that the CUDA Runtime will live. However, you also need the CUDA Toolkit (probably). Note that nvidia-smi will now work if you add /usr/lib/wsl/lib/ to your path:

~ ❯ nvidia-smi
Thu Aug 24 11:17:06 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.103                Driver Version: 537.13       CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2070        On  | 00000000:07:00.0  On |                  N/A |
|  0%   51C    P5              20W / 175W |    573MiB /  8192MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

However the "CUDA Version" at the top is the max supported CUDA Toolkit version that you can use¹. At this point in time (20230824) PyTorch doesn't really support CUDA 12.2, so it might be a huge pain in the ass if you use that. I recommend 11.8, the currently max supported version on the install page. I'm sure this will change shortly.

At this point, you probably want to add /usr/lib/wsl/lib/ to your LD_LIBRARY_PATH, which will let programs find the .so files there. Put this in your ~/.bashrc or ~/.zshrc:

export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH

If you run ldconfig -p | grep libcuda, this will now show those libraries (you might have to exec zsh or restart your shell).

Install CUDA Toolkit

It is really important to listen to NVIDIA's instructions here: the default CUDA Toolkit comes with a driver, which will overwrite the above set (i.e. piggy-backing off Windows driver). So it's important to install the WSL-specific CUDA Toolkit, which doesn't have that driver. This is also where it's really easy to install the latest CUDA Toolkit (i.e. 12.2), which might cause issues for you. I didn't run into too many the first time, but I think it's easier just to install the older toolkit.

Go here: CUDA Toolkit 11.8. You can find other versions in the "Archive of Previous CUDA Releases" link.

You want to choose: Linux > x86_64 > WSL-Ubuntu > 2.0 > (your choice of installer type)

I personally find it easier to deal with deb installation because then things like dpkg will help you manage the installation, but everyone's experience is different. In any case, you want to follow those instructions, making sure you have WSL-Ubuntu selected. If you use the deb (network) install, it might try to install 12.2 because that's the most recent version. I think you can work around this by doing sudo apt-get install cuda-toolkit-11-8, but I used the deb (local) version instead.

I personally ran:

~ ❯ wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
~ ❯ sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
~ ❯ wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda-repo-wsl-ubuntu-11-8-local_11.8.0-1_amd64.deb
~ ❯ sudo dpkg -i cuda-repo-wsl-ubuntu-11-8-local_11.8.0-1_amd64.deb
~ ❯ sudo cp /var/cuda-repo-wsl-ubuntu-11-8-local/cuda-*-keyring.gpg /usr/share/keyrings/
~ ❯ sudo apt-get update
~ ❯ sudo apt-get -y install cuda-toolkit-11-8

Note the change in the last line - I wanted to pin the specific version. This installs a lot of stuff for you: nvprof, nvcc, nvprune, libcublas-dev, libcufft, libcusparse.

~ ❯ ls -l /usr/local
total 36
drwxr-xr-x  2 root root 4096 Aug 24 11:48 bin
lrwxrwxrwx  1 root root   22 Aug 24 11:48 cuda -> /etc/alternatives/cuda
lrwxrwxrwx  1 root root   25 Aug 24 11:48 cuda-11 -> /etc/alternatives/cuda-11
drwxr-xr-x 15 root root 4096 Aug 24 11:48 cuda-11.8
drwxr-xr-x  2 root root 4096 Feb 10  2023 etc
drwxr-xr-x  2 root root 4096 Feb 10  2023 games
drwxr-xr-x  2 root root 4096 Feb 10  2023 include
drwxr-xr-x  3 root root 4096 Feb 10  2023 lib
lrwxrwxrwx  1 root root    9 Feb 10  2023 man -> share/man
drwxr-xr-x  2 root root 4096 Feb 10  2023 sbin
drwxr-xr-x  6 root root 4096 Mar  7 17:10 share
drwxr-xr-x  2 root root 4096 Feb 10  2023 src

I'm not entirely sure why there's two separate cuda and cuda-11 folders, and they don't seem to be symlinks to each other. This is a bit concerning, but I'm not going to worry about it too much. My ldconfig -p | grep libcublas points to the /usr/local/cuda variant, but when I check nvcc there, it says the version of the toolkit is 11.8.

Note, however, that it doesn't install cuDNN or TensorRT (aka nvinfer). So let's do that next.

cuDNN and TensorRT

NVIDIA now has a deep learning installation guide here: NVIDIA cuDNN. I personally grabbed cuDNN v8.9.4 for CUDA 11.x, "Local Intaller for Ubuntu22.04 x86_64 (Deb)", and then the rest of the instructions from that linked page worked fine.

Just in case it's unclear, the runtime library installation command was sudo apt-get install libcudnn8=8.9.4.25-1+cuda11.8 (after you've set up the local repository). I didn't install the code samples. After running sudo ldconfig, I was able to find cuDNN using ldconfig -p | grep libcudnn.

For TensorRT, the instructions are a bit confusing but you can basically do the same thing:

~ ❯ sudo dpkg -i nv-tensorrt-local-repo-ubuntu2204-8.6.1-cuda-11.8_1.0-1_amd64.deb
~ ❯ sudo cp /var/nv-tensorrt-local-repo-ubuntu2204-8.6.1-cuda-11.8/nv-tensorrt-local-0628887B-keyring.gpg /usr/share/keyrings/

The latter command is the output of the former, so you don't really need to worry about version information - just download the right .deb from the link, and then copy and paste the keyring command.

From here, I installed the full runtime, but not the Python packages - I like to do that inside virtualenvs (since, for example, you might want a different base Python version).

Python packages

Now you can install torch like you always wanted (in a virtualenv, right?):

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

Make sure you check it works!

~ ❯ python -c "import torch; print(torch.cuda.is_available())"
True

Uninstall

I had to temporarily uninstall as part of messing everything up, and it wasn't too bad. Mostly I purged some apt packages using stuff like:

sudo apt-get --purge remove "cuda*"

and swapped cuda for nvidia and nsight.

This gets rid of most things, but as mentioned above, this will keep nvidia-smi around, since that's from the C:\Windows\System32\lxss\lib\ install, which is kept in /usr/lib/wsl/lib/. To remove those, you need to uninstall the NVIDIA graphics driver from your Windows machine (note for me this caused Alacritty to break, but Powershell let me access WSL just fine).