ramalama/docs/readme/wsl2-docker-cuda.md

Setting Up RamaLama with CUDA Support in WSL2 Using Docker

This guide will walk you through the steps required to set up RamaLama in WSL2 with CUDA support using Docker.

Prerequisites

1. NVIDIA Game-Ready Drivers Ensure that you have the appropriate NVIDIA game-ready drivers installed on your Windows system. This is necessary for CUDA support in WSL2.

2. Docker Desktop Installation Install Docker Desktop and in settings under General, ensure the option to Use the WSL 2 based engine is checked.

Install the NVIDIA Container Toolkit

Follow the installation instructions provided in the NVIDIA Container Toolkit installation guide.

---

Installation using APT (For Distros like Ubuntu)

1. Configure the Production Repository:

   curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
   sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
   
   curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
   sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
   sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
   
   

2. Update the packages list from the repository:

   sudo apt-get update
   

3. Install the NVIDIA Container Toolkit packages:

   sudo apt-get install -y nvidia-container-toolkit
   

Note: The Nvidia Container Toolkit is required for WSL to have CUDA resources while running a container.

Testing the Setup

Based on this Documentation: Running a Sample Workload

---

1. Test the Installation Run the following command to verify your setup:

   docker run --rm --gpus all ubuntu nvidia-smi
   

2. Expected Output If everything is set up correctly, you should see an output similar to this:

      Thu Dec  5 19:58:40 2024
   +-----------------------------------------------------------------------------------------+
   | NVIDIA-SMI 565.72                 Driver Version: 566.14         CUDA Version: 12.7     |
   |-----------------------------------------+------------------------+----------------------+
   | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
   |                                         |                        |               MIG M. |
   |=========================================+========================+======================|
   |   0  NVIDIA GeForce RTX 3080        On  |   00000000:09:00.0  On |                  N/A |
   | 34%   24C    P5             31W /  380W |     867MiB /  10240MiB |      7%      Default |
   |                                         |                        |                  N/A |
   +-----------------------------------------+------------------------+----------------------+
   
   +-----------------------------------------------------------------------------------------+
   | Processes:                                                                              |
   |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
   |        ID   ID                                                               Usage      |
   |=========================================================================================|
   |    0   N/A  N/A        35      G   /Xwayland                                   N/A      |
   |    0   N/A  N/A        35      G   /Xwayland                                   N/A      |
   +-----------------------------------------------------------------------------------------+
   

Installing Nvidia CUDA Toolkit (Optional)

1. Install the CUDA Toolkit Follow the instructions in the NVIDIA CUDA WSL User Guide to install the CUDA toolkit.

   • Download the CUDA Toolkit Visit the NVIDIA CUDA Downloads page to download the appropriate version for WSL-Ubuntu.

2. Remove Existing Keys (if needed) Run this command to remove any old keys that might conflict:

   sudo apt-key del 7fa2af80
   

3. Select Your Environment Head back to the NVIDIA CUDA Downloads page and choose the environment that fits your setup. Follow the installation instructions to install the CUDA package (deb format is recommended).

Note: The Nvidia Cuda Toolkit enables the container runtime to build containers with CUDA.