Skip to main content

cuda

Setting Up RamaLama with CUDA Support on Linux systems

This guide walks through the steps required to set up RamaLama with CUDA support.

Install the NVIDIA Container Toolkit​

Follow the installation instructions provided in the NVIDIA Container Toolkit installation guide.

Installation using dnf/yum (For RPM based distros like Fedora)​

  • Install the NVIDIA Container Toolkit packages
sudo dnf install -y nvidia-container-toolkit
note

The NVIDIA Container Toolkit is required on the host for running CUDA in containers.

note

If the above installation is not working for you and you are running Fedora, try removing it and using the COPR.

Installation using APT (For Debian based distros like Ubuntu)​

  • Configure the Production Repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
  • Update the packages list from the repository
sudo apt-get update
  • Install the NVIDIA Container Toolkit packages
sudo apt-get install -y nvidia-container-toolkit
note

The NVIDIA Container Toolkit is required for WSL to have CUDA resources while running a container.

Setting Up CUDA Support​

For additional information see: Support for Container Device Interface

Generate the CDI specification file​

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

Check the names of the generated devices​

List the available CDI devices:

nvidia-ctk cdi list
INFO[0000] Found 1 CDI devices
nvidia.com/gpu=all
note

Generate a new CDI specification after any configuration change most notably when the driver is upgraded!

Testing the Setup​

Based on this Documentation: Running a Sample Workload

Test the Installation​

Run the following command to verify setup:

podman run --rm --device=nvidia.com/gpu=all fedora nvidia-smi

Expected Output​

Verify everything is configured correctly, with output similar to this:

Thu Dec  5 19:58:40 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.72                 Driver Version: 566.14         CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3080        On  |   00000000:09:00.0  On |                  N/A |
| 34%   24C    P5             31W /  380W |     867MiB /  10240MiB |      7%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A        35      G   /Xwayland                                   N/A      |
|    0   N/A  N/A        35      G   /Xwayland                                   N/A      |
+-----------------------------------------------------------------------------------------+
note

On systems that have SELinux enabled, it may be necessary to turn on the container_use_devices boolean in order to run the nvidia-smi command successfully from a container.

To check the status of the boolean, run the following:

getsebool container_use_devices

If the result of the command shows that the boolean is off, run the following to turn the boolean on:

sudo setsebool -P container_use_devices 1

CUDA_VISIBLE_DEVICES​

RamaLama respects the CUDA_VISIBLE_DEVICES environment variable if it's already set in your environment. If not set, RamaLama will default to using all the GPUs detected by nvidia-smi.

You can specify which GPU devices should be visible to RamaLama by setting this variable before running RamaLama commands:

export CUDA_VISIBLE_DEVICES="0,1"  # Use GPUs 0 and 1
ramalama run granite

This is particularly useful in multi-GPU systems where you want to dedicate specific GPUs to different workloads.

If CUDA_VISIBLE_DEVICES is set to an empty string, RamaLama treats it as unset and follows the default GPU-selection behavior.

export CUDA_VISIBLE_DEVICES=""  # Treated as unset
ramalama run granite

To revert to using all available GPUs, unset the environment variable:

unset CUDA_VISIBLE_DEVICES

Troubleshooting​

CUDA Updates​

On some CUDA software updates, RamaLama stops working complaining about missing shared NVIDIA libraries for example:

ramalama run granite

Returns this error:

crun: cannot stat /lib64/libEGL_nvidia.so.565.77: No such file or directory: OCI runtime attempted to invoke a command that was not found

Because the CUDA version is updated, the CDI specification file needs to be recreated.

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

See Also​

ramalama(1), podman(1)

Jan 2025, Originally compiled by Dan Walsh <dwalsh@redhat.com>