NVIDIA in a non-GNU environment

NVIDIA proprietary drivers on Linux expect a GNU userland and glibc. With some work they can be made to work under musl on Alpine Linux. Other distros may be possible as well, though I have not personally attempted this; glibc can be compiled manually if there is no package or framework to support it already. There is already plenty of documentation on getting this working so I will not dwell on it too much.

• Old notes on the process under Alpine by Arto
• alpine-pkg-glibc Alpine package by sgerrand

Getting drivers working in Docker

The NVIDIA Container Toolkit is a shim Docker runtime that provides basic functionality for the Docker daemon and Docker Compose files to manage how NVIDIA resources are allocated to containers, and it is not actually required to get NVIDIA cards working in containers. This is especially beneficial in non-GNU environments where the Container Toolkit is much more difficult to get running than the drivers.

With a working driver on the host, you can map the required libraries as volumes. This is how Docker alternative Singularity does it.

docker-compose.yml example

This example may not match what is actually installed on your system. Copy paths as shown in nvliblist.conf and feel free to remove any that are not present for your install.

services:
  myservice:
    devices:
      - /dev/nvidia0
    volumes:
      # https://github.com/sylabs/singularity/blob/main/etc/nvliblist.conf
      # binaries, only mount what's needed
      - /usr/bin/nvidia-smi:/usr/bin/nvidia-smi:ro
      - /usr/bin/nvidia-debugdump:/usr/bin/nvidia-debugdump:ro
      - /usr/bin/nvidia-persistenced:/usr/bin/nvidia-persistenced:ro
      - /usr/bin/nvidia-cuda-mps-control:/usr/bin/nvidia-cuda-mps-control:ro
      - /usr/bin/nvidia-cuda-mps-server:/usr/bin/nvidia-cuda-mps-server:ro
      # libs, only mount what exists
      - /usr/lib/libcuda.so:/usr/lib/libcuda.so.1:ro
      - /usr/lib/libEGL.so:/usr/lib/libEGL.so.1:ro
      - /usr/lib/libGLESv1_CM.so:/usr/lib/libGLESv1_CM.so.1:ro
      - /usr/lib/libGLESv2.so:/usr/lib/libGLESv2.so.1:ro
      - /usr/lib/libGL.so:/usr/lib/libGL.so.1:ro
      - /usr/lib/libGLX.so:/usr/lib/libGLX.so.1:ro
      - /usr/lib/libnvcuvid.so:/usr/lib/libnvcuvid.so.1:ro
      - /usr/lib/libnvidia-cfg.so:/usr/lib/libnvidia-cfg.so.1:ro
      - /usr/lib/libnvidia-encode.so:/usr/lib/libnvidia-encode.so.1:ro
      - /usr/lib/libnvidia-fbc.so:/usr/lib/libnvidia-fbc.so.1:ro
      - /usr/lib/libnvidia-ifr.so:/usr/lib/libnvidia-ifr.so.1:ro
      - /usr/lib/libnvidia-ml.so:/usr/lib/libnvidia-ml.so.1:ro
      - /usr/lib/libnvidia-ptxjitcompiler.so:/usr/lib/libnvidia-ptxjitcompiler.so.1:ro
      - /usr/lib/libOpenCL.so:/usr/lib/libOpenCL.so.1:ro
      - /usr/lib/libOpenGL.so:/usr/lib/libOpenGL.so.1:ro
      - /usr/lib/libvdpau_nvidia.so:/usr/lib/libvdpau_nvidia.so.1:ro

Run ldconfig after mounting

You may need to run ldconfig so that the new libraries are discovered. This can be a problem in Docker containers, where container initialization is often frequent, automated, and unmonitored. The quickest but least flexible way to fix this is by injecting a command before the container entrypoint.

Official documentation and files

• Container Toolkit installation guide
• libnvidia-container APT list for Debian 11
  • Good starting off point if you want to browse compiled packages to eg. copy to your own machine.
• GitHub repo, GitHub.io live page
• Driver readme from x86_64 525.85.05, driver install file for x86_64 525.85.05
  • The filename convention seems to stay consistent, so you can replace your desired driver version in the URL to get the README/direct download link without having to trawl through NVIDIA’s site. The README in particular contains a complete driver compatibility list for the current driver series and all prior drivers. This list is much more useful than NVIDIA’s feature matrices or driver compatibility lists on their website, which are often incomplete.
• CUDA installation guide
  • Not covered by this document. Arto has notes on it.

See also

This document was inspired by my adventures trying to get an old Fermi card working with Frigate, and the lack of documentation on the process.