NVIDIA in a non-GNU environment

NVIDIA proprietary drivers on Linux expect a GNU userland and glibc. With some effort, they can be made to work under musl on Alpine Linux. Other distros may be possible as well, though I have not personally attempted this; glibc can be compiled manually if there is no package or framework to support it already. There is already plenty of documentation on getting this working, and I provide some of those resources below, so I will not dwell on it too much.

Resources

• alpine-pkg-glibc package to run glibc on Alpine
  • Sasha Gerrand, the author of alpine-pkg-glibc, is a huge slacker and hasn’t built a new glibc in a while as of this posting. Since I build my own packages when I need them, feel free to use my release of the package if Sasha hasn’t built a more recent version and you need it. Here’s a script to install my repo, keys, and the package.
• Driver readme from x86_64 525.85.05, driver install file for x86_64 525.85.05
  • The filename convention seems to stay consistent, so you can replace your desired driver version in the URL to get the README/direct download link without having to trawl through NVIDIA’s site. The README in particular contains a complete driver compatibility list for the current driver series and all prior drivers. This list is much more useful than NVIDIA’s feature matrices or driver compatibility lists on their website, which are often incomplete with regard to older versions.
• CUDA installation guide
  • Not covered by this document. Arto has notes on it.

Getting drivers working in Docker

The NVIDIA Container Toolkit is a shim Docker runtime, and NVIDIA expects all Docker environments to use it. It works identically to the usual runtime in most respects, except that it exposes some configuration settings to manage how NVIDIA resources are allocated, and pushes up-to-date libraries into the container. Crucially, it is not actually required to get NVIDIA cards working in containers and is only really useful for the stated purpose. This is especially beneficial in non-GNU environments where the Container Toolkit is much more difficult to get running than the drivers.

With a working driver on the host, you can map the required libraries as volumes manually, without the Container Toolkit. This is how Docker alternative Singularity does it.

docker-compose.yml example

This example may not match what is actually installed on your system. Copy paths as shown in nvliblist.conf and feel free to remove any that are not present for your install.

services:
  myservice:
    devices:
      - /dev/nvidia0 # make card available
    volumes:
      # https://github.com/sylabs/singularity/blob/main/etc/nvliblist.conf
      # binaries, only mount what's needed
      - /usr/bin/nvidia-smi:/usr/bin/nvidia-smi:ro
      - /usr/bin/nvidia-debugdump:/usr/bin/nvidia-debugdump:ro
      - /usr/bin/nvidia-persistenced:/usr/bin/nvidia-persistenced:ro
      - /usr/bin/nvidia-cuda-mps-control:/usr/bin/nvidia-cuda-mps-control:ro
      - /usr/bin/nvidia-cuda-mps-server:/usr/bin/nvidia-cuda-mps-server:ro
      # libs, only mount what exists
      - /usr/lib/libcuda.so:/usr/lib/libcuda.so.1:ro
      - /usr/lib/libEGL.so:/usr/lib/libEGL.so.1:ro
      - /usr/lib/libGLESv1_CM.so:/usr/lib/libGLESv1_CM.so.1:ro
      - /usr/lib/libGLESv2.so:/usr/lib/libGLESv2.so.1:ro
      - /usr/lib/libGL.so:/usr/lib/libGL.so.1:ro
      - /usr/lib/libGLX.so:/usr/lib/libGLX.so.1:ro
      - /usr/lib/libnvcuvid.so:/usr/lib/libnvcuvid.so.1:ro
      - /usr/lib/libnvidia-cfg.so:/usr/lib/libnvidia-cfg.so.1:ro
      - /usr/lib/libnvidia-encode.so:/usr/lib/libnvidia-encode.so.1:ro
      - /usr/lib/libnvidia-fbc.so:/usr/lib/libnvidia-fbc.so.1:ro
      - /usr/lib/libnvidia-ifr.so:/usr/lib/libnvidia-ifr.so.1:ro
      - /usr/lib/libnvidia-ml.so:/usr/lib/libnvidia-ml.so.1:ro
      - /usr/lib/libnvidia-ptxjitcompiler.so:/usr/lib/libnvidia-ptxjitcompiler.so.1:ro
      - /usr/lib/libOpenCL.so:/usr/lib/libOpenCL.so.1:ro
      - /usr/lib/libOpenGL.so:/usr/lib/libOpenGL.so.1:ro
      - /usr/lib/libvdpau_nvidia.so:/usr/lib/libvdpau_nvidia.so.1:ro

Run ldconfig after mounting

You may need to run ldconfig so that the new libraries are discovered. This can be a problem in Docker containers, where container initialization is often frequent, automated, and unmonitored. The quickest but least flexible way to fix this is by injecting a command before the container entrypoint.

Official documentation and files

• Container Toolkit installation guide
• libnvidia-container APT list for Debian 11
  • Good starting off point if you want to browse compiled packages to eg. copy to your own machine.
• GitHub repo, GitHub.io live page

See also

This document was inspired by my adventures trying to get an old Fermi card working with Frigate, and the lack of documentation on the process.