x

NVIDIA in a non-GNU environment

NVIDIA proprietary drivers on Linux expect a GNU userland and glibc. With some effort, they can be made to work under musl on Alpine Linux. Other distros may be possible as well, though I have not personally attempted this; glibc can be compiled manually if there is no package or framework to support it already. There is already plenty of documentation on getting this working, and I provide some of those resources below, so I will not dwell on it too much.

Resources

Getting drivers working in Docker

The NVIDIA Container Toolkit is a shim Docker runtime, and NVIDIA expects all Docker environments to use it. It works identically to the usual runtime in most respects, except that it exposes some configuration settings to manage how NVIDIA resources are allocated, and pushes up-to-date libraries into the container. Crucially, it is not actually required to get NVIDIA cards working in containers and is only really useful for the stated purpose. This is especially beneficial in non-GNU environments where the Container Toolkit is much more difficult to get running than the drivers.

With a working driver on the host, you can map the required libraries as volumes manually, without the Container Toolkit. This is how Docker alternative Singularity does it.

docker-compose.yml example

This example may not match what is actually installed on your system. Copy paths as shown in nvliblist.conf and feel free to remove any that are not present for your install.

services:
  myservice:
    devices:
      - /dev/nvidia0 # make card available
    volumes:
      # https://github.com/sylabs/singularity/blob/main/etc/nvliblist.conf
      # binaries, only mount what's needed
      - /usr/bin/nvidia-smi:/usr/bin/nvidia-smi:ro
      - /usr/bin/nvidia-debugdump:/usr/bin/nvidia-debugdump:ro
      - /usr/bin/nvidia-persistenced:/usr/bin/nvidia-persistenced:ro
      - /usr/bin/nvidia-cuda-mps-control:/usr/bin/nvidia-cuda-mps-control:ro
      - /usr/bin/nvidia-cuda-mps-server:/usr/bin/nvidia-cuda-mps-server:ro
      # libs, only mount what exists
      - /usr/lib/libcuda.so:/usr/lib/libcuda.so.1:ro
      - /usr/lib/libEGL.so:/usr/lib/libEGL.so.1:ro
      - /usr/lib/libGLESv1_CM.so:/usr/lib/libGLESv1_CM.so.1:ro
      - /usr/lib/libGLESv2.so:/usr/lib/libGLESv2.so.1:ro
      - /usr/lib/libGL.so:/usr/lib/libGL.so.1:ro
      - /usr/lib/libGLX.so:/usr/lib/libGLX.so.1:ro
      - /usr/lib/libnvcuvid.so:/usr/lib/libnvcuvid.so.1:ro
      - /usr/lib/libnvidia-cfg.so:/usr/lib/libnvidia-cfg.so.1:ro
      - /usr/lib/libnvidia-encode.so:/usr/lib/libnvidia-encode.so.1:ro
      - /usr/lib/libnvidia-fbc.so:/usr/lib/libnvidia-fbc.so.1:ro
      - /usr/lib/libnvidia-ifr.so:/usr/lib/libnvidia-ifr.so.1:ro
      - /usr/lib/libnvidia-ml.so:/usr/lib/libnvidia-ml.so.1:ro
      - /usr/lib/libnvidia-ptxjitcompiler.so:/usr/lib/libnvidia-ptxjitcompiler.so.1:ro
      - /usr/lib/libOpenCL.so:/usr/lib/libOpenCL.so.1:ro
      - /usr/lib/libOpenGL.so:/usr/lib/libOpenGL.so.1:ro
      - /usr/lib/libvdpau_nvidia.so:/usr/lib/libvdpau_nvidia.so.1:ro

Run ldconfig after mounting

You may need to run ldconfig so that the new libraries are discovered. This can be a problem in Docker containers, where container initialization is often frequent, automated, and unmonitored. The quickest but least flexible way to fix this is by injecting a command before the container entrypoint.

Official documentation and files

See also

This document was inspired by my adventures trying to get an old Fermi card working with Frigate, and the lack of documentation on the process.

Left-click: follow link, Right-click: select node, Scroll: zoom
x