--- tags: - "\U0001F3F7️topic/\U0001F4BEsoftware/docker" - "\U0001F4D1context/\U0001F5D3️date/edited/2024-06-27" - "\U0001F4D1context/\U0001F5D3️date/published/2023-02-08" - "\U0001F6A9purpose/\U0001F5FA️guide" - "\U0001F3F7️topic/\U0001F4BEsoftware/linux" --- # NVIDIA in a Non-GNU Environment NVIDIA proprietary drivers on Linux expect a GNU userland and `glibc`. With some effort, they can be made to work under `musl` on Alpine Linux. Other distros may be possible as well, though I have not personally attempted this. Alpine provides packages for `glibc` compatibility, so you’ll likely need to compile it manually if there is no package or framework to support it already for your distro of choice. There is already plenty of documentation on getting this working, and I provide some of those resources below, so I will not dwell on it too much. ## Resources - [`alpine-pkg-glibc` package to run `glibc` on Alpine](https://github.com/sgerrand/alpine-pkg-glibc) - Sasha Gerrand, the author of `alpine-pkg-glibc`, is a huge slacker and ~~[hasn’t built a new `glibc` in a while](https://github.com/sgerrand/alpine-pkg-glibc/issues/210)~~ has completely disabled issues and *still* hasn’t updated the package. Since I [build my own packages](https://storage.sev.monster/alpine/edge/testing/x86_64/) when I need them, feel free to use my release of the package if Sasha hasn’t built a more recent version and you need it. ~~[Here’s a script to install my repo, keys, and the package.](https://github.com/sgerrand/alpine-pkg-glibc/issues/210#issuecomment-1841801227)~~ I have since forked Sasha’s package and provide builds [here](https://git.sev.monster/sev/alpine-pkg-glibc). - [Driver readme from x86_64 525.85.05](https://us.download.nvidia.com/XFree86/Linux-x86_64/525.85.05/README/index.html), [driver install file for x86_64 525.85.05](https://us.download.nvidia.com/XFree86/Linux-x86_64/525.85.05/NVIDIA-Linux-x86_64-525.85.05.run) - The filename convention seems to stay consistent, so you can replace your desired driver version in the URL to get the README/direct download link without having to trawl through NVIDIA’s site. The README in particular contains a *complete driver compatibility list* for the current driver series and all prior drivers. This list is much more useful than NVIDIA’s feature matrices or driver compatibility lists on their website, which are often incomplete with regard to older versions. - [CUDA installation guide](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/) - Not covered by this document. [Arto has notes on it.](https://web.archive.org/web/20241010004759/[https://arto.s3.amazonaws.com/notes/cuda)](https://arto.s3.amazonaws.com/notes/cuda)) ## Getting drivers working in Docker The [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#getting-started) is a shim Docker runtime, and NVIDIA expects all Docker environments to use it. It works identically to the usual runtime in most respects, except that it exposes some configuration settings to manage how NVIDIA resources are allocated, and pushes up-to-date libraries into the container. Crucially, it is not actually required to get NVIDIA cards working in containers and is only really useful for the stated purpose. This is especially beneficial in non-GNU environments where the Container Toolkit is much more difficult to get running than the drivers. With a working driver on the host, you can map the required libraries as volumes manually, without the Container Toolkit. [This is how Docker alternative Singularity does it.](https://docs.sylabs.io/guides/3.6/user-guide/gpu.html#library-search-options) ### `docker-compose.yml` example This example may not match what is actually installed on your system. Copy paths as shown in [`nvliblist.conf`](https://github.com/sylabs/singularity/blob/main/etc/nvliblist.conf) and feel free to remove any that are not present for your install. ```yaml services: myservice: devices: - /dev/nvidia0 # make card available volumes: # https://github.com/sylabs/singularity/blob/main/etc/nvliblist.conf # binaries, only mount what's needed - /usr/bin/nvidia-smi:/usr/bin/nvidia-smi:ro - /usr/bin/nvidia-debugdump:/usr/bin/nvidia-debugdump:ro - /usr/bin/nvidia-persistenced:/usr/bin/nvidia-persistenced:ro - /usr/bin/nvidia-cuda-mps-control:/usr/bin/nvidia-cuda-mps-control:ro - /usr/bin/nvidia-cuda-mps-server:/usr/bin/nvidia-cuda-mps-server:ro # libs, only mount what exists - /usr/lib/libcuda.so:/usr/lib/libcuda.so.1:ro - /usr/lib/libEGL.so:/usr/lib/libEGL.so.1:ro - /usr/lib/libGLESv1_CM.so:/usr/lib/libGLESv1_CM.so.1:ro - /usr/lib/libGLESv2.so:/usr/lib/libGLESv2.so.1:ro - /usr/lib/libGL.so:/usr/lib/libGL.so.1:ro - /usr/lib/libGLX.so:/usr/lib/libGLX.so.1:ro - /usr/lib/libnvcuvid.so:/usr/lib/libnvcuvid.so.1:ro - /usr/lib/libnvidia-cfg.so:/usr/lib/libnvidia-cfg.so.1:ro - /usr/lib/libnvidia-encode.so:/usr/lib/libnvidia-encode.so.1:ro - /usr/lib/libnvidia-fbc.so:/usr/lib/libnvidia-fbc.so.1:ro - /usr/lib/libnvidia-ifr.so:/usr/lib/libnvidia-ifr.so.1:ro - /usr/lib/libnvidia-ml.so:/usr/lib/libnvidia-ml.so.1:ro - /usr/lib/libnvidia-ptxjitcompiler.so:/usr/lib/libnvidia-ptxjitcompiler.so.1:ro - /usr/lib/libOpenCL.so:/usr/lib/libOpenCL.so.1:ro - /usr/lib/libOpenGL.so:/usr/lib/libOpenGL.so.1:ro - /usr/lib/libvdpau_nvidia.so:/usr/lib/libvdpau_nvidia.so.1:ro ``` #### Run `ldconfig` after mounting You may need to run [`ldconfig`](https://www.thegeekdiary.com/how-to-use-ldconfig-command-in-linux/) so that the new libraries are discovered. This can be a problem in Docker containers, where container initialization is often frequent, automated, and unmonitored. The quickest but least flexible way to fix this is by [injecting a command before the container entrypoint](../notes/Docker%20entrypoint%20injection.md). ### Official documentation and files - [Container Toolkit installation guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#installation-guide) - [`libnvidia-container` APT list for Debian 11](https://nvidia.github.io/libnvidia-container/debian11/libnvidia-container.list) - Good starting off point if you want to browse compiled packages to eg. copy to your own machine. - [GitHub repo](https://github.com/NVIDIA/libnvidia-container), [GitHub.io live page](https://nvidia.github.io/libnvidia-container/) # See also This document was inspired by my adventures trying to [get an old Fermi card working with Frigate](https://github.com/blakeblackshear/frigate/issues/4716), and the lack of documentation on the process.