> Top-down: user's APP ->`libcudart.so` -> `libcuda.so` -> `nvidia.ko` > > - CUDA $\approx$ CUDA toolkit $\approx$ CUDA runtime > - When it comes to *CUDA version* > - CUDA toolkit $\supseteq$ CUDA runtime > - CUDA runtime =`libcudart.so` > - CUDA driver = user-mode driver =`libcuda.so` > - GPU driver = kernel-mode driver = `nvidia.ko` > - Driver = CUDA driver + GPU driver = nvidia driver Nvidia's CUDA toolkit & docker images comes with [corresponding defaut CUDA driver](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#id4) So newer CUDA(`libcudart.so` + `libcuda.so`) in an official docker images run on old `nvidia.ko` driver(which means `libcuda.so` not match `nvidia.ko`) will cause [forward compatibility](#forward-compatibility) which is not support by non-server GPUs. **Solutions**: 1. . manually link to correct older CUDA driver, make `libcuda.so` in container match `nvidia.ko` in host machine - Nvidia docker container will auto mount the dir. in host below - `ln -sf /usr/lib/x86_64-linux-gnu/libcuda.so.460.80 /usr/lib/x86_64-linux-gnu/libcuda.so.1` 2. Using base image w/ no `libcuda.so` & install pytorch by conda/micromamba package manager, it will resolve & install proper `libcuda.so` as dependency of pytorch # Backward compatibility new `libcudart.so` can run user's APP build against any old CUDA toolkit[^backward] # Minor Version Compatibility Since CUDA 11, all minor CUDA version(e.g. 11.*) support the same driver - CUDA/`libcudart.so` *at least* compatible with old driver > certain version (but w/ limited feature-set) [^minor] [CUDA minimum required driver version table](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#id3) # Forward compatibility - Can update only CUDA driver - ==Works only for data center GPU==, Other can only update the whole driver (`libcuda.so`+`nvidia.ko`) ![forward-compatibility-upgrade-path](https://docs.nvidia.com/deploy/cuda-compatibility/graphics/forward-compatibility-upgrade-path.png) #cuda [^backward]: [Backward compatability](https://docs.nvidia.com/deploy/cuda-compatibility/index.html#default-to-minor-version__docs-internal-guid-024277d8-7fff-71f1-a89e-db0b56eee63d) [^minor]: [Minor Version Compatibility](https://docs.nvidia.com/deploy/cuda-compatibility/index.html#minor-version-compatibility)