> Top-down: user's APP ->`libcudart.so` -> `libcuda.so` -> `nvidia.ko`
>
> - CUDA $\approx$ CUDA toolkit $\approx$ CUDA runtime
> - When it comes to *CUDA version*
> - CUDA toolkit $\supseteq$ CUDA runtime
> - CUDA runtime =`libcudart.so`
> - CUDA driver = user-mode driver =`libcuda.so`
> - GPU driver = kernel-mode driver = `nvidia.ko`
> - Driver = CUDA driver + GPU driver = nvidia driver
Nvidia's CUDA toolkit & docker images comes with [corresponding defaut CUDA driver](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#id4)
So newer CUDA(`libcudart.so` + `libcuda.so`) in an official docker images run on old `nvidia.ko` driver(which means `libcuda.so` not match `nvidia.ko`) will cause [forward compatibility](#forward-compatibility) which is not support by non-server GPUs.
**Solutions**:
1. . manually link to correct older CUDA driver, make `libcuda.so` in container match `nvidia.ko` in host machine
- Nvidia docker container will auto mount the dir. in host below
- `ln -sf /usr/lib/x86_64-linux-gnu/libcuda.so.460.80 /usr/lib/x86_64-linux-gnu/libcuda.so.1`
2. Using base image w/ no `libcuda.so` & install pytorch by conda/micromamba package manager, it will resolve & install proper `libcuda.so` as dependency of pytorch
# Backward compatibility
new `libcudart.so` can run user's APP build against any old CUDA toolkit[^backward]
# Minor Version Compatibility
Since CUDA 11, all minor CUDA version(e.g. 11.*) support the same driver
- CUDA/`libcudart.so` *at least* compatible with old driver > certain version (but w/ limited feature-set) [^minor]
[CUDA minimum required driver version table](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#id3)
# Forward compatibility
- Can update only CUDA driver
- ==Works only for data center GPU==, Other can only update the whole driver (`libcuda.so`+`nvidia.ko`)

#cuda
[^backward]: [Backward compatability](https://docs.nvidia.com/deploy/cuda-compatibility/index.html#default-to-minor-version__docs-internal-guid-024277d8-7fff-71f1-a89e-db0b56eee63d)
[^minor]: [Minor Version Compatibility](https://docs.nvidia.com/deploy/cuda-compatibility/index.html#minor-version-compatibility)