Setting Up Docker with Linux for TensorRT

In this guide, we will set up the Windows Subsystem for Linux (WSL) and Docker to run TensorRT, a high-performance deep learning inference library. This setup will allow you to leverage the power of your NVIDIA GPU within a Linux environment on Windows.

temp saved

About TensorRT

TensorRT is a high-performance deep learning inference library and software development kit (SDK) developed by NVIDIA. It is designed to optimize deep learning models for deployment on NVIDIA GPUs, providing low-latency and high-throughput inference.

It provides significant speedups for deep learning inference, making it suitable for real-time applications such as autonomous driving, natural language processing, and computer vision. TensorRT supports various deep learning frameworks, including TensorFlow, PyTorch, and ONNX (Open Neural Network Exchange). It can import models from these frameworks and optimize them for deployment.

Further Reading

• WSL Installation: Learn how to install and configure WSL.
• Docker backend on WSL Know how to setup the Docker engine with WSL.
• Docker Documentation: Understand how to use Docker for cloud-native applications.
• Nvidia cuDNN: Get detailed information about the CUDA Deep Neural Network library.
• Nvidia TensorRT Docs construct an application running inference on TensorRT engine.
• Nvidia PyTorch Releases: Check out the latest releases & updates for Nvidia PyTorch.

Path Environment Management

Prerequisites

Before proceeding, ensure that your NVIDIA graphics drivers and necessary libraries are properly installed.

Checking NVIDIA Graphics Drivers

To verify that your NVIDIA drivers are correctly installed, run:

nvidia-smi

This command will display information about your GPU and the drivers installed. If the command is not found, you may need to install or update your NVIDIA drivers.

For Ubuntu/Debian users, you can check the installed NVIDIA packages with:

dpkg -l | grep -i nvidia

This will list all NVIDIA-related packages installed on your system.

Checking Zlib Installation

Zlib is a compression library that is required by many applications. To check if Zlib is installed, run:

dpkg -l | grep zlib1g
apt-cache policy zlib1g

These commands will display the version and installation status of the Zlib library on your system.

Downloading and Installing cuDNN for Linux

cuDNN (CUDA Deep Neural Network library) accelerates deep learning frameworks. You need to manually download and install it if it's not included in your Docker image.

1. Download cuDNN:

   • Go to the cuDNN download page and get the latest version.
   • For older versions, visit the cuDNN archive and download the desired version.
   • For 9+ cudnn version, go to this page
     Example download file: cudnn-linux-x86_64-8.9.7.29_cuda12-archive.tar

2. Install cuDNN:
   Extract the tarball and copy the necessary files to their respective directories:

   ​​​cp cudnn-*-archive/include/cudnn*.h /usr/local/cuda/include 
   ​​​cp -P cudnn-*-archive/lib/libcudnn* /usr/local/cuda/lib64 
   ​​​chmod a+r /usr/local/cuda/include/cudnn*.h /usr/local/cuda/lib64/libcudnn*
   

   Update your LD_LIBRARY_PATH to include the CUDA libraries:

   ​​​export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda/lib64
   

   Note: Replace /usr/local/cuda/ with your actual CUDA installation path.

Integrating with Docker

Docker is a platform designed to simplify the process of creating, deploying, and running applications by using containerization. Containers allow developers to package an application with all its dependencies and libraries, ensuring it runs consistently across different environments. This eliminates the "it works on my machine" problem and makes deployment more efficient and reliable.

Why Use Docker?

• Consistency: Containers encapsulate everything your application needs to run, ensuring consistent behavior across development, testing, and production environments.
• Portability: Docker containers can run on any machine that supports Docker, whether it's a developer's laptop, on-premises servers, or cloud platforms.
• Efficiency: Containers are lightweight and use system resources more efficiently than traditional virtual machines.
• Scalability: Docker makes it easy to scale applications up or down, adding or removing containers as needed.

To get started with Docker, you'll need Docker Desktop, a comprehensive solution that includes Docker Engine, Docker CLI, Docker Compose, and other essential tools for container management.

Installing Docker Desktop

Docker Desktop is available for Windows, macOS, and Linux. It provides a user-friendly interface for managing containers and integrates seamlessly with your operating system.

1. Download Docker Desktop:

   • Windows: Docker Desktop for Windows
   • macOS: Docker Desktop for Mac
   • Linux: Visit the Docker Desktop for Linux page for installation instructions specific to your distribution.

2. Install Docker Desktop:

   • Windows:
     1. Download the Docker Desktop installer.
     2. Run the installer and follow the on-screen instructions.
     3. After installation, Docker Desktop will start automatically.
   • macOS:
     1. Download the Docker Desktop .dmg file.
     2. Open the .dmg file and drag the Docker icon to the Applications folder.
     3. Open Docker from the Applications folder.
   • Linux:
     1. Follow the instructions provided on the Docker Desktop for Linux page.

3. Configure Docker Desktop:

   • After installation, you can configure Docker Desktop to suit your preferences. This includes setting resource limits (CPU, memory) for Docker containers, enabling Kubernetes, and configuring proxies if necessary.

4. Start Docker Desktop:

   • Once installed, you can start Docker Desktop from your Start menu (Windows), Applications folder (macOS), or the command line (Linux). Docker Desktop will provide a graphical interface for managing your Docker containers and images.

With Docker Desktop installed, you can now proceed to integrate Docker into your workflow.

Start setting up a Docker image

1. Create a Dockerfile

​​​# Use the Nvidia PyTorch base image
​​​FROM nvcr.io/nvidia/pytorch:24.05-py3

​​​# Install ONNX Runtime GPU with Azure
​​​RUN pip install onnxruntime-gpu --extra-index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/

​​​# Copy the cudnn tarball file to the Docker image
​​​COPY cudnn-linux-x86_64-8.9.7.29_cuda12-archive.tar.xz /tmp/

​​​# Extract the tarball
​​​RUN tar -xvf /tmp/cudnn-linux-x86_64-8.9.7.29_cuda12-archive.tar.xz -C /tmp/

​​​# Copy the necessary files to their respective directories
​​​RUN cp /tmp/cudnn-*-archive/include/cudnn*.h /usr/local/cuda/include && \
​​​    cp -P /tmp/cudnn-*-archive/lib/libcudnn* /usr/local/cuda/lib64 && \
​​​    chmod a+r /usr/local/cuda/include/cudnn*.h /usr/local/cuda/lib64/libcudnn*

​​​# Clean up
​​​RUN rm -rf /tmp/cudnn-*-archive /tmp/cudnn-linux-x86_64-8.9.7.29_cuda12-archive.tar.xz

​​​# Set the LD_LIBRARY_PATH
​​​ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda/lib64

​​​# Set the working directory back to the root
​​​WORKDIR /workspace

This Dockerfile starts with an NVIDIA PyTorch base image and installs ONNX Runtime GPU. It then adds the cuDNN files and sets the necessary environment variables.

2. Build the Docker Image:

Navigate to your working directory and build the image with:

​​​docker build -t <name>:<tag> .

Replace <name> and <tag> with your desired image name and tag.

3. Create a docker-compose.yml File:

Use Docker Compose to manage and run your Docker containers:

​​​services:
​​​  trt:
​​​    image: <name>:<tag>
​​​    deploy:
​​​      resources:
​​​        reservations:
​​​          devices:
​​​            - driver: nvidia
​​​              count: all
​​​              capabilities: [gpu]
​​​    volumes:
​​​      - ../:/workspace
​​​    ipc: host
​​​    ulimits:
​​​      memlock: -1
​​​      stack: 67108864
​​​    stdin_open: true 
​​​    tty: true

This file defines a service named trt that uses the previously built image. It configures the container to use NVIDIA GPUs and sets necessary resource limits.

4. Run the Docker Compose File:

Start your container in detached mode (running in the background) with:

​​​docker compose up -d

Use -d for detached mode. For interactive mode, you can use -it.

5. Stop and remove the Container:

To stop and remove the running container, use:

​​​docker compose down

6. Start and stop container

For future use of container, we just need to stop and start it again like this:

​​​docker compose start

​​​docker compose stop

Ship image to Dockerhub

First, you need to create an account on Dockerhub. Then you will verify your login:

docker login

Then assign a new image name and tag:

docker tag <image>:<tag> <new_image>:<new_tag>

Finally push it to your dockerhub space:

docker push <new_image>:<new_tag>

This is the image with above configuration (see Dockerfile above) that was pushed to the my dockerhub space with name deeplearning:deployment. Now this image can be easily pulled with docker pull command:

docker pull kylepaul/deeplearning:deployment

Useful Linux Commands for Path Management

These commands help manage and inspect environment variables, especially PATH:

1. Print Path with New Lines:

These commands convert the PATH variable into a more readable format by replacing colons with new lines.

​​​echo $PATH | tr ':' '\n'
​​​echo $PATH | sed 's/:/\n/g'
​​​echo $PATH | awk -v RS=':' '{print}'
​​​echo $PATH | perl -pe 's/:/\n/g'

By following this guide, you should be able to set up a robust environment for running TensorRT with Docker on WSL. This approach ensures that all dependencies are properly managed and configured, providing a seamless experience for deep learning inference on your NVIDIA GPU.