[ToC]

# Models

## Theory Behind Computer Vision models

## Training
There are several codebases implementing different versions of YOLO:
- Chris Hughes' [YOLOv7](https://github.com/Chris-hughes10/Yolov7-training/tree/main) implementation and blog
- [MMYOLO]() providing multiple models from YOLO ([v5](https://github.com/open-mmlab/mmyolo/tree/main/configs/yolov5),[v7](https://github.com/open-mmlab/mmyolo/tree/main/configs/yolov7),[v8](https://github.com/open-mmlab/mmyolo/tree/main/configs/yolov8), [YOLOX](https://github.com/open-mmlab/mmyolo/tree/main/configs/yolox)) and their own [RTMDet](https://github.com/open-mmlab/mmyolo/tree/main/configs/rtmdet).
- [Ultralytics YOLOv8](https://docs.ultralytics.com/).

Each codebase has different software repositories set up, dependencies, and training pipeline code (e.g. most codebases use PyTorch but MMYOLO uses their own framework called MMDetection).

However, in general, model training requires the following:
- Dataset in YOLO format.
- ML model framework (e.g. PyTorch, TAO Toolkit, custom framework e.g. MMYOLO).
- Training loop code (setting up metric, loss, optimizer, model checkpoints and other YOLO-specific components).
## [TAO Toolkit](https://docs.nvidia.com/tao/tao-toolkit/)
TAO Toolkit is NVIDIA's software for training models and deploying them (using `tao-deploy`) to TensorRT Engines (optimized model) for use in subsequent pipelines (e.g. DeepStream).
- YOLOv4
- OCRNet
- LPDNet
- LPRNet
### [Quantization-Aware Training](https://docs.nvidia.com/tao/tao-toolkit/text/qat_and_amp_for_training.html#quantization-aware-training)
### [Mixed Precision Training](https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html)

MPT adopts FP16 for compute-intensive yet precision-insensitive operations (general matrix multiplication GEMM) while employing FP32 for precision-sensitive functions (batch normalization). Meanwhile, activations and gradients, which contribute to most of the memory consumption, are represented in FP16, while the weights are in FP32 to reduce the round-off error that appears in long-term accumulation of gradient updates. The issue here is [**underflow**](https://www.ijcai.org/proceedings/2020/0404.pdf) and reduction overflow (norm, softmax).

Simple example of reduction overflow:

`a=torch.cuda.HalfTensor(4094).fill(4.0)` equals to 256
`a=torch.cuda.HalfTensor(4095).fill(4.0)` equals \infty

Typically, gradients in DNN can be smaller than 10−10, and the smallest value that FP16 can represent is only around 6×10−8. That is, values below 6 × 10−8 will be rounded to zero.

 _`imgt.1`_

FP32 training


Using FP32 weights as master weights

 _`imgt.2`_

MPT (Blue parts are accelerated, Orange part is overhead)

 _`imgt.3`_

More info [1](https://on-demand.gputechconf.com/gtc-taiwan/2018/pdf/5-1_Internal%20Speaker_Michael%20Carilli_PDF%20For%20Sharing.pdf)

## Datasets

### Data Annotation
Data annotation is the 2nd most important step (after data collection) in creating a model dataset.
In our case, this is where ground truth bounding boxes are manually drawn on each image depending on the specific task (e.g. for OCR, draw boxes around letters).

There are multiple options available, but the most useful and tested are:
1. There is Amazon's [MTurks](https://www.mturk.com/) (rebranded to a newer service [SageMaker Ground Truth](https://aws.amazon.com/sagemaker/groundtruth/)) which offer paid annotation services.
2. An in-house annotation server can be created based on open source [CVAT](https://opencv.github.io/cvat/about/) tool which can also be hosted on ZenTech servers (requires additional setup/networking effort).

### Data Augmentation
Using the [Albumentations](https://albumentations.ai/docs/examples/) library we can augment images with transformations (e.g. crop, resize, rotate) by creating an augmentation [Compose](https://albumentations.ai/docs/api_reference/core/composition/) pipeline.
## Model Optimization & Acceleration

 _`imgc.1`_

**TRT** (TensorRT - our conversion tool and _runtime engine?_) can take in as input [tensorflow](https://github.com/tensorflow/tensorrt), [pytorch](https://github.com/pytorch/TensorRT) or ONNX [recommended way](#ONNX - Open Neural Network Exchange) models and convert the, to a hardware optimized model for running inference on GPU.

_`imgc.2`_

### ONNX - Open Neural Network Exchange
- ONNX acts as an intermediary and models can be exported in ONNX format from any framework.
- Most efficient runtime performance **while using an automatic parser**
- ONNX models are parsed automatically by TRT
- If a layer of the NN fails 2 options:
- Write custom TRT plugin for ONNX parser
- `ONNX-Graphsurgeon`

### Hardware compatibility

Convert model to TRT on the **deployment** GPU, if not make sure that the [Compute Capability](#Compute Capability) of the **target** is **at least as high** as the **dev**.

### Workflow of Tensor RT (TRT)

- Exporting my model to a format that TRT understands (discussed above)
- **Inference** batch size (throughput vs latency)
2 batch modes supported in TRT:
- Explicit : Exact batch size (when working withONNX)
- Dynamic : Range of batch shapes
- Precision (**INT8**, **FP16**, FP32, TF32 > [Compute Capability](#Compute Capability) > 8.0)
FP32 is usually used in training but lower precision in inference can yield performance optimization without losing on accuracy.
- [Mixed Precision Training (Discussed in Training sector](https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html)
- Some GPU cards favor INT8
Coupled with [QAT](https://developer.nvidia.com/blog/improving-int8-accuracy-using-quantization-aware-training-and-tao-toolkit/) at training, [FP32 accuracy can be achieved only using INT8](https://developer.nvidia.com/blog/achieving-fp32-accuracy-for-int8-inference-using-quantization-aware-training-with-tensorrt/). Quantization is a process of determining a way of binning the weights.
- Conversion path, discussed above, selection is based on:
- performance needs
- [supported layers](https://docs.nvidia.com/deeplearning/tensorrt/support-matrix/index.html)
- Runtime, optimized pipelines contribute to inference performance. It's **equally** important to actual model optimizations.

 _`imgc.3`_

### How does TRT optimize our networks?

As gradient calculations are not an issue in inference TRT can empirically optimize abiding to the following 3 principles:
- Copying **large** matrices and operating on them is faster than doing so on smaller ones
- Minimize CPU-GPU communication
- Keeping data in place aka minimize moving and copying

In detail:

This is **layer and tensor fusion** as it was seen in _`imgc.2`_
_`imgc.4`_
It's based on vertical/horizontal fusion and on eliminating unused layers.

Minimize information loss between FP32 (training **accuracy**) and INT8, FP16 (depends on what is chosen for inference). A calibration dataset is used to make this process automatic. Usually calibration is only needed when going from FP32->INT8

 _`imgc.5`_

**Dynamic tensor memory** reduces memory footprint and manages allocation for each tensor **only** when it's being used.

 _`imgc.6`_

**Kernel auto-tuning** selects best data layers and algorithms based on the target GPU. Baseline is cuDNN / cuBLAS. [For small batch size direct convolution kernels are used with optimal tiling](https://ccrma.stanford.edu/~jos/ReviewFourier/FFT_Convolution_vs_Direct.html). [Winograd](https://www.codetd.com/en/article/12633653), [FFT1](https://arxiv.org/pdf/2010.04257.pdf) [FFT2](https://medium.com/analytics-vidhya/fast-cnn-substitution-of-convolution-layers-with-fft-layers-a9ed3bfdc99a)
Essentially the fastest kernel for each layer depending on the target platform is used.

Can process **multiple input streams** in **parallel**.

~~ **Not applicable** to Computer Vision but TRT dynamically generates kernels to optimize RNNs over time ~~

### [Why are Tensor cores so important?](https://developer.nvidia.com/blog/tensor-core-ai-performance-milestones/)

### Compute Capability of different GPU hardware

Training [compute capability](https://developer.nvidia.com/cuda-gpus#compute) $\ge$ 6.1, inference $\ge$ 5.3
[Support matrix for different caps](https://docs.nvidia.com/deeplearning/tensorrt/support-matrix/index.html#hardware-precision-matrix)

### Which Layers are supported by TRT? (2017)

- [ ] Convolution
- [ ] LSTM and GRU
- [ ] Activation: ReLU, tanh, sigmoid
- [ ] Pooling: max and average
- [ ] Scaling
- [ ] Element wise operations
- [ ] LRN
- [ ] Fully-connected
- [ ] SoftMax
- [ ] Deconvolution

# Pipelines

## Deepstream
### DS (Deepstream) Pipeline
_`imgd.1`_

**[DeepStream SDK](https://docs.nvidia.com/metropolis/deepstream/dev-guide/index.html) is based on the GStreamer framework**. Plugins, input, outputs, and control parameters and metadata from gstreamer buffers can be found [here](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_Intro.html).

It can be interfaced using **C/C++, Python, Graph Composer**.

Extensions (such as Triton Inference components or TensorRT, discussed later) can be found [here](https://docs.nvidia.com/metropolis/deepstream/dev-guide/graphtools-docs/docs/text/ExtensionsManual/TensorRTExtension.html).

The application config can be broken apart into groups. [In detail](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_ref_app_deepstream.html#Configuration-Groups):

- Application Group
Application configurations that are not related to a specific component.

- Tiled-display Group
Tiled display in the application.

- Source Group
Source properties. There can be multiple sources. The groups must be named as: [source0], [source1] …

- Streammux Group
Specify properties and modify behavior of the streammux component.

- Preprocess Group
Specify properties and modify behavior of the preprocess component.

- Primary GIE and Secondary GIE Group
Specify properties and modify behavior of the primary GIE. Specify properties and modify behavior of the secondary GIE. The groups must be named as: [secondary-gie0], [secondary-gie1] …

- Tracker Group
Specify properties and modify behavior of the object tracker.

- Message Converter Group
Specify properties and modify behavior of the message converter component.

- Message Consumer Group
Specify properties and modify behavior of message consumer components. The pipeline can contain multiple message consumer components. Groups must be named as [message-consumer0], [message-consumer1] …

- OSD Group (ON SCREEN DISPLAY)
Specify properties and modify the on-screen display (OSD) component that overlays text and rectangles on the frame.

- Sink Group
Specify properties and modify behavior of sink components that represent outputs such as displays and files for rendering, encoding, and file saving. The pipeline can contain multiple sinks. Groups must be named as: [sink0], [sink1] …

- Tests Group
Diagnostics and debugging. This group is experimental.

- NvDs-analytics Group
Specify nvdsanalytics plugin configuration file, and to add the plugin in the application

### GC (Graph Composer)

A toolkit inside Deepstream that makes our life easier with a GUI to design our DS pipeline.
Workflow:
_`imgd.2`_

Example of a graph in GraphComposer

_`imgd.3`_
In the exposed docker #5 navigate to `/opt/nvidia/deepstream/deepstream-6.1/reference_graphs/deepstream-test5`. This is a sample app that demonstrates device-to-cloud and cloud-to-device messaging, Smart Record and model on-the-fly update.

An integral part of GC is [Container Builder](https://docs.nvidia.com/metropolis/deepstream/dev-guide/graphtools-docs/docs/text/GraphComposer_Container_Builder.html). If it wasn't for that [this](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_docker_containers.html#creating-custom-deepstream-docker-for-dgpu-using-deepstreamsdk-package) would be needed.

 _`imgd.4`_

### [Python Bindings](https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/blob/master/HOWTO.md)

To compile the bindings [do](https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/blob/master/bindings/README.md). A DS pipeline can be constructed only using Gst Python as it can be seen from the [sample applications](https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/tree/master/apps).

For cross-compilation use [this Dockerifle](https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/blob/master/bindings/qemu_docker/ubuntu-cross-aarch64.Dockerfile)

More in depth info about Allocations, MemManagement, String Access, Casting, Callback Function Registration, Optimizations and Utilities, Image Data Access can be found in the title's link.

 _`imgd.5`_

Includes a Preprocess, RTSP I/O, Analytics applications

### C/C++

Includes a wide variety of extra sample apps including Smart Record, Mask-RCNN, AMQP, Azure MQTT, DS as an NMOS Node, transceive video/metadata over RDMA

### Comparison between the 3 (Python vs C/C++ vs Graph Composer)

For the easiest stream pipeline:

filesrc [H264] → decode → nvstreammux → nvinfer (primary detector) → nvdsosd → renderer.

- C/C++ can be found in `/opt/nvidia/deepstream/deepstream-6.1/sources/apps/sample_apps/deepstream-test1`
- [Python app](https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/blob/master/apps/deepstream-test1/deepstream_test_1.py)
- Graph 

### Relevant Deepstream Sample Apps by NVIDIA

[multi-stream pipeline performing 4-class object detection - now also supports triton inference server, no-display mode, file-loop and silent mode](https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/tree/master/apps/deepstream-test3)

[Uses NvDsAnalytics and sends data to cloud/microservice through Kafka](https://github.com/NVIDIA-AI-IOT/deepstream-occupancy-analytics)

[NvDsAnalytics](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvdsanalytics.html)

### Optimizations

[Crop before inference - PREPROCESSING](https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/tree/master/apps/deepstream-preprocess-test)

 _`imgd.7`_

[Introductory video on optimizing some parts of deepstream](https://www.youtube.com/watch?v=Or8vfydL69s)

## Gstreamer

# Systems

## Monitoring

### [NVIDIA Memory Profiler](https://docs.nvidia.com/cuda/profiler-users-guide/index.html#visual-profiler)
NVIDIA's Tool for profiling GPU memory, utilization etc. Need to learn this for subsequent QA and optimization of hardware.
## Orchestration

In our Deepstream/Azure pipeline it might be wise to include Grafana/Prometheus/[Node Exporter](https://prometheus.io/docs/guides/node-exporter/). To orchestrate our containers, K8 might be replaced by azure container service since we'll be using Azure.

 _`imgs.1`_

 _`imgs.2`_

 _`imgs.3`_

 _`imgs.4`_

 _`imgs.5`_

## Integration

### [Triton Inference Server Engine ¡READ-ME!](https://github.com/triton-inference-server/server/blob/r22.04/README.md#documentation)

[TRT Inference Server](https://www.youtube.com/watch?v=SekmR9YH4xQ)

> Integrates with NGINX, Kubernetes, and Kubeflow for a complete solution for real‑time and offline data center AI inference [..] It supports all popular AI frameworks and maximizes GPU utilization by serving multiple models per GPU and dynamically batching client requests

As it is discussed in [Deepstream](#Deepstream), the Deepstream SDK consists of decoupled plugins and extensions. One of those plugins is [`Gst-nvmsgbroker`](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvmsgbroker.html#azure-mqtt-protocol-adapter-libraries?azure-portal=true), which enables stream message communication with Kafka, **Azure IoT**, MQTT etc.

 _`imgc.7`_

 _`imgc.8`_

 _`imgc.9`_

 _`imgc.10`_

 _`imgc.11`_

A **runtime** option great for serving models over HTTP and multi-GPU inference.

_TensorRT inference server = Triton_

If maximum **performance** is required, a Python/C++ runtime can be used instead of Triton, this obviously requires more effort. but gives full control of TRT engine.

 _`imgc.12`_

## Containerized DeepStream

 _`imgd.6`_

## Edge Deployment

### Jetson (Edge devices)

[Deepstream Python bindings for Jetson](https://gist.github.com/priyanshthakore/bd37ff636985640af1e97ad19942d02a)