---
title: Speed and accuracy report classification models Autopilot 06022021
tags: VinBDI - OCR group
---

# Table of Contents
[toc]

# Experiment setup
**Calibration data**. The same dataset as testing data (ideal case)

**Benchmark tool**. [volksdep](https://github.com/Media-Smart/volksdep)

# Sign classification
**Command**.

| framework | input shape | data type | throughput(FPS) | latency(ms) | acc |
| :---: | :---: | :---: | :---: | :---: | :---: |
| pytorch | (1, 3, 224, 224) | fp32 | 61 | 16.27 | 0.9833 |
| tensorrt | (1, 3, 224, 224) | fp32 | 856 | 1.19 | 0.9833 |
| pytorch | (1, 3, 224, 224) | fp16 | 86 | 11.65 | 0.9833 |
| tensorrt | (1, 3, 224, 224) | fp16 | 1201 | 0.94 | 0.9833 |
| tensorrt | (1, 3, 224, 224) | int8(entropy_2) | 1074 | 1.01 | 0.9827 |

**Danger**.

| framework | input shape | data type | throughput(FPS) | latency(ms) | acc |
| :---: | :---: | :---: | :---: | :---: | :---: |
| pytorch | (1, 3, 224, 224) | fp32 | 63 | 15.8 | 0.9583 |
| tensorrt | (1, 3, 224, 224) | fp32 | 835 | 1.01 | 0.9583 |
| pytorch | (1, 3, 224, 224) | fp16 | 82 | 12.51 | 0.9583 |
| tensorrt | (1, 3, 224, 224) | fp16 | 1201 | 0.93 | 0.9583 |
| tensorrt | (1, 3, 224, 224) | int8(entropy_2) | 1075 | 1.03 | 0.9583 |

**Instruction**.

| framework | input shape | data type | throughput(FPS) | latency(ms) | acc |
| :---: | :---: | :---: | :---: | :---: | :---: |
| pytorch | (1, 3, 224, 224) | fp32 | 61 | 16.21 | 0.9831 |
| tensorrt | (1, 3, 224, 224) | fp32 | 922 | 1.19 | 0.9831 |
| pytorch | (1, 3, 224, 224) | fp16 | 86 | 11.57 | 0.9831 |
| tensorrt | (1, 3, 224, 224) | fp16 | 1210 | 0.95 | 0.9831 |
| tensorrt | (1, 3, 224, 224) | int8(entropy_2) | 1099 | 1.08 | 0.9831 |

**Prohibition**.

| framework | input shape | data type | throughput(FPS) | latency(ms) | acc |
| :---: | :---: | :---: | :---: | :---: | :---: |
| pytorch | (1, 3, 224, 224) | fp32 | 58 | 16.54 | 0.9661 |
| tensorrt | (1, 3, 224, 224) | fp32 | 955 | 1.11 | 0.9661 |
| pytorch | (1, 3, 224, 224) | fp16 | 81 | 12.32 | 0.9661 |
| tensorrt | (1, 3, 224, 224) | fp16 | 1139 | 0.83 | 0.9661 |
| tensorrt | (1, 3, 224, 224) | int8(entropy_2) | 988 | 1.17 | 0.9648 |

# Light classification
**Light color**.

| framework | input shape | data type | throughput(FPS) | latency(ms) | acc |
| :---: | :---: | :---: | :---: | :---: | :---: |
| pytorch | (1, 3, 112, 112) | fp32 | 69 | 14.54 | 0.9947 |
| tensorrt | (1, 3, 112, 112) | fp32 | 1688 | 0.78 | 0.9947 |
| pytorch | (1, 3, 112, 112) | fp16 | 102 | 9.8 | 0.9947 |
| tensorrt | (1, 3, 112, 112) | fp16 | 2706 | 0.47 | 0.9947 |
| tensorrt | (1, 3, 112, 112) | int8(entropy_2) | 2431 | 0.56 | 0.9951 |

**Light direction**.

| framework | input shape | data type | throughput(FPS) | latency(ms) | acc |
| :---: | :---: | :---: | :---: | :---: | :---: |
| pytorch | (1, 3, 112, 112) | fp32 | 69 | 14.44 | 0.9936 |
| tensorrt | (1, 3, 112, 112) | fp32 | 1786 | 0.72 | 0.9936 |
| pytorch | (1, 3, 112, 112) | fp16 | 104 | 9.6 | 0.9936 |
| tensorrt | (1, 3, 112, 112) | fp16 | 2834 | 0.49 | 0.9932 |
| tensorrt | (1, 3, 112, 112) | int8(entropy_2) | 2975 | 0.56 | 0.9932 |