---
# System prepended metadata

title: Speed and accuracy report classification models Autopilot 06022021
tags: [VinBDI - OCR group]

---

---
title: Speed and accuracy report classification models Autopilot 06022021
tags: VinBDI - OCR group
---

# Table of Contents
[toc]

# Experiment setup
**Calibration data**. The same dataset as testing data (ideal case)

**Benchmark tool**. [volksdep](https://github.com/Media-Smart/volksdep)

# Sign classification
**Command**.

| framework  |     input shape      |         data type         |   throughput(FPS)    |   latency(ms)   |         acc          |
|   :---:    |        :---:         |           :---:           |        :---:         |      :---:      |        :---:         |
|  pytorch   |   (1, 3, 224, 224)   |           fp32            |          61          |      16.27      |  0.9833  |
|  tensorrt  |   (1, 3, 224, 224)   |           fp32            |         856          |      1.19       |  0.9833  |
|  pytorch   |   (1, 3, 224, 224)   |           fp16            |          86          |      11.65      |  0.9833  |
|  tensorrt  |   (1, 3, 224, 224)   |           fp16            |         1201         |      0.94       |  0.9833  |
|  tensorrt  |   (1, 3, 224, 224)   |      int8(entropy_2)      |         1074         |      1.01       |  0.9827  |


**Danger**.

| framework  |     input shape      |         data type         |   throughput(FPS)    |   latency(ms)   |         acc          |
|   :---:    |        :---:         |           :---:           |        :---:         |      :---:      |        :---:         |
|  pytorch   |   (1, 3, 224, 224)   |           fp32            |          63          |      15.8       |  0.9583  |
|  tensorrt  |   (1, 3, 224, 224)   |           fp32            |         835          |      1.01       |  0.9583  |
|  pytorch   |   (1, 3, 224, 224)   |           fp16            |          82          |      12.51      |  0.9583  |
|  tensorrt  |   (1, 3, 224, 224)   |           fp16            |         1201         |      0.93       |  0.9583  |
|  tensorrt  |   (1, 3, 224, 224)   |      int8(entropy_2)      |         1075         |      1.03       |  0.9583  |

**Instruction**.

| framework  |     input shape      |         data type         |   throughput(FPS)    |   latency(ms)   |         acc          |
|   :---:    |        :---:         |           :---:           |        :---:         |      :---:      |        :---:         |
|  pytorch   |   (1, 3, 224, 224)   |           fp32            |          61          |      16.21      |  0.9831  |
|  tensorrt  |   (1, 3, 224, 224)   |           fp32            |         922          |      1.19       |  0.9831  |
|  pytorch   |   (1, 3, 224, 224)   |           fp16            |          86          |      11.57      |  0.9831  |
|  tensorrt  |   (1, 3, 224, 224)   |           fp16            |         1210         |      0.95       |  0.9831  |
|  tensorrt  |   (1, 3, 224, 224)   |      int8(entropy_2)      |         1099         |      1.08       |  0.9831  |

**Prohibition**.

| framework  |     input shape      |         data type         |   throughput(FPS)    |   latency(ms)   |         acc          |
|   :---:    |        :---:         |           :---:           |        :---:         |      :---:      |        :---:         |
|  pytorch   |   (1, 3, 224, 224)   |           fp32            |          58          |      16.54      |  0.9661  |
|  tensorrt  |   (1, 3, 224, 224)   |           fp32            |         955          |      1.11       |  0.9661  |
|  pytorch   |   (1, 3, 224, 224)   |           fp16            |          81          |      12.32      |  0.9661  |
|  tensorrt  |   (1, 3, 224, 224)   |           fp16            |         1139         |      0.83       |  0.9661  |
|  tensorrt  |   (1, 3, 224, 224)   |      int8(entropy_2)      |         988          |      1.17       |  0.9648  |

# Light classification
**Light color**.

| framework  |     input shape      |         data type         |   throughput(FPS)    |   latency(ms)   |         acc          |
|   :---:    |        :---:         |           :---:           |        :---:         |      :---:      |        :---:         |
|  pytorch   |   (1, 3, 112, 112)   |           fp32            |          69          |      14.54      |  0.9947  |
|  tensorrt  |   (1, 3, 112, 112)   |           fp32            |         1688         |      0.78       |  0.9947  |
|  pytorch   |   (1, 3, 112, 112)   |           fp16            |         102          |       9.8       |  0.9947  |
|  tensorrt  |   (1, 3, 112, 112)   |           fp16            |         2706         |      0.47       |  0.9947  |
|  tensorrt  |   (1, 3, 112, 112)   |      int8(entropy_2)      |         2431         |      0.56       |  0.9951  |

**Light direction**.

| framework  |     input shape      |         data type         |   throughput(FPS)    |   latency(ms)   |         acc          |
|   :---:    |        :---:         |           :---:           |        :---:         |      :---:      |        :---:         |
|  pytorch   |   (1, 3, 112, 112)   |           fp32            |          69          |      14.44      |  0.9936  |
|  tensorrt  |   (1, 3, 112, 112)   |           fp32            |         1786         |      0.72       |  0.9936  |
|  pytorch   |   (1, 3, 112, 112)   |           fp16            |         104          |       9.6       |  0.9936  |
|  tensorrt  |   (1, 3, 112, 112)   |           fp16            |         2834         |      0.49       |  0.9932  |
|  tensorrt  |   (1, 3, 112, 112)   |      int8(entropy_2)      |         2975         |      0.56       |  0.9932  |
