# i5 benchmark

openvino@a5e56ae1868d:~$ python3 run_command.py
cmd:python3 /opt/intel/openvino/deployment_tools/tools/benchmark_tool/benchmark_app.py -m /mnt/openvino_models/public/yolo-v3-tf/FP32/yolo-v3-tf.xml -d CPU -api async -t 60

MKLDNNPlugin............ version 2.1
[ INFO ] Read network took 109.47 ms
[ INFO ] Load network took 901.70 ms
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests using 4 streams for CPU, limits: 60000 ms duration)
[ INFO ] First inference took 312.66 ms
Latency: 871.92 ms
Throughput: 4.58 FPS
cmd:python3 /opt/intel/openvino/deployment_tools/tools/benchmark_tool/benchmark_app.py -m /mnt/openvino_models/public/yolo-v3-tf/FP16/yolo-v3-tf.xml -d CPU -api async -t 60

MKLDNNPlugin............ version 2.1
[ INFO ] Read network took 76.73 ms
[ INFO ] Load network took 1065.13 ms
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests using 4 streams for CPU, limits: 60000 ms duration)
[ INFO ] First inference took 311.86 ms
Latency: 862.59 ms
Throughput: 4.64 FPS
cmd:python3 /opt/intel/openvino/deployment_tools/tools/benchmark_tool/benchmark_app.py -m /mnt/openvino_models/public/yolo-v3-tf/FP16-INT8/yolo-v3-tf.xml -d CPU -api async -t 60

MKLDNNPlugin............ version 2.1
[ INFO ] Read network took 66.48 ms
[ INFO ] Load network took 968.93 ms
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests using 4 streams for CPU, limits: 60000 ms duration)
[ INFO ] First inference took 85.29 ms
Latency: 226.31 ms
Throughput: 17.68 FPS
cmd:python3 /opt/intel/openvino/deployment_tools/tools/benchmark_tool/benchmark_app.py -m /mnt/openvino_models/public/yolo-v3-tf/FP32/yolo-v3-tf.xml -d GPU -api async -t 60

clDNNPlugin............. version 2.1
[ INFO ] Read network took 103.00 ms
[ INFO ] Load network took 39881.59 ms
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests using 2 streams for GPU, limits: 60000 ms duration)
[ INFO ] First inference took 107.76 ms
Latency: 416.58 ms
Throughput: 9.57 FPS
cmd:python3 /opt/intel/openvino/deployment_tools/tools/benchmark_tool/benchmark_app.py -m /mnt/openvino_models/public/yolo-v3-tf/FP16/yolo-v3-tf.xml -d GPU -api async -t 60

clDNNPlugin............. version 2.1
[ INFO ] Read network took 61.12 ms
[ INFO ] Load network took 39660.13 ms
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests using 2 streams for GPU, limits: 60000 ms duration)
[ INFO ] First inference took 59.65 ms
Latency: 217.70 ms
Throughput: 18.30 FPS
cmd:python3 /opt/intel/openvino/deployment_tools/tools/benchmark_tool/benchmark_app.py -m /mnt/openvino_models/public/yolo-v3-tf/FP16-INT8/yolo-v3-tf.xml -d GPU -api async -t 60

clDNNPlugin............. version 2.1
[ INFO ] Read network took 45.05 ms
[ INFO ] Load network took 45793.99 ms
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests using 2 streams for GPU, limits: 60000 ms duration)
[ INFO ] First inference took 30.05 ms
Latency: 99.70 ms
Throughput: 39.89 FPS
cmd:python3 /opt/intel/openvino/deployment_tools/tools/benchmark_tool/benchmark_app.py -m /mnt/openvino_models/public/yolo-v4-tf/FP32/yolo-v4-tf.xml -d CPU -api async -t 60

MKLDNNPlugin............ version 2.1
[ INFO ] Read network took 129.29 ms
[ INFO ] Load network took 1096.62 ms
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests using 4 streams for CPU, limits: 60000 ms duration)
[ INFO ] First inference took 683.86 ms
Latency: 1881.45 ms
Throughput: 2.13 FPS
cmd:python3 /opt/intel/openvino/deployment_tools/tools/benchmark_tool/benchmark_app.py -m /mnt/openvino_models/public/yolo-v4-tf/FP16/yolo-v4-tf.xml -d CPU -api async -t 60

MKLDNNPlugin............ version 2.1
[ INFO ] Read network took 78.18 ms
[ INFO ] Load network took 1183.26 ms
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests using 4 streams for CPU, limits: 60000 ms duration)
[ INFO ] First inference took 675.68 ms
Latency: 1858.83 ms
Throughput: 2.15 FPS
cmd:python3 /opt/intel/openvino/deployment_tools/tools/benchmark_tool/benchmark_app.py -m /mnt/openvino_models/public/yolo-v4-tf/FP16-INT8/yolo-v4-tf.xml -d CPU -api async -t 60

MKLDNNPlugin............ version 2.1
[ INFO ] Read network took 70.25 ms
[ INFO ] Load network took 1120.62 ms
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests using 4 streams for CPU, limits: 60000 ms duration)
[ INFO ] First inference took 226.80 ms
Latency: 591.58 ms
Throughput: 6.76 FPS
cmd:python3 /opt/intel/openvino/deployment_tools/tools/benchmark_tool/benchmark_app.py -m /mnt/openvino_models/public/yolo-v4-tf/FP32/yolo-v4-tf.xml -d GPU -api async -t 60

clDNNPlugin............. version 2.1
[ INFO ] Read network took 109.72 ms
[ INFO ] Load network took 57584.71 ms
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests using 2 streams for GPU, limits: 60000 ms duration)
[ INFO ] First inference took 251.02 ms
Latency: 948.32 ms
Throughput: 4.20 FPS
cmd:python3 /opt/intel/openvino/deployment_tools/tools/benchmark_tool/benchmark_app.py -m /mnt/openvino_models/public/yolo-v4-tf/FP16/yolo-v4-tf.xml -d GPU -api async -t 60

clDNNPlugin............. version 2.1
[ INFO ] Read network took 62.36 ms
[ INFO ] Load network took 57727.26 ms
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests using 2 streams for GPU, limits: 60000 ms duration)
[ INFO ] First inference took 128.43 ms
Latency: 470.14 ms
Throughput: 8.44 FPS
cmd:python3 /opt/intel/openvino/deployment_tools/tools/benchmark_tool/benchmark_app.py -m /mnt/openvino_models/public/yolo-v4-tf/FP16-INT8/yolo-v4-tf.xml -d GPU -api async -t 60

clDNNPlugin............. version 2.1
[ INFO ] Read network took 49.51 ms
[ INFO ] Load network took 65218.11 ms
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests using 2 streams for GPU, limits: 60000 ms duration)
[ INFO ] First inference took 63.55 ms
Latency: 222.57 ms
Throughput: 17.89 FPS

python3 /opt/intel/openvino/deployment_tools/tools/benchmark_tool/benchmark_app.py -m /mnt/openvino_models/public/deeplabv3/FP16-INT8/deeplabv3.xml -d GPU
/opt/intel/openvino/python/python3.8/openvino/tools/benchmark/utils/utils.py:219: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if arg_name is not '':
/opt/intel/openvino/python/python3.8/openvino/tools/benchmark/utils/utils.py:225: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if arg_name is not '':
[Step 1/11] Parsing and validating input arguments
[ WARNING ] -nstreams default value is determined automatically for a device. Although the automatic selection usually provides a reasonable performance, but it still may be non-optimal for some cases, for more information look at README.
[Step 2/11] Loading Inference Engine
[ INFO ] InferenceEngine:
API version............. 2.1.2021.3.0-2787-60059f2c755-releases/2021/3
[ INFO ] Device info
GPU
clDNNPlugin............. version 2.1
Build................... 2021.3.0-2787-60059f2c755-releases/2021/3

[Step 3/11] Setting device configuration
[ WARNING ] -nstreams default value is determined automatically for GPU device. Although the automatic selection usually provides a reasonable performance,but it still may be non-optimal for some cases, for more information look at README.
[Step 4/11] Reading network files
[ INFO ] Read network took 22.40 ms
[Step 5/11] Resizing network to match image sizes and given batch
[ INFO ] Network batch size: 1
[Step 6/11] Configuring input of the model
[Step 7/11] Loading the model to the device
[ INFO ] Load network took 31586.12 ms
[Step 8/11] Setting optimal runtime parameters
[Step 9/11] Creating infer requests and filling input blobs with images
[ INFO ] Network input 'mul_1/placeholder_port_1' precision U8, dimensions (NCHW): 1 3 513 513
[ WARNING ] No input files were given: all inputs will be filled with random values!
[ INFO ] Infer Request 0 filling
[ INFO ] Fill input 'mul_1/placeholder_port_1' with random values (image is expected)
[ INFO ] Infer Request 1 filling
[ INFO ] Fill input 'mul_1/placeholder_port_1' with random values (image is expected)
[ INFO ] Infer Request 2 filling
[ INFO ] Fill input 'mul_1/placeholder_port_1' with random values (image is expected)
[ INFO ] Infer Request 3 filling
[ INFO ] Fill input 'mul_1/placeholder_port_1' with random values (image is expected)
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests using 2 streams for GPU, limits: 60000 ms duration)
[ INFO ] First inference took 28.52 ms
[Step 11/11] Dumping statistics report
Count: 3052 iterations
Duration: 60107.73 ms
Latency: 76.91 ms
Throughput: 50.78 FPS