# 【Doc】Camera Porting - USB Camera AR0234 ###### tags: `Camera`,`i.MX8MP` --- #### Platform * Platform [i.MX 8M Plus](https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/i-mx-applications-processors/i-mx-8-applications-processors/i-mx-8m-plus-arm-cortex-a53-machine-learning-vision-multimedia-and-industrial-iot:IMX8MPLUS) ![](https://i.imgur.com/aPUks4s.png) * Sensor [基於安森美半導體200萬像素全局曝光AR0234搭配凌陽主控的影像識別USB Camera方案](https://www.wpgdadatong.com/solution/detail/36405) --- #### Device status * v4l2-ctl --list-devices ``` mxc-isi-m2m (platform:32e00000.isi:m2m_devic): /dev/video2 vsi_v4l2dec (platform:vsi_v4l2dec): /dev/video1 vsi_v4l2enc (platform:vsi_v4l2enc): /dev/video0 USB 2.0 Camera: USB Camera (usb-xhci-hcd.1.auto-1): /dev/video3 /dev/video4 /dev/media0 ``` Note : we need to find out capture device (32e00000, /dev/video3) --- #### AR0234CS Status * v4l2-ctl --device=/dev/video3 --all ```c root@opkiller:~# v4l2-ctl --device=/dev/video3 --all Driver Info: Driver name : uvcvideo Card type : USB 2.0 Camera: USB Camera Bus info : usb-xhci-hcd.1.auto-1 Driver version : 6.1.55 Capabilities : 0x84a00001 Video Capture Metadata Capture Streaming Extended Pix Format Device Capabilities Device Caps : 0x04200001 Video Capture Streaming Extended Pix Format Media Driver Info: Driver name : uvcvideo Model : USB 2.0 Camera: USB Camera Serial : SN0001 Bus info : usb-xhci-hcd.1.auto-1 Media version : 6.1.55 Hardware revision: 0x00000100 (256) Driver version : 6.1.55 Interface Info: ID : 0x03000002 Type : V4L Video Entity Info: ID : 0x00000001 (1) Name : USB 2.0 Camera: USB Camera Function : V4L2 I/O Flags : default Pad 0x01000007 : 0: Sink Link 0x02000010: from remote pad 0x100000a of entity 'Extension 3' (Video Pixel Formatter): Data, Enabled, Immutable Priority: 2 Video input : 0 (Input 1: ok) Format Video Capture: Width/Height : 1920/1080 Pixel Format : 'MJPG' (Motion-JPEG) Field : None Bytes per Line : 0 Size Image : 4147789 Colorspace : sRGB Transfer Function : Default (maps to sRGB) YCbCr/HSV Encoding: Default (maps to ITU-R 601) Quantization : Default (maps to Full Range) Flags : Crop Capability Video Capture: Bounds : Left 0, Top 0, Width 1920, Height 1080 Default : Left 0, Top 0, Width 1920, Height 1080 Pixel Aspect: 1/1 Selection Video Capture: crop_default, Left 0, Top 0, Width 1920, Height 1080, Flags: Selection Video Capture: crop_bounds, Left 0, Top 0, Width 1920, Height 1080, Flags: Streaming Parameters Video Capture: Capabilities : timeperframe Frames per second: 25.000 (25/1) Read buffers : 0 User Controls brightness 0x00980900 (int) : min=-64 max=64 step=1 default=0 value=0 contrast 0x00980901 (int) : min=0 max=64 step=1 default=32 value=32 saturation 0x00980902 (int) : min=0 max=128 step=1 default=64 value=64 hue 0x00980903 (int) : min=-40 max=40 step=1 default=0 value=0 white_balance_automatic 0x0098090c (bool) : default=1 value=1 gamma 0x00980910 (int) : min=72 max=500 step=1 default=100 value=100 gain 0x00980913 (int) : min=0 max=100 step=1 default=0 value=0 power_line_frequency 0x00980918 (menu) : min=0 max=2 default=1 value=1 (50 Hz) 0: Disabled 1: 50 Hz 2: 60 Hz white_balance_temperature 0x0098091a (int) : min=2800 max=6500 step=1 default=4600 value=4600 flags=inactive sharpness 0x0098091b (int) : min=0 max=6 step=1 default=3 value=3 backlight_compensation 0x0098091c (int) : min=0 max=2 step=1 default=1 value=1 Camera Controls auto_exposure 0x009a0901 (menu) : min=0 max=3 default=3 value=3 (Aperture Priority Mode) 1: Manual Mode 3: Aperture Priority Mode exposure_time_absolute 0x009a0902 (int) : min=1 max=5000 step=1 default=157 value=157 flags=inactive exposure_dynamic_framerate 0x009a0903 (bool) : default=0 value=1 ``` --- #### Camera format * AR0234CS format : ```c root@opkiller:~# v4l2-ctl -d /dev/video3 --list-formats-ext ioctl: VIDIOC_ENUM_FMT Type: Video Capture [0]: 'MJPG' (Motion-JPEG, compressed) Size: Discrete 1920x1080 Interval: Discrete 0.017s (60.000 fps) Size: Discrete 1280x720 Interval: Discrete 0.017s (60.000 fps) Interval: Discrete 0.033s (30.000 fps) Size: Discrete 800x600 Interval: Discrete 0.017s (60.000 fps) Interval: Discrete 0.033s (30.000 fps) Size: Discrete 640x480 Interval: Discrete 0.017s (60.000 fps) Interval: Discrete 0.033s (30.000 fps) [1]: 'YUYV' (YUYV 4:2:2) Size: Discrete 1920x1080 Interval: Discrete 0.200s (5.000 fps) Size: Discrete 1280x720 Interval: Discrete 0.100s (10.000 fps) Size: Discrete 800x600 Interval: Discrete 0.050s (20.000 fps) Size: Discrete 640x480 Interval: Discrete 0.033s (30.000 fps) ``` --- #### Eable Debug * Info : ``` $ dmesg -n 8 $ echo "8 4 1 7" > /proc/sys/kernel/printk ``` --- #### Uasge Method * Gstreamer * MJPG (720@FPS45) $ gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1280, height=720, framerate=60/1 ! jpegdec ! videoconvert ! fpsdisplaysink ```C root@opkiller:~# gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1280, height=720, framerate=60/1 ! jpegdec ! videoconvert ! fpsdisplaysink Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock Redistribute latency... ^Chandling interrupt. Interrupt: Stopping pipeline ... Execution ended after 0:00:18.382880620 Setting pipeline to NULL ... Total showed frames (836), playing for (0:00:18.382849493), fps (45.477) ``` * **MJPG (1080p@FPS35)** $ gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1920, height=1080, framerate=60/1 ! jpegdec ! videoconvert ! fpsdisplaysink sync=false ```C root@opkiller:~# gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1920, height=1080, framerate=60/1 ! jpegdec ! videoconvert ! fpsdisplaysink sync=false Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock Redistribute latency... ^Chandling interrupt. Interrupt: Stopping pipeline ... Execution ended after 0:00:09.816128125 Setting pipeline to NULL ... Total showed frames (341), playing for (0:00:09.816094500), fps (34.739). ``` * MJPG + G2D (720p@FPS28) $ gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1280, height=720, framerate=60/1 ! jpegdec ! queue ! imxvideoconvert_g2d ! video/x-raw, width=1920, height=1080 ! waylandsink ``` root@opkiller:~# gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1280, height=720, framerate=60/1 ! jpegdec ! queue ! imxvideoconvert_g2d ! video/x-raw, width=1920, height=1080 ! waylandsink Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock Redistribute latency... ^Chandling interrupt. Interrupt: Stopping pipeline ... Execution ended after 0:00:07.887531259 Setting pipeline to NULL ... Total showed frames (657), playing for (0:00:23.394841999), fps (28.083). ``` * MJPG (1080p@FPS20) $ gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1920, height=1080 ! queue ! jpegdec ! fpsdisplaysink ``` Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock Redistribute latency... ^Chandling interrupt. Interrupt: Stopping pipeline ... Execution ended after 0:00:12.551903119 Setting pipeline to NULL ... Total showed frames (245), playing for (0:00:12.551873368), fps (19.519). ``` * Note : framerate=60/1 & fpsdisplaysink 會頓 * YUV (Default) ```C $ GST_DEBUG=3 gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,format=YUV,width=1280,height=720, framerate=10/1 ! fpsdisplaysink root@imx8mpevk:~# GST_DEBUG=1 gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,format=RGB16,width=1280,height=720 ! fpsdisplaysink Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Pipeline is PREROLLED ... Setting pipeline to PLAYING ... New clock: GstSystemClock [ 292.468985] WPI [ 292.470818] YUV2RGB [ 292.472939] input fmt(WPI2) YUV4 [ 292.476209] output fmt RGBP ``` * FPS list |Name |MJPG|YUV| |-----|--------|-------| |1080p@60|5.94 (19.519) | | |1080p@60| | 5| |720p@30 |26.74 | | |720p@60 |47.99 | | |720p@10 || 10.0| |600p@60 |49.00 | | |600p@30 |29.55 | | |600p@30 | |20.01 | |480p@60 |49.85 | | |480p@30 |29.97 | 30.4| --- #### V4L2 Control * 調整 Camera 參數 ``` root@opkiller:~# v4l2-ctl -d /dev/video3 --list-ctrls User Controls brightness 0x00980900 (int) : min=-64 max=64 step=1 default=0 value=0 contrast 0x00980901 (int) : min=0 max=64 step=1 default=32 value=32 saturation 0x00980902 (int) : min=0 max=128 step=1 default=64 value=64 hue 0x00980903 (int) : min=-40 max=40 step=1 default=0 value=0 white_balance_automatic 0x0098090c (bool) : default=1 value=1 gamma 0x00980910 (int) : min=72 max=500 step=1 default=100 value=100 gain 0x00980913 (int) : min=0 max=100 step=1 default=0 value=0 power_line_frequency 0x00980918 (menu) : min=0 max=2 default=1 value=1 (50 Hz) white_balance_temperature 0x0098091a (int) : min=2800 max=6500 step=1 default=4600 value=4600 flags=inactive sharpness 0x0098091b (int) : min=0 max=6 step=1 default=3 value=3 backlight_compensation 0x0098091c (int) : min=0 max=2 step=1 default=1 value=1 Camera Controls ``` * v4l2-ctl --device=/dev/video3 --set-ctrl=gamma=72 ![image](https://hackmd.io/_uploads/Hyc8Kd670.png =80%x) * v4l2-ctl --device=/dev/video3 --set-ctrl=gamma=200 ![image](https://hackmd.io/_uploads/H1QYYOpQ0.png =80%x) * 調整格式 * 查詢格式 $ v4l2-ctl --list-framesizes=MJPG -d /dev/video3 ``` root@opkiller:~# v4l2-ctl --list-framesizes=MJPG -d /dev/video3 ioctl: VIDIOC_ENUM_FRAMESIZES Size: Discrete 1920x1080 Size: Discrete 1280x720 Size: Discrete 800x600 Size: Discrete 640x480 ``` * 設置 1080p@30fps $ v4l2-ctl -d /dev/video3 --set-fmt-video=width=1920,height=1080,pixelformat=MJPG $ v4l2-ctl -d /dev/video3 --set-parm=30 ``` root@opkiller:~# v4l2-ctl -d /dev/video3 --set-parm=30 Frame rate set to 60.000 fps ``` Note : 由於 Driver 不相容關係無法順利改成 30 FPS [參考文獻 - Linux配置USB免驱摄像头以及利用V4L2驱动调整参数](https://blog.csdn.net/weixin_44942126/article/details/115110762) * Camera 輸出 MJPEG 但 host 能力不足的問題 * 問題 ``` $ gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1920, height=1080, framerate=60/1 ! jpegdec ! videoconvert ! fpsdisplaysink ../git/libs/gst/base/gstbasesink.c(3143): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstWaylandSink:waylandsink0: There may be a timestamping problem, or this computer is too slow. WARNING: from element /GstPipeline:pipeline0/GstWaylandSink:waylandsink0: A lot of buffers are being dropped. ``` * CPU 資源 ``` MiB Mem : 5611.7 total, 5009.3 free, 445.5 used, 156.9 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 5052.5 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1660 root 20 0 333020 34076 31468 S 101.3 0.6 0:39.01 gst-launch-1.0 1575 root 0 -20 0 0 0 I 7.6 0.0 0:22.34 kworker/u9:1-uvcvideo 688 root 20 0 1497084 49716 31808 S 0.7 0.9 0:03.91 containerd 1659 root 20 0 5632 2696 2136 R 0.7 0.0 0:00.21 top 1 root 20 0 160140 9000 6332 S 0.0 0.2 0:04.27 systemd ``` * 解法 ``` gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1920, height=1080, framerate=60/1 ! jpegdec ! videoconvert ! fpsdisplaysink sync=false ```