---
# System prepended metadata

title: 【Doc】Camera Porting - USB Camera AR0234
tags: [i.MX8MP, Camera]

---

# 【Doc】Camera Porting - USB Camera AR0234 


###### tags: `Camera`,`i.MX8MP`

---

#### Platform  

* Platform
    [i.MX 8M Plus](https://www.nxp.com/products/processors-and-microcontrollers/arm-processors/i-mx-applications-processors/i-mx-8-applications-processors/i-mx-8m-plus-arm-cortex-a53-machine-learning-vision-multimedia-and-industrial-iot:IMX8MPLUS)
    
    ![](https://i.imgur.com/aPUks4s.png)
  
* Sensor
    [基於安森美半導體200萬像素全局曝光AR0234搭配凌陽主控的影像識別USB Camera方案](https://www.wpgdadatong.com/solution/detail/36405)

---

#### Device status
* v4l2-ctl --list-devices
    ```
    mxc-isi-m2m (platform:32e00000.isi:m2m_devic):
            /dev/video2

    vsi_v4l2dec (platform:vsi_v4l2dec):
            /dev/video1

    vsi_v4l2enc (platform:vsi_v4l2enc):
            /dev/video0

    USB 2.0 Camera: USB Camera (usb-xhci-hcd.1.auto-1):
            /dev/video3
            /dev/video4
            /dev/media0
    ```
Note : we need to find out capture device (32e00000, /dev/video3)

---
#### AR0234CS Status 
* v4l2-ctl --device=/dev/video3 --all
    ```c
    root@opkiller:~# v4l2-ctl --device=/dev/video3 --all
    Driver Info:
            Driver name      : uvcvideo
            Card type        : USB 2.0 Camera: USB Camera
            Bus info         : usb-xhci-hcd.1.auto-1
            Driver version   : 6.1.55
            Capabilities     : 0x84a00001
                    Video Capture
                    Metadata Capture
                    Streaming
                    Extended Pix Format
                    Device Capabilities
            Device Caps      : 0x04200001
                    Video Capture
                    Streaming
                    Extended Pix Format
    Media Driver Info:
            Driver name      : uvcvideo
            Model            : USB 2.0 Camera: USB Camera
            Serial           : SN0001
            Bus info         : usb-xhci-hcd.1.auto-1
            Media version    : 6.1.55
            Hardware revision: 0x00000100 (256)
            Driver version   : 6.1.55
    Interface Info:
            ID               : 0x03000002
            Type             : V4L Video
    Entity Info:
            ID               : 0x00000001 (1)
            Name             : USB 2.0 Camera: USB Camera
            Function         : V4L2 I/O
            Flags            : default
            Pad 0x01000007   : 0: Sink
              Link 0x02000010: from remote pad 0x100000a of entity 'Extension 3' (Video Pixel Formatter): Data, Enabled, Immutable
    Priority: 2
    Video input : 0 (Input 1: ok)
    Format Video Capture:
            Width/Height      : 1920/1080
            Pixel Format      : 'MJPG' (Motion-JPEG)
            Field             : None
            Bytes per Line    : 0
            Size Image        : 4147789
            Colorspace        : sRGB
            Transfer Function : Default (maps to sRGB)
            YCbCr/HSV Encoding: Default (maps to ITU-R 601)
            Quantization      : Default (maps to Full Range)
            Flags             :
    Crop Capability Video Capture:
            Bounds      : Left 0, Top 0, Width 1920, Height 1080
            Default     : Left 0, Top 0, Width 1920, Height 1080
            Pixel Aspect: 1/1
    Selection Video Capture: crop_default, Left 0, Top 0, Width 1920, Height 1080, Flags:
    Selection Video Capture: crop_bounds, Left 0, Top 0, Width 1920, Height 1080, Flags:
    Streaming Parameters Video Capture:
            Capabilities     : timeperframe
            Frames per second: 25.000 (25/1)
            Read buffers     : 0

    User Controls

                         brightness 0x00980900 (int)    : min=-64 max=64 step=1 default=0 value=0
                           contrast 0x00980901 (int)    : min=0 max=64 step=1 default=32 value=32
                         saturation 0x00980902 (int)    : min=0 max=128 step=1 default=64 value=64
                                hue 0x00980903 (int)    : min=-40 max=40 step=1 default=0 value=0
            white_balance_automatic 0x0098090c (bool)   : default=1 value=1
                              gamma 0x00980910 (int)    : min=72 max=500 step=1 default=100 value=100
                               gain 0x00980913 (int)    : min=0 max=100 step=1 default=0 value=0
               power_line_frequency 0x00980918 (menu)   : min=0 max=2 default=1 value=1 (50 Hz)
                                    0: Disabled
                                    1: 50 Hz
                                    2: 60 Hz
          white_balance_temperature 0x0098091a (int)    : min=2800 max=6500 step=1 default=4600 value=4600 flags=inactive
                          sharpness 0x0098091b (int)    : min=0 max=6 step=1 default=3 value=3
             backlight_compensation 0x0098091c (int)    : min=0 max=2 step=1 default=1 value=1

    Camera Controls

                      auto_exposure 0x009a0901 (menu)   : min=0 max=3 default=3 value=3 (Aperture Priority Mode)
                                    1: Manual Mode
                                    3: Aperture Priority Mode
             exposure_time_absolute 0x009a0902 (int)    : min=1 max=5000 step=1 default=157 value=157 flags=inactive
         exposure_dynamic_framerate 0x009a0903 (bool)   : default=0 value=1
    ```
 
---

#### Camera format 
* AR0234CS format :

    ```c
    root@opkiller:~# v4l2-ctl -d /dev/video3 --list-formats-ext
    ioctl: VIDIOC_ENUM_FMT
            Type: Video Capture

            [0]: 'MJPG' (Motion-JPEG, compressed)
                    Size: Discrete 1920x1080
                            Interval: Discrete 0.017s (60.000 fps)
                    Size: Discrete 1280x720
                            Interval: Discrete 0.017s (60.000 fps)
                            Interval: Discrete 0.033s (30.000 fps)
                    Size: Discrete 800x600
                            Interval: Discrete 0.017s (60.000 fps)
                            Interval: Discrete 0.033s (30.000 fps)
                    Size: Discrete 640x480
                            Interval: Discrete 0.017s (60.000 fps)
                            Interval: Discrete 0.033s (30.000 fps)
            [1]: 'YUYV' (YUYV 4:2:2)
                    Size: Discrete 1920x1080
                            Interval: Discrete 0.200s (5.000 fps)
                    Size: Discrete 1280x720
                            Interval: Discrete 0.100s (10.000 fps)
                    Size: Discrete 800x600
                            Interval: Discrete 0.050s (20.000 fps)
                    Size: Discrete 640x480
                            Interval: Discrete 0.033s (30.000 fps)
    ```

---

#### Eable Debug 

* Info :
    ```
    $ dmesg -n 8
    $ echo "8 4 1 7" > /proc/sys/kernel/printk
    ```

---

#### Uasge Method
* Gstreamer 

    * MJPG (720@FPS45)
        $ gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1280, height=720, framerate=60/1  ! jpegdec ! videoconvert ! fpsdisplaysink
        ```C
        root@opkiller:~#  gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1280, height=720, framerate=60/1  ! jpegdec ! videoconvert ! fpsdisplaysink
        Setting pipeline to PAUSED ...
        Pipeline is live and does not need PREROLL ...
        Pipeline is PREROLLED ...
        Setting pipeline to PLAYING ...
        New clock: GstSystemClock
        Redistribute latency...
        ^Chandling interrupt.
        Interrupt: Stopping pipeline ...
        Execution ended after 0:00:18.382880620
        Setting pipeline to NULL ...
        Total showed frames (836), playing for (0:00:18.382849493), fps (45.477)
        ```

    * **MJPG (1080p@FPS35)**
        $ gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1920, height=1080, framerate=60/1  ! jpegdec ! videoconvert ! fpsdisplaysink sync=false
        ```C
        root@opkiller:~# gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1920, height=1080, framerate=60/1  ! jpegdec ! videoconvert ! fpsdisplaysink sync=false
        Setting pipeline to PAUSED ...
        Pipeline is live and does not need PREROLL ...
        Pipeline is PREROLLED ...
        Setting pipeline to PLAYING ...
        New clock: GstSystemClock
        Redistribute latency...
        ^Chandling interrupt.
        Interrupt: Stopping pipeline ...
        Execution ended after 0:00:09.816128125
        Setting pipeline to NULL ...
        Total showed frames (341), playing for (0:00:09.816094500), fps (34.739).
        ```
        
    * MJPG + G2D (720p@FPS28)
        $ gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1280, height=720, framerate=60/1 ! jpegdec ! queue ! imxvideoconvert_g2d ! video/x-raw, width=1920, height=1080 ! waylandsink
        ```
        root@opkiller:~# gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1280, height=720, framerate=60/1 ! jpegdec ! queue ! imxvideoconvert_g2d ! video/x-raw, width=1920, height=1080 ! waylandsink
        Setting pipeline to PAUSED ...
        Pipeline is live and does not need PREROLL ...
        Pipeline is PREROLLED ...
        Setting pipeline to PLAYING ...
        New clock: GstSystemClock
        Redistribute latency...
        ^Chandling interrupt.
        Interrupt: Stopping pipeline ...
        Execution ended after 0:00:07.887531259
        Setting pipeline to NULL ...
        Total showed frames (657), playing for (0:00:23.394841999), fps (28.083).
        ```   
        
    
        
    * MJPG (1080p@FPS20)
        $ gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1920, height=1080 ! queue ! jpegdec ! fpsdisplaysink
        
        ```
        Setting pipeline to PAUSED ...
        Pipeline is live and does not need PREROLL ...
        Pipeline is PREROLLED ...
        Setting pipeline to PLAYING ...
        New clock: GstSystemClock
        Redistribute latency...
        ^Chandling interrupt.
        Interrupt: Stopping pipeline ...
        Execution ended after 0:00:12.551903119
        Setting pipeline to NULL ...
        Total showed frames (245), playing for (0:00:12.551873368), fps (19.519).

        ```
        * Note : framerate=60/1 & fpsdisplaysink 會頓
 
 
     * YUV (Default)
        ```C
        $ GST_DEBUG=3 gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,format=YUV,width=1280,height=720, framerate=10/1 ! fpsdisplaysink

        root@imx8mpevk:~# GST_DEBUG=1 gst-launch-1.0 v4l2src device=/dev/video3 ! video/x-raw,format=RGB16,width=1280,height=720 ! fpsdisplaysink
        Setting pipeline to PAUSED ...
        Pipeline is live and does not need PREROLL ...
        Pipeline is PREROLLED ...
        Setting pipeline to PLAYING ...
        New clock: GstSystemClock
        [  292.468985] WPI
        [  292.470818] YUV2RGB
        [  292.472939] input fmt(WPI2) YUV4
        [  292.476209] output fmt RGBP
        ```
        

    * FPS list
        |Name |MJPG|YUV|
        |-----|--------|-------|
        |1080p@60|5.94 (19.519)  | |
        |1080p@60|  | 5|
        |720p@30 |26.74 | | 
        |720p@60 |47.99 | | 
        |720p@10 || 10.0|
        |600p@60 |49.00 | | 
        |600p@30 |29.55 | | 
        |600p@30 | |20.01 |
        |480p@60 |49.85 | | 
        |480p@30 |29.97 | 30.4| 
        
---

#### V4L2 Control

* 調整 Camera 參數

    ```
    root@opkiller:~# v4l2-ctl -d /dev/video3 --list-ctrls

    User Controls

                         brightness 0x00980900 (int)    : min=-64 max=64 step=1 default=0 value=0
                           contrast 0x00980901 (int)    : min=0 max=64 step=1 default=32 value=32
                         saturation 0x00980902 (int)    : min=0 max=128 step=1 default=64 value=64
                                hue 0x00980903 (int)    : min=-40 max=40 step=1 default=0 value=0
            white_balance_automatic 0x0098090c (bool)   : default=1 value=1
                              gamma 0x00980910 (int)    : min=72 max=500 step=1 default=100 value=100
                               gain 0x00980913 (int)    : min=0 max=100 step=1 default=0 value=0
               power_line_frequency 0x00980918 (menu)   : min=0 max=2 default=1 value=1 (50 Hz)
          white_balance_temperature 0x0098091a (int)    : min=2800 max=6500 step=1 default=4600 value=4600 flags=inactive
                          sharpness 0x0098091b (int)    : min=0 max=6 step=1 default=3 value=3
             backlight_compensation 0x0098091c (int)    : min=0 max=2 step=1 default=1 value=1

    Camera Controls
    ```
    
    * v4l2-ctl --device=/dev/video3 --set-ctrl=gamma=72
      ![image](https://hackmd.io/_uploads/Hyc8Kd670.png =80%x)

    * v4l2-ctl --device=/dev/video3 --set-ctrl=gamma=200
        ![image](https://hackmd.io/_uploads/H1QYYOpQ0.png =80%x)        

* 調整格式
    * 查詢格式
        $ v4l2-ctl --list-framesizes=MJPG -d /dev/video3
        
        ```
        root@opkiller:~# v4l2-ctl --list-framesizes=MJPG -d /dev/video3
        ioctl: VIDIOC_ENUM_FRAMESIZES
                Size: Discrete 1920x1080
                Size: Discrete 1280x720
                Size: Discrete 800x600
                Size: Discrete 640x480
        ```
    
    * 設置 1080p@30fps
        $ v4l2-ctl -d /dev/video3 --set-fmt-video=width=1920,height=1080,pixelformat=MJPG
        $ v4l2-ctl -d /dev/video3 --set-parm=30
        ```
        root@opkiller:~# v4l2-ctl -d /dev/video3 --set-parm=30
        Frame rate set to 60.000 fps
        ```
        Note : 由於 Driver 不相容關係無法順利改成 30 FPS
        
[參考文獻 - Linux配置USB免驱摄像头以及利用V4L2驱动调整参数](https://blog.csdn.net/weixin_44942126/article/details/115110762)


* Camera 輸出 MJPEG 但 host 能力不足的問題
    * 問題
        ```
        $ gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1920, height=1080, framerate=60/1 ! jpegdec ! videoconvert ! fpsdisplaysink
        ../git/libs/gst/base/gstbasesink.c(3143): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstWaylandSink:waylandsink0:
        There may be a timestamping problem, or this computer is too slow.
        WARNING: from element /GstPipeline:pipeline0/GstWaylandSink:waylandsink0: A lot of buffers are being dropped.
        ```
        
    * CPU 資源
        ```
        MiB Mem :   5611.7 total,   5009.3 free,    445.5 used,    156.9 buff/cache
        MiB Swap:      0.0 total,      0.0 free,      0.0 used.   5052.5 avail Mem

            PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
           1660 root      20   0  333020  34076  31468 S 101.3   0.6   0:39.01 gst-launch-1.0
           1575 root       0 -20       0      0      0 I   7.6   0.0   0:22.34 kworker/u9:1-uvcvideo
            688 root      20   0 1497084  49716  31808 S   0.7   0.9   0:03.91 containerd
           1659 root      20   0    5632   2696   2136 R   0.7   0.0   0:00.21 top
              1 root      20   0  160140   9000   6332 S   0.0   0.2   0:04.27 systemd
        ```
    
    * 解法 
        ```
        gst-launch-1.0 v4l2src device=/dev/video3 ! image/jpeg, width=1920, height=1080, framerate=60/1 ! jpegdec ! videoconvert ! fpsdisplaysink sync=false
        ```