2021q1 Final Project (vcam)

# 2021q1 Final Project (vcam) contributed by < [`WayneLin1992`](https://github.com/WayneLin1992) > ###### tags: `linux2021` `vcam` ### 進度規劃 - [x] 重現 vcam 測試紀錄 - [x] 理解 framebuffer 原理 - [x] 理解 V4L2 框架 - [x] 理解程式運作原理 - [x] 整合之前的實作 ### 重現vcam測試紀錄參考資料 >[vcam github](https://github.com/WayneLin1992/vcam) >[vcam 測試紀錄](https://hackmd.io/@bentu/SkUFns9XI?fbclid=IwAR0dip-LYX8zLmaOWMcCdEu5w8fCWPtjBLh9TV7vH0FRQG1jmfNboDlIvI4) ### ubuntu 20.04.1 測試環境： ```shell $ uname -a Linux wayne-ThinkPad-T14s-Gen-1 5.8.0-50-generic #56~20.04.1-Ubuntu SMP Mon Apr 12 21:46:35 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux ``` 為了編譯 kernel driver 安裝需要的 linux headers `$ sudo apt install linux-headers-$(uname -r)` 為了獲得相關信息，及驗證安裝 `v4l-utils` `$ sudo apt install v4l-utils` 執行 `make` 即可 ```shell wayne@wayne-ThinkPad-T14s-Gen-1:~/linux2021/vcam$ make make -C /lib/modules/5.8.0-50-generic/build M=/home/wayne/linux2021/vcam modules make[1]: 進入目錄「/usr/src/linux-headers-5.8.0-50-generic」 CC [M] /home/wayne/linux2021/vcam/module.o CC [M] /home/wayne/linux2021/vcam/control.o CC [M] /home/wayne/linux2021/vcam/device.o /home/wayne/linux2021/vcam/device.c: In function ‘create_vcam_device’: /home/wayne/linux2021/vcam/device.c:774:39: error: ‘VFL_TYPE_GRABBER’ undeclared (first use in this function); did you mean ‘VFL_TYPE_SUBDEV’? 774 | ret = video_register_device(vdev, VFL_TYPE_GRABBER, -1); | ^~~~~~~~~~~~~~~~ | VFL_TYPE_SUBDEV /home/wayne/linux2021/vcam/device.c:774:39: note: each undeclared identifier is reported only once for each function it appears in make[2]: *** [scripts/Makefile.build:286：/home/wayne/linux2021/vcam/device.o] 錯誤 1 make[1]: *** [Makefile:1783：/home/wayne/linux2021/vcam] 錯誤 2 make[1]: 離開目錄「/usr/src/linux-headers-5.8.0-50-generic」 make: *** [Makefile:14：kmod] 錯誤 2 ``` 由 [v4l2-dev.h: remove VFL_TYPE_GRABBER](https://patchwork.kernel.org/project/linux-media/patch/20200203114119.1177490-12-hverkuil-cisco@xs4all.nl/) 及 [media: rename VFL_TYPE_GRABBER to _VIDEO](https://patchwork.kernel.org/project/linux-media/patch/20200203114119.1177490-2-hverkuil-cisco@xs4all.nl/)可以知道在 2020 3月被移除，但相對應的可以使用 VFL_TYPE_VIDEO 來代替。由此我將 VFL_TYPE_GRABBER 改為 VFL_TYPE_VIDEO 嘗試看看編譯結果 :::info 在 kernel 4.15 VFL_TYPE_GRABBER: /dev/videoX for video input/output devices 在 kernel 5.8 VFL_TYPE_VIDEO: /dev/videoX for video input/output devices 參考資料: [linux kernel 4.15v v4l2](https://www.kernel.org/doc/html/v4.15/media/kapi/v4l2-dev.html) [linux kernel 5.8v v4l2](https://www.kernel.org/doc/html/v5.8/driver-api/media/v4l2-device.html) ::: device.o 成功編譯但出現其他問題。 ```shell make -C /lib/modules/5.8.0-50-generic/build M=/home/wayne/linux2021/vcam modules make[1]: 進入目錄「/usr/src/linux-headers-5.8.0-50-generic」 CC [M] /home/wayne/linux2021/vcam/module.o CC [M] /home/wayne/linux2021/vcam/control.o CC [M] /home/wayne/linux2021/vcam/device.o CC [M] /home/wayne/linux2021/vcam/videobuf.o CC [M] /home/wayne/linux2021/vcam/fb.o /home/wayne/linux2021/vcam/fb.c: In function ‘init_framebuffer’: /home/wayne/linux2021/vcam/fb.c:120:54: error: passing argument 4 of ‘proc_create_data’ from incompatible pointer type [-Werror=incompatible-pointer-types] 120 | procf = proc_create_data(proc_fname, 0666, NULL, &vcamfb_fops, dev); | ^~~~~~~~~~~~ | | | struct file_operations * In file included from /home/wayne/linux2021/vcam/fb.c:3: ./include/linux/proc_fs.h:103:31: note: expected ‘const struct proc_ops *’ but argument is of type ‘struct file_operations *’ 103 | extern struct proc_dir_entry *proc_create_data(const char *, umode_t, | ^~~~~~~~~~~~~~~~ cc1: some warnings being treated as errors make[2]: *** [scripts/Makefile.build:286：/home/wayne/linux2021/vcam/fb.o] 錯誤 1 make[1]: *** [Makefile:1783：/home/wayne/linux2021/vcam] 錯誤 2 make[1]: 離開目錄「/usr/src/linux-headers-5.8.0-50-generic」 make: *** [Makefile:14：kmod] 錯誤 2 ``` proc_create_data 的 arguments 的 type 有因為 linux 更新有所產生改變，由 [How to fix error: passing argument 4 of 'proc_create' from incompatible pointer type](https://stackoverflow.com/questions/64931555/how-to-fix-error-passing-argument-4-of-proc-create-from-incompatible-pointer) 可以知道這是 kernel version 5.6 之後的更新，查看一下 `struct proc_ops` 及看一下 `vcamfb_fops` 在 fb.c 中 ```cpp static struct file_operations vcamfb_fops = { .owner = THIS_MODULE, .open = vcamfb_open, .release = vcamfb_release, .write = vcamfb_write, }; ``` 發現到其中 `owner` 是其中最大的差距 `struct module *owner` 和 `unsigned int proc_flags` 大小也不相等，不能直接透過轉，嘗試直接將 fb.c 中的 `file_operations` 換成 `proc_ops` .owner 換成 .proc_flags。 ```diff static struct proc_ops vcamfb_fops = { + .proc_open = vcamfb_open, + .proc_release = vcamfb_release, + .proc_write = vcamfb_write, }; ``` 可以發現到已經成功掛載 `vcam` `$ sudo ./vcam-util -l` ```shell Available virtual V4L2 compatible devices: 1. vcamfb2(640,480,rgb24) -> /dev/video2 ``` `$ sudo v4l2-compliance -d /dev/video2 -f` :::info v4l2-compliance - An application to test video4linux drivers >The v4l2-compliance tool is used to test video4linux devices, either video, vbi, radio or swradio, both input and output. It attempts to test almost all aspects of a V4L2 device and it covers almost all V4L2 ioctls. It has very good support for video capture and output, VBI capture and output and (software) radio tuning and transmitting. >-d, --device=<dev> Use device <dev> as the video device. If <dev> is a number, then /dev/video<dev> is used. 其中 -f, --stream-all-formats [<count>] >Test whether all available formats can be streamed. This attempts to stream using MMAP mode or read/write (if V4L2_MEMORY_MMAP is not available) for one second for all formats, at all sizes, at all intervals and with all field values. In addition if the driver supports scaling, cropping or composing it will test that as well in various combinations. 因此可以知道 stream-all-formats 測試將會嘗試不同的格式。注意到 Buffer ioctls fail test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: FAIL 可以了解到這裡是對 buffer 進行測試，初始、創立、詢問狀態，失敗 VIDIOC_REQBUFS - Initiate Memory Mapping, User Pointer I/O or DMA buffer I/O VIDIOC_CREATE_BUFS - Create buffers for Memory Mapped or User Pointer or DMA Buffer I/O VIDIOC_QUERYBUF - Query the status of a buffer 其中 vcam 中 device.c .vidioc_reqbufs = vb2_ioctl_reqbufs 這可能是有改變的地方。參考資料： [v4l2-compliance](http://manpages.ubuntu.com/manpages/focal/man1/v4l2-compliance.1.html) [VIDIOC_REQBUFS](https://01.org/linuxgraphics/gfx-docs/drm/media/uapi/v4l/vidioc-reqbufs.html) [VIDIOC_CREATE_BUFS](https://www.kernel.org/doc/html/v4.9/media/uapi/v4l/vidioc-create-bufs.html) [VIDIOC_QUERYBUF](https://www.kernel.org/doc/html/v4.9/media/uapi/v4l/vidioc-querybuf.html) ::: `$ sudo v4l2-ctl -d /dev/video2 --all` ```shell Driver Info: Driver name : vcam Card type : vcam Bus info : platform: virtual Driver version : 5.8.18 Capabilities : 0x85200001 Video Capture Read/Write Streaming Extended Pix Format Device Capabilities Device Caps : 0x05200001 Video Capture Read/Write Streaming Extended Pix Format Priority: 2 Video input : 0 (vcam_in 0: ok) Format Video Capture: Width/Height : 640/480 Pixel Format : 'RGB3' (24-bit RGB 8-8-8) Field : None Bytes per Line : 1920 Size Image : 921600 Colorspace : sRGB Transfer Function : Default (maps to sRGB) YCbCr/HSV Encoding: Default (maps to ITU-R 601) Quantization : Default (maps to Full Range) Flags : Streaming Parameters Video Capture: Frames per second: 29.970 (30000/1001) Read buffers : 1 ``` ### ubuntu 18.04.1 經過老師的建議，我重新安裝 ubuntu 18.04 kernel 4.15 的版本，進行測試。測試環境： ```shell Linux wayne-MacBookAir 4.15.0-29-generic #31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux ``` `$ sudo insmod vcam.ko` ```shell insmod: ERROR: could not insert module vcam.ko: Unknown symbol in module ``` `dmesg` ```shell [ 3152.479469] videobuf2_vmalloc: unknown parameter 'videobuf2_v4l2' ignored [ 3169.618202] vcam: Unknown symbol vb2_queue_init (err 0) [ 3169.618231] vcam: Unknown symbol video_ioctl2 (err 0) ... [ 4237.804522] vcam: Unknown symbol video_device_release_empty (err 0) [ 4237.804541] vcam: Unknown symbol vb2_ioctl_reqbufs (err 0) ``` `$ sudo modprobe -a videobuf2_vmalloc videobuf2_v4l2` 這樣就可以成功掛載 `vcam` `$ sudo insmod vcam.ko` `sudo v4l2-ctl -d /dev/video0 --all` ```shell Driver Info (not using libv4l2): Driver name : vcam Card type : vcam Bus info : platform: virtual Driver version: 4.15.18 Capabilities : 0x85200001 Video Capture Read/Write Streaming Extended Pix Format Device Capabilities Device Caps : 0x05200001 Video Capture Read/Write Streaming Extended Pix Format Priority: 2 Video input : 0 (vcam_in 0: ok) Format Video Capture: Width/Height : 640/480 Pixel Format : 'RGB3' Field : None Bytes per Line : 1920 Size Image : 921600 Colorspace : sRGB Transfer Function : Default (maps to sRGB) YCbCr/HSV Encoding: Default (maps to ITU-R 601) Quantization : Default (maps to Full Range) Flags : Streaming Parameters Video Capture: Frames per second: 29.970 (30000/1001) Read buffers : 1 ``` 能夠成功開啟。 ### 理解 framebuffer 原理 vcam 是一個與 v4l2 兼容，並且由 framebuffer 當輸入，所以了解 vcam 必須先了解 framebuffer。 framebuffer 顧名思義就是就是記憶體的一個區域，裡面存放著圖片或影片的信息(如 RGB YUV)，讓 vcam 讀取，所以可以做到使 camera 讀到虛假的畫面。 ![](https://i.imgur.com/3dPk5tN.jpg) 圖中 DAC(Digital to analog converter) 將數位信號轉成類比訊號。就像 [MMAP(Memory-mapped file)](https://en.wikipedia.org/wiki/Memory-mapped_file) 一樣，將圖像直接寫入記憶體當中，給 I/O 直接讀取，省去執行 I/O 的時間延遲，所以顯示卡才會有對應的獨立的記憶體空間，這也是為了使 I/O 執行較快，再來還可以透過 [SIMD](https://en.wikipedia.org/wiki/SIMD) 處理，來加快 I/O 的執行速度，對於網路影像，也可以透過 [zero-copy](https://en.wikipedia.org/wiki/Zero-copy) 的方式直接將網路圖像直接寫入對應的記憶中，直接讀取。 #### Double buffering vcam 主要是使用雙緩衝原理來避免畫面撕裂。畫面的撕裂如下所示，當顯示的同時有 CPU 或 GPU 對 buffer 做處理就會產生畫面的撕裂。 ![](https://i.imgur.com/VQF1X34.jpg) 雙緩衝是透過一個 buffer 為顯示，而另一個 buffer 同時在填充資料，當 buffer 填充完資料後顯示，同時原本顯示的 buffer 又拿來填充，如下圖 ![](https://i.imgur.com/0a9wBvv.jpg) 以時間軸表示如下圖在顯示 buffer 0 時 GPU 和 CPU 對 buffer 1 做處理， VSync(Vertical sync) 代表螢幕同步刷新的時間點，當刷新時間和處理時間必須配合時，否則會產生撕裂畫面。 ![](https://i.imgur.com/CalJ3UA.jpg) :::info 參考資料: [what is frame buffer](https://ecomputernotes.com/computer-graphics/basic-of-computer-graphics/what-is-frame-buffer) [frame buffer](https://en.wikipedia.org/wiki/Framebuffer) [double buffering, triple buffering Analysis](https://www.programmersought.com/article/88712838197/) ::: ### 理解程式運作原理 **由於主要執行是透過 framebuffer 的操作，所以要特別觀察其中的操作。** - [ ] module.c 將 vcam 初始化 `vcam_init` 實作在 `create_control_device` 表示，其中使用 kmalloc 進行配置 `struct control_device` 空間，還可以開啟模組參數如: allow_pix_conversion 可以接受 rgb 轉換成 YUV ， `static int __init vcam_init(void)` 實作 `__init create_control_device` :::info 在 `include/linux/init.h` 看到相關解釋， [`__init`](https://stackoverflow.com/questions/8832114/what-does-init-mean-in-the-linux-kernel-code) >used to mark some functions or initialized data (doesn't apply to uninitialized data) as `initialization' functions. The kernel can take this as hint that the function is used only during the initialization phase and free up used memory resources after 將會標註為初始化函數，只會使用一次。 [module_param](https://b8807053.pixnet.net/blog/post/339227792-linux-driver-modules-param) >定義一個模組參數 `MODULE_PARM_DESC()` 說明模組功能 ::: - [ ] control.c 當 module 在 kernel 中被建立， class_create 會透過 [udev](https://zh.wikipedia.org/wiki/Udev) 將會在 /dev 動態的建立一個 device 文件。 :::info __class_create 為 class_create 的內部實作。 `struct class * __class_create(struct module * owner, const char * name, struct lock_class_key * key);` __class_create - create a struct class structure > This is used to create a struct class pointer that can then be used in calls to device_create. Returns struct class pointer on success, or ERR_PTR on error. `struct device * device_create(struct class * class, struct device * parent, dev_t devt, void * drvdata, const char * fmt, ...);` device_create - creates a device and registers it with sysfs >This function can be used by char device classes. A struct device will be created in sysfs, registered to the specified class. A “dev” file will be created, showing the dev_t for the device. Returns struct device pointer on success, or ERR_PTR on error. ::: `create_control_device` 掛載: device 註冊透過 `cdev_init` 將 struct cdev 初始化， cdev 是 The struct cdev is the kernel’s internal structure that represents char devices. This field contains a pointer to that structure when the inode refers to a char device file. char devices 的操作由 `file_operations control_fops` 來存取，其中有一項 `.unlocked_ioctl = control_ioctl` 代表對 char device ioctl 操作。 :::info [unlocked_ioctl()](https://unix.stackexchange.com/questions/4711/what-is-the-difference-between-ioctl-unlocked-ioctl-and-compat-ioctl) 取代 .ioctl 是避免 [big kernel lock(BKL)](https://en.wikipedia.org/wiki/Giant_lock) ::: `control_ioctl` 包括 `VCAM_IOCTL_CREATE_DEVICE` 會調用 `request_vcam_device` 會 `create_vcam_device` 創立出 `vcam_device` 並且預設大小為 640 * 480 rgb-24 ，並將創立的 `vcam` 並連接 `ctldev->vcam_devices[idx] = vcam` 。 `VCAM_IOCTL_DESTROY_DEVICE` 調用 `control_iocontrol_destroy_device` 將 `vcam` destroy 並對應 idx-- 。 `VCAM_IOCTL_GET_DEVICE` 調用 `control_iocontrol_get_device` 將 `vcam` format 提取出來並寫入 `dev_spec` 並 `snprintf` 出。 `VCAM_IOCTL_MODIFY_SETTING` 調用 `control_iocontrol_modify_input_setting` `dev_spec` 修改 `input_format` 的 `setting` 。透過 `alloc_chrdev_region` 取得動態配置 device 的 major number 透過 `cdev_add` 向 kernel 登入驅動程式 :::info void cdev_init (struct cdev * cdev, const struct file_operations * fops); cdev_init — initialize a cdev structure >Initializes cdev, remembering fops, making it ready to add to the system with cdev_add. `int alloc_chrdev_region (dev_t * dev, unsigned baseminor, unsigned count, const char * name);` alloc_chrdev_region — register a range of char device numbers >Allocates a range of char device numbers. The major number will be chosen dynamically, and returned (along with the first minor number) in dev. Returns zero or a negative error code. `int cdev_add (struct cdev * p, dev_t dev, unsigned count);` cdev_add — add a char device to the system >adds the device represented by p to the system, making it live immediately. A negative error code is returned on failure. ::: `destroy_control_device` 卸載: device 透過 `cdev_del` 將其中 cdev 從 kernel 中移除。透過 `unregister_chrdev_region` 將其中 major number 移除 :::info `cdev` 是一種與 kernel 溝通的結構，主要表達 char device，所以需要註冊，`struct inode` 會在指向 `cdev`，`struct inode` 代表 file 在 rootfs 中。參考資料: [why to register struct cdev in driver code](https://stackoverflow.com/questions/14705189/why-to-register-struct-cdev-in-driver-code) ::: `request_vcam_device` 將執行 `create_vcam_device` 而 `create_vcam_device` 實作將在 device.c 中，也會在之後說明 `spin_lock_irqsave` 及 `spin_unlock_irqrestore` 保證了 vcam_device_count 正確性。 :::info `spin_lock_irqsave `\ `spin_unlock_irqrestore` ```cpp spin_lock_irqsave(&mLock, flags); // Save the state of interrupt enable in flags and then disable interrupts // Critical section spin_unlock_irqrestore(&mLock, flags); // Return to the previous state saved in flags ``` `spin_lock_irqsave`會保存當下狀態及中斷的狀態，並把 interrupt 關掉，等到 critical section 處裡完，將會回復到原本狀態，並將 interrupt 開啟。 `spin_lock_irq` \ `spin_unlock_irq` ```cpp spin_lock_irq(&mLock); // Does not know if interrupts are already disabled // Critical section spin_unlock_irq(&mLock); // Could result in an unwanted interrupt re-enable... ``` 只能在中斷時 isr 中斷處理程式中使用因此大部分都會使用 `spin_lock_irqsave` 來上鎖。參考資料: [spin_lock_irqsave vs spin_lock_irq](https://stackoverflow.com/questions/2559602/spin-lock-irqsave-vs-spin-lock-irq) ::: - [ ] device.c `create_vcam_device` 使用 kzmalloc 配置一個 `struct vcam_device` 的空間， `v4l2_device_register` 將會註冊一個 v4l2_device 的空間並初始化， `vcam_out_videobuf2_setup` 將 video buffer 初始化，將在 videobuf.c 實作，將在之後說明，`video_set_drvdata` 將 device pruvate data 指向 video_device 中， `video_register_device` 註冊一個 video_device，其中 VFL_TYPE_GRABBER 對應為 /dev/videoX for video input/output devices， `init_framebuffer` 初始化 framebuffer 將 fb.c 中實作， `vcam_in_queue_setup` 初始化 `&vcam->in_queue` 會在 videobuf.c 實作。 :::info `struct vcam_device -> struct video_device -> struct device -> void *driver_data = struct vcam_device *vcam` 所以才可以應用 `struct vcam_device *dev = (struct vcam_device *) video_drvdata(file);` 呼叫出 `struct vcam_device` 因為 `video_devices[iminor(file_inode(file))]` 可以得到 `struct video_device` 又透過 `struct device -> driver_data` 得到 `struct vcam_device` ::: 其中 `struct v4l2_ioctl_ops vcam_ioctl_ops` 也是 compliance 測試的地方， ```cpp static const struct v4l2_ioctl_ops vcam_ioctl_ops = { .vidioc_querycap = vcam_querycap, //查詢驅動的功能 .vidioc_enum_input = vcam_enum_input, .vidioc_g_input = vcam_g_input, //Query or select the current video input .vidioc_s_input = vcam_s_input, .vidioc_enum_fmt_vid_cap = vcam_enum_fmt_vid_cap, .vidioc_g_fmt_vid_cap = vcam_g_fmt_vid_cap, //Get or set the data format, try a format .vidioc_try_fmt_vid_cap = vcam_try_fmt_vid_cap, .vidioc_s_fmt_vid_cap = vcam_s_fmt_vid_cap, .vidioc_s_parm = vcam_g_parm, //Get or set streaming parameters .vidioc_g_parm = vcam_s_parm, .vidioc_enum_frameintervals = vcam_enum_frameintervals, //Enumerate frame intervals .vidioc_enum_framesizes = vcam_enum_framesizes, //Enumerate frame sizes .vidioc_reqbufs = vb2_ioctl_reqbufs, //Initiate Memory Mapping .vidioc_create_bufs = vb2_ioctl_create_bufs, //Create buffers for Memory Mapped .vidioc_prepare_buf = vb2_ioctl_prepare_buf, .vidioc_querybuf = vb2_ioctl_querybuf, //Query the status of a buffer .vidioc_qbuf = vb2_ioctl_qbuf, .vidioc_dqbuf = vb2_ioctl_dqbuf, .vidioc_expbuf = vb2_ioctl_expbuf, .vidioc_streamon = vb2_ioctl_streamon, //Start or stop streaming I/O .vidioc_streamoff = vb2_ioctl_streamoff}; ``` 其中 `vb2_ioctl_xx` 被定義在 `<media/v4l2-ioctl.h>` 中，主要對 video buffer 進行操作。 `vcam_x_xx` 為 vcam 所建立的 function 。 `vcam_querycap` 設定 vcam 名字及目標功能，可以由<uapi/linux/videodev2.h> 查到相關 capability macro。 `vcam_enum_input` 設定 input 名稱，可以查到相關<uapi/linux/videodev2.h> intput macro。 `vcam_g_input` 與 `vivid_g_input_tch` 一樣。 `vcam_s_input` 與 `vivid_set_touch` 一樣。 `vcam_enum_fmt_vid_cap` 為設定 fromt video capability 設定輸入的規格(如: RGB, YUYV) 與 `vivid_enum_fmt_cap` 可以互相對照。 `vcam_g_parm` 和 `vcam_s_parm` 一樣，只有在 vcam_s_parm 有設定 timeperframe ，這也是為什麼要顛倒，因為在 compliance test g 時有測 timperframe。 `vcam_try_fmt_vid_cap` vivid 部分是將 multi plane 的資料，合併為 single plane ，而 vcam 是single plane 設定。 `vcam_enum_frameintervals` `vcam_enum_framesizes` 與 `vidioc_enum_framesizes` 一樣 `vb2_ioctl_reqbufs` 其中 `fill_buf_caps(vdev->queue, &p->capabilities);` 是在 [Merge tag 'media/v4.20-2'](https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/diff/drivers/media/common/videobuf2?h=linux-5.8.y&id=b3491d8430dd25f0a4e00c33d60da22a9bd9d052) 時才新增的，所以就算 compiler 沒出現 error warning function 還是有可能已經受到更新更改。 `video_device` ```cpp static const struct video_device vcam_video_device_template = { .fops = &vcam_fops, .ioctl_ops = &vcam_ioctl_ops, .release = video_device_release_empty, }; ``` :::info `video_device` 定義在 include/media/v4l2-dev.h struct video_device - Structure used to create and manage the V4L2 device nodes. fops: pointer to &struct v4l2_file_operations for the video device * struct v4l2_file_operations - file system operations used by a V4L2 device release: video device release() callback ioctl_ops: pointer to &struct v4l2_ioctl_ops with ioctl callbacks * struct v4l2_ioctl_ops - describe operations for each V4L2 ioctl ::: - [ ] videobuf.c `vcam_in_queue_setup` 在 `init_vcam_in_buffer` 對 queue 中初始化，由 `q->pending = &q->buffers[0];` 及 `q->ready = &q->buffers[1];` 可以知道，這資料結構為 queue FIFO 的結構。 `swap_in_queue_buffers` 就是對上面 queue 的維護，當 ready 和 pending 重疊時，通常為 queue 為 fill 的狀況，才需要 swap ，可以在 fb.c 觀察到。前面有提到的 `vcam_out_videobuf2_setup` 其中 queue 結構為 `struct vb2_queue` `q->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;` 使用單畫面擷取，適用於畫面捕捉。 `q->io_modes = VB2_MMAP | VB2_USERPTR | VB2_READ;` 代表 queue 的讀取方式 :::info `struct vb2_queue` struct vb2_queue — a videobuf queue type - private buffer type whose content is defined by the vb2-core caller. For example, for V4L2, it should match the V4L2_BUF_TYPE_xx V4L2_BUF_TYPE_VIDEO_CAPTURE - Buffer of a single-planar video capture stream io_modes - supported io methods (see vb2_io_modes enum) vb2_io_modes - queue access methods 參考資料: [vb2_queue](https://linuxtv.org/downloads/v4l-dvb-internals/device-drivers/API-struct-vb2-queue.html) [type](https://www.kernel.org/doc/html/v4.15/media/uapi/v4l/buffer.html) [io_modes](https://linuxtv.org/downloads/v4l-dvb-internals/device-drivers/API-enum-vb2-io-modes.html) ::: `vcam_vb2_ops` 存放對於 vb2_buffer 的操作方式及 queue 的操作，其中要注意 `vcam_start_streaming` `vcam_start_streaming` - 主要是當 streaming 他會將 `vcam_in_queue` 的 buffer 內容複製到 `vcam_out_queue` 當中。 `kthread_create(submitter_thread, dev, "vcam_submitter");` 目的就是 copy buffer `vcam_out_buffer_queue` 會將 vb2_queue 串入 queue 中輸出 - [ ] fb.c `init_framebuffer` 主要執行 `proc_create_data` 會在 proc 中建立檔案， proc 為一個 virtual file system ，主要提供目前執行的 process 信息，也提供 kernel 和 user space 通訊，0666 為設定檔案權限，參數 file_operations vcamfb_fops 將會把 file write open release 文件操作方式寫入， data 為任何要傳入 proc 的信息。 :::info `extern struct proc_dir_entry *proc_create_data(const char *, umode_t, struct proc_dir_entry *, const struct file_operations *, void *);` ```cpp struct proc_dir_entry *proc_create_data(const char *name, umode_t mode, struct proc_dir_entry *parent, const struct proc_ops *file_operations, void *data) { struct proc_dir_entry *p; p = proc_create_reg(name, mode, &parent, data); if (!p) return NULL; p->proc_ops = proc_ops; pde_set_flags(p); return proc_register(parent, p); } ``` 如前面所說，在 kernel 5.7 之後，proc_create_data 參數從 const struct file_operations 改為 const struct proc_ops ，這也是需要注意的地方。 proc >Proc file system (procfs) is virtual file system created on fly when system boots and is dissolved at time of system shut down. It contains the useful information about the processes that are currently running, it is regarded as control and information centre for kernel. The proc file system also provides communication medium between kernel space and user space. `$ls -l /proc` ```shell -rw-rw-rw- 1 root root 0 5月 8 18:35 vcamfb0 -r--r--r-- 1 root root 0 5月 8 15:33 version -r--r--r-- 1 root root 0 5月 8 18:35 version_signature -r-------- 1 root root 0 5月 8 18:35 vmallocinfo ``` 參考資料: [proc file system](https://www.geeksforgeeks.org/proc-file-system-linux/) ::: 並`struct vcam_out_buffer` 包括了由 head_list 其連接， `struct vb2_buffer` 的 video buffer。 :::info `struct vb2_buffer` struct vb2_buffer - represents a video buffer, that can be read by the driver and relevant entries can be changed by the driver in case of CAPTURE types 參考資料: [vb2_buffer](https://docs.huihoo.com/doxygen/linux/kernel/3.7/structvb2__buffer.html) ::: - [ ] vcam-util.c `create_device` 會設置 dev.width = 640, .height = 480 .pix_fmt = VCAM_PIXFMT_RGB24 ，`ioctl(fd, VCAM_IOCTL_CREATE_DEVICE, dev)` 將會創建 device 。 `remove_device` `ioctl(fd, VCAM_IOCTL_DESTROY_DEVICE, dev)` 將 device remove 。 `modify_device` `ioctl(fd, VCAM_IOCTL_GET_DEVICE, &orig_dev)` 先得到 device format 在 `ioctl(fd, VCAM_IOCTL_MODIFY_SETTING, dev)` 修改 foramt 。 `list_devices` `ioctl(fd, VCAM_IOCTL_GET_DEVICE, &dev)` 把所有 device 列出來。 :::info `int ioctl(int fd, unsigned long request, ...);` ioctl - control device > The ioctl() system call manipulates the underlying device parameters of special files. In particular, many operating characteristics of character special files (e.g., terminals) may be controlled with ioctl() requests. The argument fd must be an open file descriptor. The second argument is a device-dependent request code. Third argument is an untyped pointer to memory. Usually, on success zero is returned. ioctl 將會以第二參數，操作檔案，以下為對應的操作代碼 - [ ] vcam.h ```cpp #define VCAM_IOCTL_CREATE_DEVICE 0x111 #define VCAM_IOCTL_DESTROY_DEVICE 0x222 #define VCAM_IOCTL_GET_DEVICE 0x333 #define VCAM_IOCTL_ENUM_DEVICES 0x444 #define VCAM_IOCTL_MODIFY_SETTING 0x555 ``` 參考資料: [ioctl](https://man7.org/linux/man-pages/man2/ioctl.2.html) ::: ### 理解 v4l2 框架 #### [v4l2 (video for linux version 2) 框架](https://android.googlesource.com/kernel/msm/+/android-msm-bullhead-3.10-marshmallow-dr/Documentation/video4linux/v4l2-framework.txt) v4l2 Structure of the framework >The framework closely resembles the driver structure: it has a v4l2_device struct for the device instance data, a v4l2_subdev struct to refer to sub-device instances, the video_device struct stores V4L2 device node data and the v4l2_fh struct keeps track of filehandle instances. The V4L2 framework also optionally integrates with the media framework. If a driver sets the struct v4l2_device mdev field, sub-devices and video nodes will automatically appear in the media framework as entities. 可以對應下圖進行了解， video_device 將會存取 v4l2_device , v4l2_fh ，而 v4l2_device 將會保存 v4l2_subdev 也就是子系統並使用 head_list 的方式將所有子系統連接起來。 ![](https://i.imgur.com/xK5YXDM.png) `video_register_device` :::info 參考資料: [2020 final project (vcam)](https://hackmd.io/@eecheng/B16rQ3GjU#2020q1-Final-Project-vcam) [linux kernel Media Subsystem documentation](https://www.kernel.org/doc/html/v4.14/media/kapi/v4l2-intro.html) [v4l2 framworks](https://elinux.org/images/8/89/V4l2-frameworks_0.pdf) ::: ## 今年的改進 ### vcam 支援最近的 kernel 版本更新 vcam 使他能支援從原本 linux kernel v4.15~v5.8 首先了解 vivid :::info The Virtual Video Test Driver (vivid) >This driver emulates video4linux hardware of various types: video capture, video output, vbi capture and output ... In addition a simple framebuffer device is available for testing capture and output overlays. 一個由 linux kernel 所維護的 virtual video device ，會隨著 kernel 不斷的更新，可以當作版本更新後驅動程式修改的參考， vcam 專案也參考 vivid 所做的框架, function 等... ，所以可以透過 vivid 對 function 或 API 的修正，vcam 也做出對應的調整，使 module 也可以適應 kernel 更新。參考資料: [The Virtual Video Test Driver](https://linuxtv.org/downloads/v4l-dvb-apis-new/admin-guide/vivid.html) ::: 在 kernel v5.8 vivid 的變化由 git 可以知道被修正的地方 [media: vivid: fix incorrect PA assignment to HDMI outputs](https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/diff/drivers/media/platform/vivid/vivid-core.c?h=linux-5.8.y&id=063d1942247668eb0bb800aef5afbbef337344be) 中可以確定 `VFL_TYPE_GRABBER` 可以改為 `VFL_TYPE_VIDEO`。在 krenel v5.6 genernic.c 的變化由 git 可以知道被修正的地方 [proc: decouple proc from VFS with "struct proc_ops"](https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-5.6.y&id=d56c0d45f0e27f814e87a1676b6bdccccbc252e9) 中可以確定由 `struct file_operations` 換成 `struct proc_ops` 將上面進行修改後可以成功編譯，所以可以經過 `include version.h` 來知道你的 kernel 的版本，來進行調整 `function` 變化，我有嘗試 v5.11 也一樣可以正常執行。 ### 支援其他解析度 720p ![](https://i.imgur.com/gL2rNFG.png) ![](https://i.imgur.com/iTnp2RB.png) #### 了解 vivid 如何調整解析度 ```cpp if (vivid_is_webcam(dev)) { const struct v4l2_frmsize_discrete *sz = v4l2_find_nearest_size(webcam_sizes, VIVID_WEBCAM_SIZES, width, height, mp->width, mp->height); w = sz->width; h = sz->height; ``` 由 `vivid_try_fmt_vid_cap` 可以觀察到裡面使用到 `v4l2_find_nearest_size` 會找到適合的解析度大小，解析度大小由 `webcam_sizes` 中定義如(640 * 480 , 1280 * 720 等...)，其中的期望的解析度大小由 mp->height 及 mp->width 控制。 mp 為 `struct v4l2_pix_format_mplane` 為 multiplanar format definition ，再來觀察期中 mp->height 及 mp->width 在哪被給予大小，在 ` vivid_g_fmt_vid_cap` 中有對 `mp->height = dev->fmt_cap_rect.height;` 及 `mp->width = dev->fmt_cap_rect.width;` 其中 `fmt_cap_rect` 跟設備有關係。在 `vivid_s_fmt_vid_cap` 裡有使用到 `vivid_update_format_cap` 這 function 的功用就是由不同的輸入源更新對應的大小及設定， #### 經過修改後: 透過設定 `vcam_sizes` array 將存放著 vcam 支援的 frame size ，經過 `v4l2_find_nearest_size` 在 `vcam_sizes` 中找到適合的大小並設定 `vcam_update_foramt_cap` 設定 buffer 大小，並重新 `vmalloc` buffer ，當 vcam module.c 中 `allow_scaling` 將他 enable 時就會將 480p 轉 720p。 ```cpp static const struct v4l2_frmsize_discrete vcam_sizes[] = { {480, 360}, {VGA_WIDTH, VGA_HEIGHT}, /* 640 * 480 */ {HD_720_WIDTH, HD_720_HEIGHT}, /* 1280 * 720 */ }; ``` ```shell $ v4l2-ctl -d /dev/video0 --list-formats-ext ioctl: VIDIOC_ENUM_FMT Type: Video Capture [0]: 'RGB3' (24-bit RGB 8-8-8) Size: Discrete 1280x720 Interval: Stepwise 0.017s - 1001.000s with step 0.017s (0.001-59.940 fps) ``` 也因為這樣設定，當修改 `v4l2_find_nearest_size` 中的參數時就能支援 360p。 ```shell ioctl: VIDIOC_ENUM_FMT Type: Video Capture [0]: 'RGB3' (24-bit RGB 8-8-8) Size: Discrete 480x360 Interval: Stepwise 0.017s - 1001.000s with step 0.017s (0.001-59.940 fps) ``` ### 檢查 vcam 是否有 memory leak 一般都瞭解如何在 userspace 檢查是否有 memory leak 可以透過 Address Sanitizer(ASan) 或 valgrind，由 [ gcc -fsanitize=leak ](https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html) 編譯時開啟或 [valgrind --leak-check=full](https://valgrind.org/)，就可以檢查該程式是否有未釋放的記憶體，但是由於 vcam 將會變成 kernel module ，所以就不能使用這方式，因此 kernel 也有屬於 kernel 的檢查 memory leak 的方式，可以透過 [kernel Sanitizer(KASAN)](https://www.kernel.org/doc/html/v5.8/dev-tools/kasan.html) 還有一個 [kmemleck](https://www.kernel.org/doc/html/v5.8/dev-tools/kmemleak.html) 也可以檢查 kernel 的 memory leak :::info KASAN 適用於 SLUB 及 SLAB 記憶體配置，並支援 x86, arm, arm64 ， Kernel memory leak director(kmemleak) 也適用於適用於 SLUB 及 SLAB 記憶體配置，並支援 x86, arm, arm64，兩者都需要透過 kernel hacking 設定 CONFIG_DEBUG_KMEMLEAK=y 及 CONFIG_KASAN=y 之後編譯自己的 kernel ，這裡編譯後需要空間要較大的容量及時間，大約需要 40G 左右，編譯後並取代自己本來的 kernel ，並重新開機完成後可以用 `sudo ls /sys/kernel/debug/` 檢查是否多了 kmemleak 也可以由 `uname -rms` 知道自己 kernel version。想嘗試可以使用 `kmemleak-test` 可以知道功能是否能正常使用， `kmemleak-test` 為 kernel 中嘗試 kmemleak 的 module，也是需要透過 kernel hacking 進行設定。 ```shell # modprobe kmemleak-test # echo scan > /sys/kernel/debug/kmemleak ``` ```shell # cat /sys/kernel/debug/kmemleak unreferenced object 0xffff88814aa9d640 (size 32): comm "modprobe", pid 3004, jiffies 4294943661 (age 528.300s) hex dump (first 32 bytes): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ backtrace: [<0000000077ade272>] kmem_cache_alloc_trace+0x101/0x2a0 [<00000000e9f0731a>] 0xffffffffc0f00047 [<0000000099bba30a>] do_one_initcall+0x9c/0x2d0 [<000000005588bf62>] do_init_module+0x10f/0x3c0 [<00000000ad39151b>] load_module+0x4462/0x48a0 [<000000001b11a8d1>] __do_sys_finit_module+0x130/0x1c0 [<000000002a20745d>] __x64_sys_finit_module+0x43/0x50 [<0000000058fe518f>] do_syscall_64+0x52/0xc0 [<00000000a17cd98c>] entry_SYSCALL_64_after_hwframe+0x44/0xa9 ``` ```shell # echo clear > /sys/kernel/debug/kmemleak ``` :bell: 要注意剛 `insmod` 或 `modprobe` module 不能馬上 scan 需要一段時間後再 scan 才可以檢查出 leak。 ::: 發現到 kmemleak 沒辦法出現報告，但是 KASAN 有報告。 #### vcam KASAN 報告先了解一下 KASAN 的報告由 [The Kernel Address Sanitizer (KASAN)](https://www.kernel.org/doc/html/latest/dev-tools/kasan.html) 可以了解報告的內容解讀。 ```shell [ 1782.469808] ================================================================== [ 1782.469828] BUG: KASAN: slab-out-of-bounds in __init_vb2_v4l2_buffer+0x19/0x30 [videobuf2_v4l2] [ 1782.469831] Write of size 4 at addr ffff88814e99a21c by task vlc/8923 [ 1782.469837] CPU: 3 PID: 8923 Comm: vlc Tainted: G OE 5.8.6 #1 [ 1782.469839] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 1782.469840] Call Trace: [ 1782.469858] dump_stack+0x9d/0xda [ 1782.469863] print_address_description.constprop.0+0x1f/0x210 [ 1782.469867] ? _raw_spin_lock_irqsave+0x8e/0xf0 [ 1782.469869] ? _raw_spin_trylock_bh+0x100/0x100 [ 1782.469875] ? __init_vb2_v4l2_buffer+0x19/0x30 [videobuf2_v4l2] [ 1782.469877] kasan_report.cold+0x37/0x7c [ 1782.469882] ? __init_vb2_v4l2_buffer+0x19/0x30 [videobuf2_v4l2] [ 1782.469886] __asan_store4+0x8b/0xb0 [ 1782.469890] __init_vb2_v4l2_buffer+0x19/0x30 [videobuf2_v4l2] [ 1782.469897] __vb2_queue_alloc+0x1d3/0x660 [videobuf2_common] [ 1782.469904] vb2_core_reqbufs+0x45a/0x690 [videobuf2_common] ... [ 1782.470068] ? __ia32_compat_sys_ppoll_time64+0x1a0/0x1a0 [ 1782.470071] do_syscall_64+0x52/0xc0 [ 1782.470074] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1782.470076] RIP: 0033:0x7f467e89faff [ 1782.470079] Code: 54 24 1c 48 89 74 24 10 48 89 7c 24 08 e8 79 1c f8 ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 48 8b 7c 24 08 b8 07 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2b 44 89 c7 89 44 24 08 e8 ad 1c f8 ff 8b 44 [ 1782.470080] RSP: 002b:00007f465c8f5a90 EFLAGS: 00000297 ORIG_RAX: 0000000000000007 [ 1782.470083] RAX: ffffffffffffffda RBX: 00007f4664000c20 RCX: 00007f467e89faff [ 1782.470084] RDX: 00000000ffffffff RSI: 0000000000000002 RDI: 00007f465c8f5b90 [ 1782.470085] RBP: 0000000000000001 R08: 0000000000000001 R09: 0000000000000001 [ 1782.470086] R10: 0000000000000001 R11: 0000000000000297 R12: 0000000000000001 [ 1782.470088] R13: 0000000000000001 R14: 0000000001000000 R15: 00007f4656fff010 [ 1782.470092] Allocated by task 8923: [ 1782.470095] save_stack+0x23/0x50 [ 1782.470097] __kasan_kmalloc.constprop.0+0xcf/0xe0 [ 1782.470098] kasan_kmalloc+0x9/0x10 [ 1782.470100] __kmalloc+0x146/0x2f0 [ 1782.470105] __vb2_queue_alloc+0xae/0x660 [videobuf2_common] [ 1782.470111] vb2_core_reqbufs+0x45a/0x690 [videobuf2_common] [ 1782.470116] __vb2_init_fileio+0x189/0x470 [videobuf2_common] [ 1782.470121] vb2_core_poll+0x260/0x320 [videobuf2_common] [ 1782.470126] vb2_poll+0x34/0xd0 [videobuf2_v4l2] [ 1782.470130] vb2_fop_poll+0x9f/0x1b0 [videobuf2_v4l2] [ 1782.470149] v4l2_poll+0xf2/0x110 [videodev] [ 1782.470151] do_sys_poll+0x454/0x780 [ 1782.470153] __x64_sys_poll+0x99/0x230 [ 1782.470155] do_syscall_64+0x52/0xc0 [ 1782.470157] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1782.470159] Freed by task 8791: [ 1782.470161] save_stack+0x23/0x50 [ 1782.470163] __kasan_slab_free+0x137/0x180 [ 1782.470164] kasan_slab_free+0xe/0x10 [ 1782.470166] kfree+0xa8/0x280 [ 1782.470168] load_elf_binary+0x13a2/0x244f [ 1782.470171] __do_execve_file.isra.0+0x9e6/0x11f0 [ 1782.470173] __x64_sys_execve+0x59/0x70 [ 1782.470175] do_syscall_64+0x52/0xc0 [ 1782.470177] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 1782.470180] The buggy address belongs to the object at ffff88814e99a000 which belongs to the cache kmalloc-1k of size 1024 [ 1782.470183] The buggy address is located 540 bytes inside of 1024-byte region [ffff88814e99a000, ffff88814e99a400) [ 1782.470184] The buggy address belongs to the page: [ 1782.470189] page:ffffea00053a6600 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 head:ffffea00053a6600 order:3 compound_mapcount:0 compound_pincount:0 [ 1782.470192] flags: 0x17ffffc0010200(slab|head) [ 1782.470196] raw: 0017ffffc0010200 ffffea00051f7000 0000000200000002 ffff8881c8c0ea00 [ 1782.470199] raw: 0000000000000000 0000000080100010 00000001ffffffff 0000000000000000 [ 1782.470199] page dumped because: kasan: bad access detected [ 1782.470201] Memory state around the buggy address: [ 1782.470204] ffff88814e99a100: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1782.470206] ffff88814e99a180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 1782.470209] >ffff88814e99a200: 00 00 00 fc fc fc fc fc fc fc fc fc fc fc fc fc [ 1782.470210] ^ [ 1782.470212] ffff88814e99a280: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 1782.470215] ffff88814e99a300: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc [ 1782.470216] ================================================================== ``` 一開始為 KASAN 的 summary `slab-out-of-bounds in __init_vb2_v4l2_buffer+0x19/0x30 [videobuf2_v4l2]` 有越界的情況，在vb2_v4l2_buffer， `Call Trace:` 為記憶體的堆疊追蹤， `Allocated by task 8923:` 代表有記憶體配置的位置追蹤， `Freed by task 8791:` 為釋放記憶體配置的位置追蹤，接下來的部分為描述 slab object 資訊及 memory page 資訊，最後的部分為 `Memory state around the buggy address:` 這說明 KASAN 追蹤記憶體及地址 `0x00` 代表可配置、可用的記憶體， `0xFB` 代表已釋放的記憶體， `0xFC` 代表為 RedZone 區域，假如 KASAN 詢問，就會產生 slab-out-of-bounds 狀況。 `__init_vb2_v4l2_buffer` vcam 並沒有被使用到，但是在 `vcam_out_videobuf2_setup` 有使用到 `v4l2_buf_ops` 其中有個 `.init_buffer` 的操做，就會使用到 `__init_vb2_v4l2_buffer` 使用到 `to_vb2_v4l2_buffer` 就會用到 `container_of` 由 `vb2_buffer` 回到結構體 `vb2_v4l2_buffer` **這可能是會出現出 out-of-bounds 的原因** 但在 vcam 中 `vb2_buffer` 的結構體為 `vcam_out_buffer` ```cpp struct vb2_v4l2_buffer { struct vb2_buffer vb2_buf; __u32 flags; __u32 field; struct v4l2_timecode timecode; __u32 sequence; __s32 request_fd; bool is_held; struct vb2_plane planes[VB2_MAX_PLANES]; }; ``` ```cpp struct vcam_out_buffer { struct vb2_buffer vb; struct list_head list; size_t filled; }; ``` ### 當改變 videobuffer 大小後 vcamfb 要註冊失敗，並重新創立 #### 目標: 當 `conv_res_on` 轉換 video buffer 大小由 480p -> 720p 時 vcamfb 無法觀察到改變，期待 vcamfb 也可以跟著改變。 #### 研究: 在 `/proc/vcamfb` 使 `vcam` 不在是一般的 v4l2 device driver 他能夠接受主動寫入資料，其中 vcamfb 的操作 function `vcamfb_fops` 接受開啟、寫入、釋放。 ```cpp static struct proc_ops vcamfb_fops = { .proc_open = vcamfb_open, .proc_release = vcamfb_release, .proc_write = vcamfb_write, }; ``` 其中在 `vcamfb_write` 中有 `waiting_bytes = dev->input_format.sizeimage;` 代表 waiting 的資料大小會隨著 input 跟著改變，但是 default 的大小為 480p ，但是在執行動作時 input_format 改變成 720p ，所以才使得當你開啟 `conv_res_on` 也不會產生錯誤。 #### 實際修改 `device.c` ```cpp destroy_framebuffer((const char *) vcam->vcam_fb_fname); snprintf(vcam->vcam_fb_fname, sizeof(vcam->vcam_fb_fname), "vcamfb%d", MINOR(vcam->vdev.dev.devt)); struct proc_dir_entry *pde pde = init_framebuffer((const char *) vcam->vcam_fb_fname, vcam); if(!pde) destroy_framebuffer(vcam->vcam_fb_fname); vcam->vcam_fb_procf = pde; vcam->fb_isopen = 0; ``` #### 實際結果 ``` Available virtual V4L2 compatible devices: 1. vcamfb0(640,480,rgb24) -> /dev/video0 ``` 開啟 `conv_res_on` video 後 ```shell Available virtual V4L2 compatible devices: 1. vcamfb1(1280,720,rgb24) -> /dev/video0 ``` 觀察開啟 create device 是否會出現錯誤 ```shell Available virtual V4L2 compatible devices: 1. vcamfb1(1280,720,rgb24) -> /dev/video0 2. vcamfb2(1280,720,rgb24) -> /dev/video1 ``` :::info int minor(dev_t dev); >minor - manage a device number A device ID consists of two parts: a major ID, identifying the class of the device, and a minor ID, identifying a specific instance of a device in that class. A device ID is represented using the type dev_t. ::: 因為隨意的重新修改註冊名子會可能會覆蓋到其他 device 的 vcamfb 而產生錯誤。 ### 經過 pull request 建議我看[Linux Virtual Framebuffer Device Driver](https://github.com/hadi77ir/linux-module-virtfb) #### `virtual_fb.c` 在 `virtual_fb.c` 中 `vfb_init` 和 `vfb_setup`為 `__init` 初始化，`platform_driver_register` 對 `vfb_driver` 註冊 [platform_device](https://www.kernel.org/doc/html/latest/driver-api/driver-model/platform.html) 在 probe 中有 `register_framebuffer` 其中 `fb_ops` 將會是 framebuffer 的操作 ```cpp static const struct fb_ops vfb_ops = { /* For framebuffers with strange non linear layouts or that do not * work with normal memory mapped access */ .fb_read = fb_sys_read, .fb_write = fb_sys_write, .fb_check_var = vfb_check_var, /* checks var and eventually tweaks it to something supported, * DO NOT MODIFY PAR */ .fb_set_par = vfb_set_par, /* set the video mode according to info->var */ .fb_setcolreg = vfb_setcolreg, /* set color register */ .fb_pan_display = vfb_pan_display, /* pan display */ .fb_fillrect = sys_fillrect, /* Draws a rectangle */ .fb_copyarea = sys_copyarea, /* Copy data from area to another */ .fb_imageblit = sys_imageblit, /* Draws a image to the display */ .fb_mmap = vfb_mmap, /* perform fb specific mmap */ }; ``` :::info 詳細資料可以參考 [整理 virtual framebuffer 報告](https://hackmd.io/@WayneLin1992/Bk7MRdYpO) ::: 可以透過 `fbset` 了解到 framebuffer 信息，都會皆露出來。 `sudo fbset -fb /dev/fb1 --info` ```shell mode "640x480" geometry 640 480 640 480 24 timings 0 0 0 0 0 0 0 rgba 8/0,8/8,8/16,0/0 endmode Frame buffer device information: Name : vcamfb Address : 0xffffa8dfc39e5000 Size : 921600 Type : PACKED PIXELS Visual : TRUECOLOR XPanStep : 1 YPanStep : 1 YWrapStep : 1 LineLength : 1920 Accelerator : No ``` 當開啟 scaling 之後 ``` mode "1280x720" geometry 1280 720 1280 720 24 timings 0 0 0 0 0 0 0 rgba 8/0,8/8,8/16,0/0 endmode Frame buffer device information: Name : vcamfb Address : 0xffffa8dfc6549000 Size : 2764800 Type : PACKED PIXELS Visual : TRUECOLOR XPanStep : 1 YPanStep : 1 YWrapStep : 1 LineLength : 3840 Accelerator : No ``` 並修改 fb_ops 將原本 proc_ops 取代，使得 /dev/fbX 就能接受 raw data 顯示出圖形。為了使操作介面更為簡潔，所以只在 `device.h` 中只在 `struct vcam_device` 中存取 `void *fb_priv` ， `fb_priv` 的實際定義及操作會在 `fb.c` 中，這就是一種物件導向封裝的方式，對外面只接露出一個地址。可以透過 ` struct vcamfb_info *fb_data = (struct vcamfb_info *) dev->fb_priv;` 提取，在進行操作 - [ ] [fb.c](https://github.com/jserv/vcam/blob/master/fb.c) ```cpp struct vcamfb_info { struct fb_info info; /* framebuffer information */ void *addr; /* framebuffer address */ unsigned int offset; /* framebuffer offset */ char name[FB_NAME_MAXLENGTH]; /* framebuffer name */ }; ``` 需要 offset 是因為一開始 `vmalloc` 兩倍 buffer 大的空間，並透過 offset 分割成兩個 buffer 大小，來達到 double buffering。 ### 增加 crop 功能 #### 目標: crop 的功能說明 [Crop_factor](https://en.wikipedia.org/wiki/Crop_factor) 可以理解為擷取部分畫面，如圖所示，將本來圖片大小，只擷取部分想呈現的影像。 ![](https://i.imgur.com/XrHeOFY.jpg) 其中 crop size 也有規格對應的大小 ![](https://i.imgur.com/Tww4xgD.png) #### 研究: vivid 的 webcam 是不支援 crop 的，但還是可以了解到 vivid 設定了大小 `struct v4l2_rect crop_cap` 又設定了 `bool has_crop_cap;` 來確定是否執行 crop，由 `vivid_s_fmt_vid_cap` 擷取設定 crop 大小部分。 ```cpp if (dev->has_crop_cap && !dev->has_compose_cap) { struct v4l2_rect min_r = { 0, 0, r.width / MAX_ZOOM, factor * r.height / MAX_ZOOM }; struct v4l2_rect max_r = { 0, 0, r.width * MAX_ZOOM, factor * r.height * MAX_ZOOM }; v4l2_rect_set_min_size(crop, &min_r); v4l2_rect_set_max_size(crop, &max_r); v4l2_rect_map_inside(crop, &dev->crop_bounds_cap); ``` 透過 `v4l2_rect_set_min_size` 及 `v4l2_rect_set_max_size` 確定 crop 要比 max 小 min 大，假如沒有就設定 crop 的大小，`v4l2_rect_map_inside` 確保 crop 的大小，一定在 map 之中。 #### 實際修改 `module.c` ```cpp unsigned char allow_cropping = 0; module_param(allow_cropping, byte, 0); MODULE_PARM_DESC(allow_cropping, "Allow image cropping by default\n"); ``` `device.h` ```cpp struct v4l2_rect crop_cap; struct v4l2_pix_format crop_output_format; bool conv_crop_on; ``` `device.c` ```cpp extern unsigned char allow_cropping;` vcam->conv_crop_on = (bool) allow_cropping; ``` `vcam_try_fmt_vid_cap` ```cpp #include <media/v4l2-rect.h> #define MAX_ZOOM 4 #define factor 3 if (dev->conv_crop_on) { struct v4l2_rect *crop = &dev->crop_cap; struct v4l2_rect r = { 0, 0, f->fmt.pix.width, f->fmt.pix.height}; struct v4l2_rect min_r = { 0, 0, r.width * factor / MAX_ZOOM, r.height * factor / MAX_ZOOM }; struct v4l2_rect max_r = { 0, 0, r.width, r.height }; v4l2_rect_set_min_size(crop, &min_r); v4l2_rect_set_max_size(crop, &max_r); dev->crop_output_format = dev->output_format; dev->crop_output_format.width = crop->width; dev->crop_output_format.height = crop->height; pr_debug("Output crop %d\n", dev->conv_crop_on); f->fmt.pix.width = dev->crop_output_format.width; f->fmt.pix.height = dev->crop_output_format.height; } ``` `vcam_g_fmt_vid_cap` ```cpp if (dev->conv_crop_on){ memcpy(&f->fmt.pix, &dev->crop_output_format, sizeof(struct v4l2_pix_format)); } ``` `vcam_s_fmt_vid_cap` ```cpp if(dev->conv_crop_on && !dev->conv_res_on){ dev->crop_output_format = f->fmt.pix; } ``` #### 結果當沒開啟 scalering 時 crop 將 width height 會縮小4倍 ```shell Video input : 0 (vcam_in 0: ok) Format Video Capture: Width/Height : 480/360 Pixel Format : 'RGB3' (24-bit RGB 8-8-8) Field : None Bytes per Line : 1440 Size Image : 518400 ``` 將圖片 crop 3/4 如下圖 ![](https://i.imgur.com/m6g359w.png) 可以看到圖片還是有錯誤，這主要原因為 `bytesperline` 及寫入的大小關係，只要在 `bytesperline` 不影響，就會顯示下圖 ![](https://i.imgur.com/kHi3Lq1.png) ``` Video input : 0 (vcam_in 0: ok) Format Video Capture: Width/Height : 480/360 Pixel Format : 'RGB3' (24-bit RGB 8-8-8) Field : None Bytes per Line : 1920 Size Image : 691200 ``` 開啟 scaling -> 720p ![](https://i.imgur.com/6eofjI7.png) ``` Format Video Capture: Width/Height : 960/540 Pixel Format : 'RGB3' (24-bit RGB 8-8-8) Field : None Bytes per Line : 3840 Size Image : 2073600 ```

Read more

2021q1 Homework4 (quiz4)

2021q1 Homework2 (quiz2)

2021q3 Homework1 (quiz1)

2021q3 Homework2 (quiz2)