> 此處為 [學長筆記](https://hackmd.io/@xswzaq44321/ryyFgMffyx) 內所有 command 及程式的解釋
> [改好的 source code](https://github.com/weber-lin-nthu/NCKU_Computer-Architecture_Qemu-SystemC-Co-Simulation)
# QEMU
## QEMU (`qemu-6.2.0-rc2/`)
```sh
# 在解壓縮 QEMU 後的 qemu-6.2.0-rc2，新增 build 資料夾後進去
../configure --target-list="arm-softmmu,arm-linux-user" --prefix=${HOME}/workspace/qemu-bin-6.2.0-rc2
```
* `target-list`: 只編譯指定的模擬器 target
所有 target 可在 `configs/targets` 看到
* `arm-softmmu`: 針對整個平台 (包含 CPU 與周邊硬體) 做模擬 (如 realview versatile family)，執行檔為 `qemu-bin-6.2.0-rc2/qemu-system-arm`
realview versatile family 為 ARM 的一系列硬體開發平台
* `arm-linux-user`: 只做 CPU 指令集架構的轉換模擬，執行檔為 `qemu-bin-6.2.0-rc2/qemu-arm`
* `prefix`: 設定安裝在此資料夾
```sh
make -j $(nproc)
make install
```
* `make -j $(nproc)`: 編譯 QEMU 原始碼，產生 `qemu-system-arm` 跟 `qemu-arm` 執行檔等檔案。 `-j $(nproc)` 為根據 CPU core 數量做平行處理加快編譯速度
* `make install`: 把那些編譯好的檔案複製到剛剛 configure 設定的 `prefix` 資料夾
## Cross Compile Toolchain (`arm-2014.05/`)
* `arm-2014.05/bin/` 內會放 cross compile toolchain 的執行檔。例如 `arm-2014.05/bin/arm-none-linux-gnueabi-gcc`:
* `arm`: target 架構為 arm
* `none`: 表示沒有特定廠商限制，為通用
* `linux`: target OS 為 Linux
* `gnueabi`
* EABI (Embedded Application Binary Interface) 為 ARM 定義的 ABI 標準，包含:
* 函式怎麼呼叫 (calling convention)
* 資料怎麼排版（結構對齊、stack 對齊）
* 參數怎麼傳遞（register 還是 stack）
* 哪些暫存器由 caller/callee 負責保存
* 浮點參數如何處理
* GNU EABI 即為 GNU 針對 EABI 的一個實作
## Linux Kernel (`linux-4.14.85/`)
* 原本需先下 `make ARCH=arm menuconfig` 來以互動式介面修改 `linux-4.14.85/.config` 檔來設定哪些功能、驅動程式、架構支援會被編譯進 kernel，此 Lab 直接使用 RealView EB 最簡單的設定檔複製到資料夾內
RealView EB (Emulation Baseboard) 為 ARM 官方的硬體開發板，用來模擬尚未量產的 ARM SoC
* `linux-4.14.85/.config` 範例
* `CONFIG_ARM=y`: 表示為 ARM 架構編譯的 kernel
* `CONFIG_MIGHT_HAVE_PCI=y`: kernel 可能支援 PCI 裝置
* `CONFIG_STACKTRACE_SUPPORT=y`: 啟用 stack 追蹤功能 (用來 debug crash)
* `CONFIG_PGTABLE_LEVELS=2`: 使用 2 階段 page table (ARMv5 架構常見)
```sh
make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- zImage
```
* cross compile linux kernel 給 ARM 架構使用，產生可開機的 kernel 壓縮映像檔 (zImage)
* `ARCH=arm`: target 架構為 arm
* `CROSS_COMPILE=arm-none-linux-gnueabi-`: 使用的 cross compile toolchain 的 prefix
* `zImage`: 要求產出一個壓縮過的 Linux kernel 映像檔，路徑為 `arch/arm/boot/zImage`
```sh
make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- dtbs
```
* cross compile linux kernel 中的 Device Tree Blob 檔案 (dtb) (硬體描述檔)
Device Tree 為一種文字化的硬體描述檔 (`.dts` (device tree source)、`.dtsi` (device tree source include，類似 C++ 的 header) 檔)，告訴 Linux:
1. 有哪些硬體裝置 (CPU、RAM、UART、Ethernet ...)
2. 它們的記憶體位置在哪裡
3. 有哪些 driver 要綁定到哪些裝置
編譯後會變成二進位 `.dtb` 檔（Device Tree Blob），在開機時由 bootloader (或 QEMU) 傳給 kernel
* `dtbs`: 要求 kernel 編譯 dtb 檔案。會編譯出多個 dtb 檔，路徑為 `arch/arm/boot/dts/`，我們只會用到 `arch/arm/boot/dts/arm-realview-eb.dtb`
## initrd ramdisk (initrd) 及 BusyBox (`busybox-1.29.3/`)
* initrd ramdisk (initrd) 為在啟動階段被 Linux kernel 呼叫的臨時檔案系統，用於 root file system (RFS) 被掛載之前的準備工作
* Linux 首先要將 kernel 載入到主記憶體。initrd 通常被壓縮成 gzip，開機時由 bootloader (如 LILO、GRUB) 來告知 kernel initrd 的位置，使其被 kernel 存取，掛載成一個 loop device
* loop device (`/dev/loop*`) 為 Linux 的一種虛擬裝置，允許把一個普通的檔案當成一個 block device 來掛載
* [BusyBox](https://busybox.net/) 為將許多小版本的 UNIX 工具 (ex: `ls`, `mount`) 包裝成一個執行檔，非常適合拿來組一個 initrd
* 同 [Linux kernel](https://hackmd.io/p4MBgIlaRbm7LBsd5Zh1gw?stext=1279%3A24%3A0%3A1749741786%3AjMkH5u&both=) 一樣需先下 `make ARCH=arm menuconfig` 修改設定來產生 .config，這裡也一樣直接複製提供好的設定檔
* 有一個 config `CONFIG_LINUXRC=y` 表示要編譯出 linuxrc 檔
linuxrc 為早期 linux kernel 掛載 initrd 之後第一個執行的程式 (比 `/sbin/init` 還早，其中一個原因為因為 `/sbin/init` 在 root file system 內，還沒被掛載)，現在的 linux kernel 不需要了
```sh
make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi-
make install CROSS_COMPILE=arm-none-linux-gnueabi-
```
* 同 [Linux kernel](https://hackmd.io/p4MBgIlaRbm7LBsd5Zh1gw?both=&stext=1757%3A65%3A0%3A1749743167%3AafNm-V) 的參數，只是 BusyBox `make install` 預設的位置是 `_install/`
* 安裝完之後，`_install/` 內再自己加入 `dev/`, `lib/` (內放 [cross compile toolchain](https://hackmd.io/p4MBgIlaRbm7LBsd5Zh1gw?both=#Cross-Compile-Toolchain-arm-201405) 的函式庫 (`.so`) 檔 (在 `arm-2014.05/arm-none-linux-gnueabi/libc/lib/`)), `opt/`, `proc/`, `sys/`, `tmp/`, `var/`, `home/` 及提供的 `etc/` 資料夾，即成為一個 initrd。最後使用 `cpio` 指令將所有檔案打包成一個 cpio archive，並用 `gzip` 壓縮成 `.gz`
* `etc/init.d/` 資料夾為早期 System V 系統在開機時用來啟動服務的所有 script。現代 Linux 皆已改用 systemd，但還是有這個資料夾，是為了向下相容
## 執行 QEMU
```sh
~/workspace/qemu-bin-6.2.0-rc2/bin/qemu-system-arm -M realview-eb -m 128M -cpu arm1136 -kernel zImage -initrd ./initrd.gz -nographic -serial mon:stdio -dtb arm-realview-eb.dtb
```
* `-M`: 指定要模擬的開發版
* `-m`: 指定模擬機的記憶體大小
* `-cpu`: 指定使用的 CPU 型號
* `-kernel`: kernel 映像檔路徑
* `-initrd`: initrd 路徑。**之後模擬機開機後的 root file system 即為 initrd**，因為在嵌入式環境通常不用再掛載外部磁碟，故也不需要真正的 root file system
* `-nographic`: 使用純文字模式
* `-serial mon:stdio`: 讓 QEMU 的 monitor (mon, 控制介面) 和模擬機的 serial port **共用 host 的 terminal 的 stdio**。這樣可以在同一個 terminal 透過 **切換 (按 ctrl-a c)** 來對模擬機的 serial port 做 I/O 或對 QEMU 的 monitor 做 I/O
QEMU 的 monitor 為用來控制和觀察模擬機狀態的介面，屬於 QEMU
模擬機的 serial port 是因為模擬機沒有顯示器，所以跟 Linux 互動可以透過 serial port
* `-dtb`: Device Tree Blob 路徑
## Virtual Hardware
此範例為定義一個 calculator 硬體計算單元 (character device)，透過 memory-mapped I/O (MMIO) 操作，並可支援 QEMU monitor 及 VMState 機制
### 定義
放在 `qemu-6.2.0-rc2/hw/char/`，表示為一個 character device
:::spoiler `caslab_calculator.h`
```cpp=
#ifndef CASLAB_CALCULATOR_H
#define CASLAB_CALCULATOR_H
#include "hw/sysbus.h"
#include "chardev/char-fe.h"
#include "hw/qdev-core.h"
#include "qapi/error.h"
#define TYPE_CASLAB_CALC "caslab_calc"
#define CASLAB_CALC(obj) OBJECT_CHECK(CASLabCalcState, \
(obj), \
TYPE_CASLAB_CALC); // OBJECT_CHECK() 為 QEMU 提供的 macro，為將 pointer 轉型並檢查型別是否正確
#define CASLAB_CALC_RegCount 6
/*
* 儲存此裝置的 state
*/
typedef struct {
SysBusDevice parent_obj; // 使用 C 模擬繼承關係，繼承自 SysBusDevice，表示此裝置是一個在 SysBus 上的裝置
MemoryRegion iomem; // 定義此裝置的 MMIO address 及大小
uint32_t Reg[6]; // 定義此裝置有 6 個 register (分別為 control, operator, operand1, operand2, result, status)
CharBackend chr; // 為 character device 與 host 通訊的管道
qemu_irq irq; // IRQ 線
const unsigned char *id; // 裝置 ID
} CASLabCalcState;
/*
* 創建裝置
*/
static inline DeviceState* caslab_calc_create(hwaddr addr,
qemu_irq irq,
Chardev *chr)
{
DeviceState *dev;
SysBusDevice *s;
dev = qdev_new(TYPE_CASLAB_CALC); // 建立名為 caslab_calc 的 QEMU 裝置
s = SYS_BUS_DEVICE(dev); // 將裝置轉型為 SysBusDevice parent class
qdev_prop_set_chr(dev, "chardev", chr); // 設定 QEMU property "chardev"。在 .c 檔有以 DEFINE_PROP_CHR 定義了 chardev 的 property。QEMU property 為裝置初始化時由 QEMU 傳給模擬裝置的設定值
qdev_realize_and_unref(dev, NULL, &error_fatal); // 正式實體化 (realize) 裝置
sysbus_mmio_map(s, 0, addr); // 把裝置的第 0 個 MMIO 區域 map 到系統 bus 的 addr 位址
sysbus_connect_irq(s, 0, irq); // 把裝置的第 0 條 IRQ 線連接到 CPU 的 irq interrupt port
return dev;
}
#endif
```
:::
:::spoiler `caslab_calculator.c`
```c=
#include "qemu/osdep.h"
#include "hw/sysbus.h"
#include "qemu/log.h"
// added
#include "migration/vmstate.h"
#include "hw/qdev-properties-system.h"
#include "hw/qdev-core.h"
#include "hw/irq.h"
#include "caslab_calculator.h"
//#define CASLAB_CALC_DEBUG
#ifdef CASLAB_CALC_DEBUG
#define DB_PRINT(...) \
do \
{ \
fprintf(stderr, "[DEBUG]: %s: ", __func__);\
fprintf(stderr, ##__VA_ARGS__); \
} while(0);
#else
#define DB_PRINT(...)
#endif
#define CALC_REG_CONTROL 0x00
#define CALC_REG_OPERATOR 0x01
#define CALC_REG_OPERAND1 0x02
#define CALC_REG_OPERAND2 0x03
#define CALC_REG_RESULT 0x04
#define CALC_REG_STATUS 0x05
/*
* 控制暫存器 (CALC_REG_CONTROL) 的各種操作指令
*/
#define CALC_CTRL_GO 0x01 // 開始運算 (呼叫 caslab_calc_op())
#define CALC_CTRL_ABORT 0x02 // 終止進行中的運算 (如果狀態是 CALC_STATUS_BUSY，會立刻變為 CALC_STATUS_IDLE 並發出 interrupt)
#define CALC_CTRL_CLRI 0x04 // 清除中斷 (clear IRQ, 代表 guest 程式已處理完結果)
#define CALC_STATUS_IDLE 0x00
#define CALC_STATUS_BUSY 0x01
#define CALC_OP_ADD 0x01
#define CALC_OP_SUB 0x02
#define CALC_OP_MUL 0x03
#define CALC_OP_DIV 0x04
static void caslab_calc_reset(DeviceState *dev); //forward decleration
static void caslab_calc_op(CASLabCalcState *s)
{
uint32_t op;
uint32_t src1, src2;
uint32_t result = 0;
if(s == NULL)
return;
s->Reg[CALC_REG_STATUS] = CALC_STATUS_BUSY;
op = s->Reg[CALC_REG_OPERATOR];
src1 = s->Reg[CALC_REG_OPERAND1];
src2 = s->Reg[CALC_REG_OPERAND2];
DB_PRINT("Doing OP %u with value %u, %u\n", op, src1, src2);
switch(op) {
case CALC_OP_ADD:
result = src1 + src2;
break;
case CALC_OP_SUB:
result = src1 - src2;
break;
case CALC_OP_MUL:
result = src1 * src2;
break;
case CALC_OP_DIV:
if(src2 == 0)
fprintf(stderr, "Calculator: Invalid operand_2 !!\n");
else
result = src1 / src2;
break;
default:
result = 0;
break;
}
s->Reg[CALC_REG_STATUS] = CALC_STATUS_IDLE;
s->Reg[CALC_REG_RESULT] = result;
qemu_irq_raise(s->irq); // 觸發 IRQ，同等於 qemu_set_irq(s->irq, 1);
}
/*
* 此裝置的 ID
*/
static const unsigned char caslab_calc_id_arm[8] =
{0x11, 0x10, 0x88, 0x08, 0x0d, 0xf0, 0x05, 0xb1};
/*
* 根據讀取的位址 (offset) 判斷要回傳裝置 ID 還是 register 值
*/
static uint64_t caslab_calc_read(void *opaque, hwaddr offset, unsigned size)
{
CASLabCalcState *s = opaque;
uint64_t ret = 0;
offset >>= 2; // 因為 register 是 32 bit，所以要把以 32 bit 為單位的位址 (0x0, 0x4, 0x8, ...) 轉為 array 的 index (0, 1, 2, ...) 時就要 shift 2 bit
switch(offset) {
case 0x3F8: // 0x3F8 是一般 legacy UART (COM1) 的位址，這邊不是拿來模擬 UART，而是把此區段當成裝置 ID 區
ret = caslab_calc_id_arm[0];
break;
case 0x3F9:
ret = caslab_calc_id_arm[1];
break;
case 0x3FA:
ret = caslab_calc_id_arm[2];
break;
case 0x3FB:
ret = caslab_calc_id_arm[3];
break;
case 0x3FC:
ret = caslab_calc_id_arm[4];
break;
case 0x3FD:
ret = caslab_calc_id_arm[5];
break;
case 0x3FE:
ret = caslab_calc_id_arm[6];
break;
case 0x3FF:
ret = caslab_calc_id_arm[7];
break;
case CALC_REG_OPERAND1:
case CALC_REG_OPERAND2:
case CALC_REG_RESULT:
case CALC_REG_STATUS:
ret = s->Reg[offset];
break;
default:
break;
};
DB_PRINT("Read value %lu @ 0x%02lx\n", ret, offset<<2);
return ret;
}
static void caslab_calc_write(void *opaque, hwaddr offset, uint64_t value, unsigned size)
{
CASLabCalcState *s = opaque;
DB_PRINT("Write value %lu @ 0x%02lx\n", value, offset);
offset >>= 2;
if(offset >= CASLAB_CALC_RegCount)
return;
else
{
switch(offset)
{
case CALC_REG_CONTROL:
if(CALC_STATUS_IDLE == s->Reg[CALC_REG_STATUS]) {
if( value == CALC_CTRL_GO ) {
DB_PRINT("CASlab Calculator commit\n");
caslab_calc_op(s);
} else if ( value == CALC_CTRL_CLRI ) {
qemu_set_irq(s->irq, 0);
}
} else if(CALC_STATUS_BUSY == s->Reg[CALC_REG_STATUS]) {
if ( value == CALC_CTRL_ABORT ) {
//Cancel operations
s->Reg[CALC_REG_STATUS] = CALC_STATUS_IDLE;
qemu_set_irq(s->irq, 1);
}
}
break;
case CALC_REG_OPERATOR://Writable register
case CALC_REG_OPERAND1:
case CALC_REG_OPERAND2:
if(s->Reg[CALC_REG_STATUS] == CALC_STATUS_IDLE)
s->Reg[offset] = value;
break;
default: //Read only register
break;
}
}
}
/*
* 在裝置被 instantiate 時會被呼叫
*/
static void caslab_calc_realize(DeviceState *dev, Error **errp)
{
//Nothing here curreltly
}
static void caslab_calc_reset(DeviceState *dev)
{
CASLabCalcState *s = CASLAB_CALC(dev);
memset(s->Reg, 0, CASLAB_CALC_RegCount * sizeof(*(s->Reg)));
qemu_set_irq(s->irq, 0);
}
/*
* 表示這個模擬裝置的 MMIO 存取操作規則 (當有 guest 程式對此裝置的記憶體做讀寫時，QEMU 就會參考這個來決定怎麼處理讀寫行為)
*/
static const MemoryRegionOps caslab_calc_ops = {
.read = caslab_calc_read,
.write = caslab_calc_write,
.endianness = DEVICE_NATIVE_ENDIAN,
.valid.min_access_size = 4, // 允許 guest 程式存取的最小資料長度
.valid.max_access_size = 4 // 允許 guest 程式存取的最大資料長度
};
static void caslab_calc_init(Object *obj)
{
// 型別轉換
SysBusDevice *sbd = SYS_BUS_DEVICE(obj);
CASLabCalcState *s = CASLAB_CALC(obj);
memory_region_init_io(&s->iomem, OBJECT(s), &caslab_calc_ops, s,
TYPE_CASLAB_CALC, 0x1000); // 初始化一塊 MMIO 區塊，大小為 4 KB
sysbus_init_mmio(sbd, &s->iomem); // 把此 MMIO 區塊掛載到 SysBus，讓 bus 可以存取到此塊位址
sysbus_init_irq(sbd, &s->irq); // 初始化 IRQ 腳位，註冊一條中斷線給此裝置
DB_PRINT("INIT\n");
}
/*
* 此部分為 optional，為 QEMU 定義此裝置的快照狀態描述 (VM migration state)
*/
static const VMStateDescription vmstate_caslab_calc = {
.name = TYPE_CASLAB_CALC,
.version_id = 1,
.minimum_version_id = 1,
.fields = (VMStateField[]) { // 需要保存的欄位
VMSTATE_UINT32_ARRAY(Reg, CASLabCalcState,
CASLAB_CALC_RegCount), // 儲存 CASLabCalcState 中的 Reg 這個 uint32 array，array 長度為 CASLAB_CALC_RegCount
VMSTATE_END_OF_LIST()
}
};
/*
* 定義一個 character device property，名稱為 "chardev"
*
* 常見的使用方式為:
* 1. 在 QEMU 啟動時先使用 -chardev 參數來建立一個 character device
* > ex:
* > -chardev socket,id=mysock,path=/tmp/mysock,server,nowait
* 2. 再使用 -device 參數將該 character device 指定給 caslab_calc 裝置
* > ex:
* > -device caslab_calc,chardev=mysock
* > 此設定會將 id 為 mysock 的 character device 綁定到 caslab_calc 裝置的 "chardev" property，進而對應到 CasLabCalcState.chr 欄位 (因為此段程式碼有設定好了)。這樣之後 caslab_calc 裝置就可以透過 chr 欄位來做字元 I/O ，實現與外部的通訊功能。但此 Lab 沒有用到這個
*/
static Property caslab_calc_properties[] = {
DEFINE_PROP_CHR("chardev", CASLabCalcState, chr),
DEFINE_PROP_END_OF_LIST(),
};
/*
* 「裝置類別 (device class)」的初始化
* (caslab_calc_init() 是針對 instance device 做初始化，這個是針對 class 做初始化)
*/
static void caslab_calc_class_init(ObjectClass *oc, void *data)
{
// 型別轉換
DeviceClass *dc = DEVICE_CLASS(oc);
// 註冊
dc->realize = caslab_calc_realize;
dc->vmsd = &vmstate_caslab_calc;
dc->reset = caslab_calc_reset;
dc->props_ = caslab_calc_properties;
}
/*
* 定義一個新的「裝置型別 (device type)」
* device type 為針對 device class 的描述
*/
static const TypeInfo caslab_calc_info = {
.name = TYPE_CASLAB_CALC,
.parent = TYPE_SYS_BUS_DEVICE, // 定義裝置是掛在 SysBus 上
.instance_size = sizeof(CASLabCalcState), // instance 大小
.instance_init = caslab_calc_init, // 註冊
.class_init = caslab_calc_class_init, // 註冊
};
static void caslab_calc_register_types(void)
{
type_register_static(&caslab_calc_info);
}
type_init(caslab_calc_register_types)
```
:::
### 設定
#### `qemu-6.2.0-rc2/hw/char/meson.build`
```meson
# ...
softmmu_ss.add(files('caslab_calculator.c')) # Add
# ...
```
表示使用 [softmmu](https://hackmd.io/p4MBgIlaRbm7LBsd5Zh1gw?both=&stext=270%3A7%3A0%3A1749910537%3ARXND8E) 模擬器時，ss (source set) 新增需要編譯此檔案
#### `qemu-6.2.0-rc2/hw/arm/realview.c`
```c
// ...
for (n = 0; n < 64; n++) {
pic[n] = qdev_get_gpio_in(dev, n);
}
sysbus_create_simple("caslab_calc", 0x80000000, pic[30]); // Add
// ...
```
建立一個簡單的 System Bus 裝置 "caslab_calc"，此裝置的實體位址為 0x80000000，並設定 PIC (在 ARM 稱為 GIC) 第 30 號 interrupt 線連接至此裝置 (則 IRQ number 為 30)
#### `linux-4.14.85/.config`
使用 `make ARCH=arm menuconfig` 修改 config "Enable loadable module support" 為開啟
### Kernel Module
此 kernel module 的功能為 calculator 的 driver
:::spoiler `calc_drv.c` (編譯成 `.ko` 來使用)
```c=
#include <linux/init.h>
#include <linux/module.h>
#include <linux/types.h>
#include <linux/fs.h>
#include <linux/cdev.h>
#include <linux/ioctl.h>
#include <linux/wait.h>
#include <linux/mm.h>
#include <linux/uaccess.h>
#include <linux/vmalloc.h>
#include <linux/of.h>
#include <linux/of_irq.h>
#include <asm/atomic.h>
#include <asm/io.h>
#include <asm/page.h>
#include <linux/device.h>
#include <asm/irq.h>
#include <linux/sched.h>
#include <linux/interrupt.h>
#include <linux/irqflags.h>
// 和 device 定義 (caslab_calculator.c) 內相同 ///////////////
#define CALC_REG_CONTROL 0x00
#define CALC_REG_OPERATOR 0x04
#define CALC_REG_OPERAND1 0x08
#define CALC_REG_OPERAND2 0x0C
#define CALC_REG_RESULT 0x10
#define CALC_REG_STATUS 0x14
#define CALC_STATUS_IDLE 0x00
#define CALC_CTRL_GO 0x01
#define CALC_CTRL_ABORT 0x02
#define CALC_CTRL_CLRI 0x04
#define CALC_CTRL_RESET 0x08
#define CALC_OP_ADD 0x01
#define CALC_OP_SUB 0x02
#define CALC_OP_MUL 0x03
#define CALC_OP_DIV 0x04
////////////////////////////////////////////////////////////
// Some CMD number is invaild(used by linux kernel)
// For detail read linux kernel document (Documentation/ioctl/ioctl-number.txt)
// and http://man7.org/linux/man-pages/man2/ioctl_list.2.html
#define CMD_STATUS 0x00
#define CMD_OP_SET 0x03
#define CMD_CTRL 0x04
#define BUFFER_SIZE 20
#define DRIVER_NAME "caslab_calc_hw"
#define DEVICE_NAME "caslab-calc"
#define DEVICE_COMP "caslab,calc" // device 的 compatible property 名稱，定義在 linux-4.14.85/arch/arm/boot/dts/arm-realview-eb.dtsi。每個 device node 都有一個 compatible property，用來描述該裝置的類型、廠商、相容性等資訊
// Can be found within linux kernel Documentation/admin-guide/devices.txt
// 252 is LOCAL/EXPERIMENTAL USE
#define CASCALC_MAJOR 252
#define CASCALC_MINOR 0
#define CASCALC_BASE 0x80000000
#define CASCALC_MEMREGION 0x1000
#define CASCALC_IRQ 30
MODULE_AUTHOR("DUN-JIE CHEN");
MODULE_LICENSE("GPL");
static DECLARE_WAIT_QUEUE_HEAD(wq); // DECLARE_WAIT_QUEUE_HEAD 為 linux 提供的 macro，定義 wq 為一個 driver 內的 waiting queue 的 head，用來放執行此 driver 的 user process，等待 interrupt 發生再繼續
static int flag = 0; // 0: 裝置尚未完成運算或 interrupt 尚未到來，process 還不能醒
// 1: 裝置已完成運算，interrupt 已發生，process 可被喚醒
static unsigned int irqnum = CASCALC_IRQ;
static int cascalc_major = CASCALC_MAJOR;
static int cascalc_minor = CASCALC_MINOR;
typedef struct {
unsigned int buffer[BUFFER_SIZE];
} DevicePrivate; // 每個使用此裝置的 process 都有一個此空間
static void __iomem *base; // 為 CASCALC_BASE (physical address) 的 virtual address
/*
* 透過 dev_compatible_name 字串在 device tree 中找到該節點，並取出它的 IRQ number
*/
static int get_dut_irq(const char* dev_compatible_name)
{
struct device_node* dev_node; // 作業系統筆記 p54
int irq = -1;
dev_node = of_find_compatible_node(NULL, NULL, dev_compatible_name); // 透過 compatible 名稱找出對應的 device node
if(!dev_node)
return -1;
irq = irq_of_parse_and_map(dev_node, 0);
of_node_put(dev_node); // 把剛剛 reference 的 device node 結構釋放 (降低 reference count)
return irq;
}
static DevicePrivate* allocate_memory_region(void)
{
DevicePrivate* _prv;
_prv = (DevicePrivate*)vmalloc(sizeof(DevicePrivate));
return _prv;
}
static void free_allocated_memory(DevicePrivate *_prv)
{
vfree(_prv);
_prv = NULL;
}
static int calc_open(struct inode *inode, struct file *file)
{
file->private_data = allocate_memory_region();
if(file->private_data == NULL)
return -ENOMEM;
return 0;
}
static int calc_release(struct inode *inode, struct file *file)
{
free_allocated_memory(file->private_data);
return 0;
}
static ssize_t calc_read(struct file *filp, char *buff, size_t len, loff_t *off)
{
int i;
DevicePrivate *prv_data = (DevicePrivate*)filp->private_data;
/* Only result is readable memory address */
if(len > 4)
len = 4;
for( i = 0; i < len/4; i++ )
prv_data->buffer[i] = readl(base + CALC_REG_RESULT + i * 4); // 從 MMIO 區域讀取 4 byte (l)
i = copy_to_user(buff, prv_data->buffer, len); // copy_to_user() 為 Linux 提供的 API，把 kernel space 的 buffer (prv_data->buffer) 複製到 user space 的 buffer (buff)
return len;
}
static ssize_t calc_write(struct file *filp, const char *buff, size_t len, loff_t *off)
{
int i;
DevicePrivate *prv_data = (DevicePrivate*)filp->private_data;
/* Two operands are writeable memory address */
if ( len > 8 )
len = 8;
i = copy_from_user(prv_data->buffer, buff, len);
for(i = 0; i < len/4; i++)
writel(prv_data->buffer[i], base + CALC_REG_OPERAND1 + i*4);
return len;
}
/*
* 較新版 kernel 從 ioctl 改成 unlocked_ioctl 了 (user program 呼叫還是用 ioctl())
* 可以把 ioctl 理解成一般操作 (read/write/open/...) 外的雜項
*/
static long calc_unlocked_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
{
long ret = 0;
unsigned int tmp;
int i;
printk("IOCTL CMD = %u, value = %lu\n", cmd, arg);
switch(cmd)
{
case CMD_STATUS: // 查看 status
tmp = readl(base + CALC_REG_STATUS);
i = copy_to_user((void*)arg, &tmp, sizeof(unsigned int));
case CMD_OP_SET: // 設定 operator
if(arg >= CALC_OP_ADD && arg <= CALC_OP_DIV)
writel(arg, base + CALC_REG_OPERATOR);
else
printk("Invaild OP %lu\n", arg);
break;
case CMD_CTRL: // 控制硬體行為
tmp = readl(base + CALC_REG_STATUS);
if(tmp == CALC_STATUS_IDLE) { // 裝置處於 idle，啟動計算
writel(arg, base + CALC_REG_CONTROL); // 寫入 control register 之後，根據 caslab_calculator.c 的定義，會讓 hardware 開始計算
wait_event_interruptible(wq, flag != 0); // 將呼叫此 ioctl 的 user process 放入 waiting queue wq，並等待 hardware 計算，hardware 計算完之後會發出 IRQ，CPU 被 interrupt，執行 cascalc_interrupt()，把 flag 設為 1、並執行 wake_up_interruptible() 把 user process 從 wq 移出。此時 (flag != 0) 為 true，就可以繼續往下執行了
flag = 0;
} else {
if(arg == CALC_CTRL_ABORT || arg == CALC_CTRL_RESET) {
writel(arg, base + CALC_REG_CONTROL);
flag = 0;
}
}
break;
default:
printk("Invaild CMD %u\n", cmd);
ret = -ENOTTY;
break;
}
return ret;
}
/*
* struct cdev 為 linux kernel 中管理 character device 的核心資料結構，在 driver 初始化時 (cascalc_init) 與 file_operations 一起註冊，使裝置能透過 /dev/ 介面被使用
*/
static struct cdev cascalc_cdev = {
.kobj = { .name = DRIVER_NAME }, // kernel object (kobject)，目前僅供內部使用或 debug
.owner = THIS_MODULE // 指定此裝置由目前這個 kernel module 擁有，可防止使用中被卸載
};
static struct file_operations cascalc_fops ={
.owner = THIS_MODULE,
.read = calc_read,
.write = calc_write,
.unlocked_ioctl = calc_unlocked_ioctl,
.open = calc_open,
.release = calc_release
};
/*
* 即 ISR，由 linux kernel 執行
*/
static irqreturn_t cascalc_interrupt(int irq, void *dev_id, struct pt_regs *regs)
{
// 參考 calc_unlocked_ioctl 的註解
if (irq == irqnum) {
printk("received interrupt from %s\n", DRIVER_NAME);
writel(CALC_CTRL_CLRI, base + CALC_REG_CONTROL);
printk("Release interrupt \n");
flag = 1;
wake_up_interruptible(&wq);
}
return IRQ_HANDLED;
}
/*
* 執行 insmod 時會呼叫此函數做初始化
*/
static int __init cascalc_init(void) {
dev_t dev = 0;
int result;
if(CASCALC_MAJOR) { // 有指定 major number
dev = MKDEV(cascalc_major, cascalc_minor); // linux kernel 提供的 macro，把 major number 跟 minor number 組合成 dev_t type
result = register_chrdev_region( dev, 1, DRIVER_NAME); // 註冊裝置號，註冊 1 個裝置
} else { // 沒有指定 major number
result = alloc_chrdev_region( &dev, cascalc_minor, 1, DRIVER_NAME); // 動態分配一個 major number (存入 dev)，註冊裝置號，註冊 1 個裝置
cascalc_major = MAJOR(dev);
}
if(result < 0) {
printk(KERN_ERR DRIVER_NAME "Can't allocate major number %d for %s\n", cascalc_major, DEVICE_NAME);
return -EAGAIN;
}
cdev_init( &cascalc_cdev, &cascalc_fops); // 註冊 struct cdev 及 operations
result = cdev_add( &cascalc_cdev, dev, 1); // 加入 linux kernel，正式成為可用的 character device
if(result < 0) {
printk(KERN_ERR DRIVER_NAME "Can't add device %d\n", dev);
return -EAGAIN;
}
/* allocate memory */
base = ioremap_nocache(CASCALC_BASE , CASCALC_MEMREGION); // 把 physical address CASCALC_BASE map 成 virtual address，才能讓 kernel 做存取
if(base == NULL)
return -ENXIO;
irqnum = get_dut_irq(DEVICE_COMP);
if(irqnum == -1) {
printk(KERN_ERR DRIVER_NAME " Can't request IRQ, Device not found\n");
return -ENXIO;
}
result = request_irq(irqnum, (void *)cascalc_interrupt, 0, DEVICE_NAME, NULL); // 註冊 ISR
if(result) {
printk(KERN_ERR DRIVER_NAME " Can't request IRQ %d (return %d)\n", irqnum, result);
return -EFAULT;
}
return 0;
}
static void __exit cascalc_exit(void) {
dev_t dev;
dev = MKDEV(cascalc_major, cascalc_minor);
unregister_chrdev_region( dev, 1);
free_irq(irqnum, NULL);
iounmap((unsigned long *) base);
printk("CASLab_Calc driver exit.\n");
}
module_init(cascalc_init);
module_exit(cascalc_exit);
```
:::
### Device Tree
把主要的 device 定義放在 `.dtsi`，`.dts` 則用來表示這塊板子要啟用哪些裝置，以及針對這些裝置提供實際的使用設定。因為 `arm-realview-eb.dts` 內有 include `arm-realview-eb.dtsi`，所以編譯出的 `.dtb` 會包含 `arm-realview-eb.dtsi` 的所有 device
:::spoiler `linux-4.14.85/arch/arm/boot/dts/arm-realview-eb.dtsi`
```dtsi=
// ...
// Add
cascalc: cascalc@80000000 { // 表示有一個 device，實體位址為 80000000
compatible = "caslab,calc"; // 參考 driver 中的註解
reg = <0x80000000 0x1000>; // 裝置的記憶體範圍
clocks = <&pclk>; // 裝置的 clock source 為 pclk (此檔案有定義)
clock-names = "apb_pclk"; // 指定此裝置每個 clock source 的名稱。因為一個裝置的 clock source 可以有多個，這樣可以在 driver 使用這些名字來 access clock source
};
};
```
:::
:::spoiler `linux-4.14.85/arch/arm/boot/dts/arm-realview-eb.dts`
```dts=
// ...
// Add
&cascalc{ // 指向 .dtsi 內定義的 cascalc
interrupt-parent = <&intc>; // 表示此裝置的 interrupt 來源為 intc (此檔案有定義為 interrupt controller)
interrupts = <0 30 IRQ_TYPE_LEVEL_HIGH>;
}; // 0: SPI (Shared Peripheral Interrupt)，為 ARM GIC interrupt 的一種類型，表示 GIC 可以分發此 interrupt 到任何一顆 CPU core
// 30: IRQ number
// IRQ_TYPE_LEVEL_HIGH: interrupt 訊號保持為高準位時視為觸發 interrupt 中，回到低準位時才表示結束 interrupt
```
:::
## User Program
使用 [Virtual Hardware](https://hackmd.io/p4MBgIlaRbm7LBsd5Zh1gw?both#Virtual-Hardware) 中定義好的 caslab_calc device，做所有 operation
:::spoiler `calc.c`
```c
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/ioctl.h>
#define DevPath "/dev/calculator" // 在此範例中會手動下 mknod /dev/calculator c 252 0 來在 /dev/ 建立一個 device node (作業系統筆記 p54)
// 和 device 定義 (caslab_calculator.c) 內相同 ///////////////
#define CALC_CTRL_GO 0x01
#define CALC_CTRL_ABORT 0x02
#define CALC_CTRL_CLRI 0x04
#define CALC_CTRL_RESET 0x08
#define CALC_OP_ADD 0x01
#define CALC_OP_SUB 0x02
#define CALC_OP_MUL 0x03
#define CALC_OP_DIV 0x04
#define CMD_STATUS 0x00
#define CMD_OP_SET 0x03
#define CMD_CTRL 0x04
////////////////////////////////////////////////////////////
static const unsigned int OP_LIST[4] = {
CALC_OP_ADD,
CALC_OP_SUB,
CALC_OP_MUL,
CALC_OP_DIV };
int main(int argc, char* argv[]) {
int i;
int buffer[2] = {0};
int result[4] = {0};
int calc_fd = 0;
if(argc < 3) {
printf("Usage: \n");
printf(" %s <operand1> <operand2>\n", argv[0]);
return 0;
} else {
// argv[1] 為 operand1、argv[2] 為 operand 2
buffer[0] = atoi(argv[1]);
buffer[1] = atoi(argv[2]);
}
calc_fd = open(DevPath, O_RDWR);
if(calc_fd == 0) {
perror("Error opening device\n");
return 1;
}
write(calc_fd, (void*)buffer, sizeof(unsigned int) * 2);
for(i = 0; i < 4; i++) {
ioctl(calc_fd, CMD_OP_SET, OP_LIST[i]);
ioctl(calc_fd, CMD_CTRL, CALC_CTRL_GO);
read(calc_fd, (void*)&(result[i]), sizeof(int));
}
printf("%d + %d = %d\n", buffer[0], buffer[1], result[0]);
printf("%d - %d = %d\n", buffer[0], buffer[1], result[1]);
printf("%d * %d = %d\n", buffer[0], buffer[1], result[2]);
printf("%d / %d = %d\n", buffer[0], buffer[1], result[3]);
close(calc_fd);
return 0;
}
```
:::
# QEMU-SystemC co-simulation
* 要做 QEMU-SystemC co-simulation 時才會用到
* 是把 QEMU-SystemC bridge 的 **QEMU 端** (`qemu-6.2.0-rc2/hw/char/caslab_sysbridge.c`) 視為一 virtual hardware，替代掉上面的 `caslab_calculator.h`、`caslab_calculator.c`；**SystemC 端** (`SystemC_Module/qsysbridge`) 使用另一 terminal 獨立執行
## SystemC 部分 (`SystemC_Module/`)
### calculator
:::spoiler `calculator.h`
```cpp=
#ifndef _H_CALCULATOR_
#define _H_CALCULATOR_
#include <systemc.h>
#include <ahb_slave_if.h>
#include <stdint.h>
#define CASLAB_CALC_RegCount 6
#define CAL_MEM_Region 0x1000
#define CALC_REG_CONTROL 0x00
#define CALC_REG_OPERATOR 0x01
#define CALC_REG_OPERAND1 0x02
#define CALC_REG_OPERAND2 0x03
#define CALC_REG_RESULT 0x04
#define CALC_REG_STATUS 0x05
#define CALC_CTRL_IDLE 0x00
#define CALC_CTRL_GO 0x01
#define CALC_CTRL_ABORT 0x02
#define CALC_CTRL_CLRI 0x04
#define CALC_STATUS_IDLE 0x00
#define CALC_STATUS_BUSY 0x01
#define CALC_OP_ADD 0x01
#define CALC_OP_SUB 0x02
#define CALC_OP_MUL 0x03
#define CALC_OP_DIV 0x04
class calculator:public ahb_slave_if, public sc_module
{
public:
sc_in_clk clk;
sc_out<bool> irq_out;
sc_in<bool> rst_in;
sc_uint<32> Reg[CASLAB_CALC_RegCount];
sc_uint<8> state;
SC_HAS_PROCESS(calculator);
calculator(sc_module_name name, uint32_t mapping_size);
~calculator(void);
void run();
bool read (uint32_t*, uint32_t, int);
bool write (uint32_t, uint32_t, int);
virtual bool local_access (bool write, uint32_t addr, uint32_t& data, unsigned int length);
private:
void caslab_calc_op(void);
const static unsigned char caslab_calc_id_arm[];
};
#endif
```
:::
:::spoiler `calculator.cpp`
```cpp=
#include "calculator.h"
using namespace std;
#ifdef CASLAB_CALC_DEBUG
#define DB_PRINT(...) \
do \
{ \
fprintf(stderr, "[CALC]: "); \
fprintf(stderr, ##__VA_ARGS__); \
} while (0);
#else
#define DB_PRINT(...)
#endif
const unsigned char calculator::caslab_calc_id_arm[] = {0x11, 0x10, 0x88, 0x08, 0x0d, 0xf0, 0x05, 0xb1};
calculator::calculator(sc_module_name name, uint32_t mapping_size) : ahb_slave_if(mapping_size), sc_module(name)
{
SC_THREAD(run);
sensitive << clk.pos();
reset_signal_is(rst_in, true);
}
calculator::~calculator(void)
{
}
void calculator::caslab_calc_op(void)
{
uint32_t i;
uint32_t op;
uint32_t src1, src2;
uint32_t result = 0;
op = Reg[CALC_REG_OPERATOR];
src1 = Reg[CALC_REG_OPERAND1];
src2 = Reg[CALC_REG_OPERAND2];
switch (op)
{
case CALC_OP_ADD:
result = src1 + src2;
wait();
break;
case CALC_OP_SUB:
result = src1 - src2;
wait();
break;
case CALC_OP_MUL:
result = src1 * src2;
for (i = 0; i < 4; i++)
wait();
break;
case CALC_OP_DIV:
if (src2 == 0)
fprintf(stderr, "Calculator: Invalid operand_2 !!\n");
else
{
result = src1 / src2;
for (i = 0; i < 13; i++)
wait();
}
break;
default:
break;
}
Reg[CALC_REG_RESULT] = result;
}
void calculator::run()
{
// Reset registers
for (unsigned i = 0; i < CASLAB_CALC_RegCount; i++)
Reg[i] = 0x00;
// Clear IRQ
irq_out = false;
state = 0; // 用來當作 state machine
while (1)
{
wait();
if (Reg[CALC_REG_STATUS] == CALC_STATUS_IDLE)
{
switch (Reg[CALC_REG_CONTROL])
{
case CALC_CTRL_GO:
irq_out = false;
state = 0;
Reg[CALC_REG_STATUS] = CALC_STATUS_BUSY;
break;
case CALC_CTRL_CLRI:
irq_out = false;
DB_PRINT("Reset IRQ.\n");
break;
default:
break;
}
}
else
{ // Run the state machine
if (Reg[CALC_REG_CONTROL] == CALC_CTRL_ABORT)
{
state = 0;
Reg[CALC_REG_STATUS] = CALC_STATUS_IDLE;
}
else
{
switch (state)
{
case 0:
caslab_calc_op();
state = state + 1;
break;
case 1:
irq_out = true;
Reg[CALC_REG_STATUS] = CALC_STATUS_IDLE;
state = 0;
DB_PRINT("Done, raising IRQ.\n");
break;
default:
state = 0;
break;
}
}
}
Reg[CALC_REG_CONTROL] = CALC_CTRL_IDLE;
}
}
bool calculator::read(uint32_t *data, uint32_t addr, int length)
{
bool success = true;
DB_PRINT("Reading data from address 0x%08x\n", addr);
if (addr < (CASLAB_CALC_RegCount << 2)) // addr 是要讀取 register
*data = Reg[addr >> 2];
else // addr 是要讀取裝置 ID
{
switch (addr >> 2)
{
case 0x3F8 ... 0x3FF:
*data = caslab_calc_id_arm[(addr >> 2) - 0x3F8];
break;
default:
*data = 0x00;
success = false;
}
}
return success;
}
bool calculator::write(uint32_t data, uint32_t addr, int length)
{
bool success = true;
DB_PRINT("Writting data [0x%08x] to address 0x%08x\n", data, addr);
if (addr < (CASLAB_CALC_RegCount << 2))
{
switch (addr >> 2)
{
case CALC_REG_OPERATOR:
case CALC_REG_OPERAND1:
case CALC_REG_OPERAND2:
case CALC_REG_CONTROL:
Reg[addr >> 2] = data;
break;
default: // Read only.
success = false;
break;
}
}
else
success = false;
return success;
}
bool calculator::local_access(bool write, uint32_t addr, uint32_t &data, unsigned int length)
{
bool success = true;
uint32_t local_address = addr & get_address_mask();
if (write)
success = this->write(data, local_address, length);
else
success = this->read(&data, local_address, length);
return success;
}
```
:::
### qemu_sc_bridge
即 QEMU-SystemC bridge SystemC 端
:::spoiler `qsbridge.h`
```cpp=
#ifndef _H_QSBRIDGE_
#define _H_QSBRIDGE_
#include <semaphore.h>
#include <systemc.h>
#include <ahb_master_if.h>
#include <stdint.h>
#define CAS_SYSBRG_SMID 0x8000
#define CAS_SYSBRG_IRQSEM "/CASSYS_IRQ"
#define CAS_SYSBRG_DATASEM "/CASSYS_DATA"
#define CAS_SYSBRG_ACKSEM "/CASSYS_ACK"
#define CAS_SYSBUS_CMD_Read 0x00
#define CAS_SYSBUS_CMD_Write 0x01
#define CAS_SYSBUS_CMD_Reset 0x02
#define CAS_SYSBUS_CMD_Exit 0x04
typedef struct
{
uint32_t Address;
uint32_t Data;
uint32_t ACK; // For bus access ack/nak
uint32_t CMD;
uint32_t ISR;
} CAS_SYSBUS_DATA;
class qsbridge : public sc_module, public ahb_master_if
{
public:
sc_in_clk clk;
sc_in<bool> irq_in; // 代收 calculator 的 interrupt
sc_out<bool> rst_out;
SC_HAS_PROCESS(qsbridge);
qsbridge(sc_module_name name);
~qsbridge();
void run();
protected:
bool bus_read(uint32_t *data, uint32_t addr, uint32_t length);
bool bus_read64(uint64_t *data, uint64_t addr, uint32_t length);
bool bus_write(uint32_t data, uint32_t addr, uint32_t length);
bool bus_write64(uint64_t data, uint64_t addr, uint32_t length);
private:
sem_t *SEM_IRQ;
sem_t *SEM_BUS; // 表示 QEMU 端 bridge 向 SystemC 端 bridge 發出各種指令 (CAS_SYSBUS_CMD_Read, CAS_SYSBUS_CMD_Write, CAS_SYSBUS_CMD_Reset, CAS_SYSBUS_CMD_Exit)
sem_t *SEM_BUS_Ack; // 表示:
// 1. SystemC 端 bridge 已準備好
// 2. SystemC 向 QEMU 表示命令處理完畢
// sem_t SEM_BUS_Lock;
int shm_id;
volatile CAS_SYSBUS_DATA *sh_mem;
bool connected;
};
#endif
```
:::
:::spoiler `qsbridge.cpp`
```cpp=
#include "qsbridge.h"
#include <signal.h>
#include <fcntl.h> /* For O_* constants */
#include <sys/ipc.h>
#include <sys/shm.h>
using namespace std;
#define CASLAB_SYSBRG_DEBUG
#ifdef CASLAB_SYSBRG_DEBUG
#define DB_PRINT(...) \
do \
{ \
fprintf(stderr, "[QSBRIDGE]: ");\
fprintf(stderr, ##__VA_ARGS__); \
} while(0);
#else
#define DB_PRINT(...)
#endif
qsbridge::qsbridge(sc_module_name name): sc_module(name)
{
int init_state = 3;
shm_id = -1;
connected = false;
do {
SEM_IRQ = sem_open(CAS_SYSBRG_IRQSEM, O_CREAT, S_IRUSR | S_IWUSR, 0);
SEM_BUS = sem_open(CAS_SYSBRG_DATASEM, O_CREAT, S_IRUSR | S_IWUSR, 0);
SEM_BUS_Ack = sem_open(CAS_SYSBRG_ACKSEM, O_CREAT, S_IRUSR | S_IWUSR, 0);
if(SEM_FAILED == SEM_IRQ || SEM_FAILED == SEM_BUS || SEM_FAILED == SEM_BUS_Ack) {
fprintf(stderr, "ERROR: semaphore failed\n");
break;
}
init_state--;
//Clear semaphore (non-blocking 把 semaphore 值減到 0 為止)
while(sem_trywait(SEM_IRQ) == 0);
while(sem_trywait(SEM_BUS_Ack) == 0);
shm_id = shmget(CAS_SYSBRG_SMID, sizeof(CAS_SYSBUS_DATA), 0666 | IPC_CREAT);
if(shm_id == -1){
fprintf(stderr, "ERROR: shmget failed\n");
break;
}
init_state--;
sh_mem = (volatile CAS_SYSBUS_DATA*)shmat(shm_id, NULL, 0);
if(sh_mem == (void*)-1) {
fprintf(stderr, "ERROR: shmat failed\n");
break;
}
memset((void*)sh_mem, 0, sizeof(CAS_SYSBUS_DATA));
init_state--;
} while(0);
if(init_state != 0) {
struct shmid_ds buf;
switch(init_state) {
case 1://shmat failed
sh_mem = NULL;
if(shm_id != -1 && shmctl(shm_id, IPC_STAT, &buf) == 0)
shmctl(shm_id, IPC_RMID, &buf);//Delete shared memory.
case 2://shmget failed
sem_close(SEM_IRQ);
while(sem_trywait(SEM_BUS) == 0);
sem_close(SEM_BUS);
sem_close(SEM_BUS_Ack);
case 3://sem_open failed
SEM_IRQ = NULL;
SEM_BUS = NULL;
SEM_BUS_Ack = NULL;
exit(-1);
break;
default://Should not happen
exit(0);
break;
}
}
SC_THREAD(run);
sensitive << clk.pos();
dont_initialize();
}
qsbridge::~qsbridge()
{
DB_PRINT("qsbridge deinit.\n");
if(SEM_BUS != NULL) {
while(sem_trywait(SEM_BUS) == 0);
sem_close(SEM_BUS);
}
if(SEM_BUS_Ack != NULL) {
while(sem_trywait(SEM_BUS_Ack) == 0);
sem_close(SEM_BUS_Ack);
}
if(SEM_IRQ != NULL) {
if(sh_mem != NULL && connected) {
DB_PRINT("Sending exit signal(0x%02x) to QEMU\n", CAS_SYSBUS_CMD_Exit);
sh_mem->CMD = CAS_SYSBUS_CMD_Exit;
sem_post(SEM_IRQ);
}
sem_close(SEM_IRQ);
}
if(sh_mem != NULL)
shmdt((void*) sh_mem);
if(shm_id != -1) {
struct shmid_ds buf;
if(shmctl(shm_id, IPC_STAT, &buf) == 0)
shmctl(shm_id, IPC_RMID, &buf);//Delete shared memory.
}
}
void qsbridge::run()
{
bool success = false;
bool _previous_irq = false;
printf("Waiting QEMU UP....\n");
sem_wait(SEM_BUS); // QEMU 準備好後在 caslab_sysbridge.c 的 Thread_UpdateIRQ() post SEM_BUS
sem_post(SEM_BUS_Ack);
printf("Connected to QEMU.\n");
connected = true;
while(connected) {
/* At first, we handle the IRQ */
if(_previous_irq != irq_in) {
if(irq_in)
sh_mem->ISR |= 0x01;
else
sh_mem->ISR &= (uint32_t)(-1) ^ 0x01; // 清除 LSb。(0xFFFFFFFF ^ 0x00000001 = 0xFFFFFFFE)
sem_post(SEM_IRQ);
_previous_irq = irq_in;
}
/* And then, we process the memory access */
if(sem_trywait(SEM_BUS) == 0) {//There's bus command (non-blocking 對 SEM_BUS 減 1，如果原本 semaphore 值 > 0 則會回傳 0)
uint32_t temp;
switch(sh_mem->CMD) {
case CAS_SYSBUS_CMD_Read:
DB_PRINT("BUS Read, read from 0x%08x\n", sh_mem->Address);
success = this->bus_read(&temp, sh_mem->Address, 4);
sh_mem->Data = temp;
break;
case CAS_SYSBUS_CMD_Write:
DB_PRINT("BUS Write, Write 0x%04x to 0x%08x\n", sh_mem->Data, sh_mem->Address);
success = this->bus_write(sh_mem->Data, sh_mem->Address, 4);
break;
case CAS_SYSBUS_CMD_Reset:
DB_PRINT("BUS Reset\n");
rst_out = true;
wait();//Wait 1 cycle
rst_out = false;
success = true;
break;
case CAS_SYSBUS_CMD_Exit:
DB_PRINT("Simulation end\n");
success = true;
connected = false;
break;
}
sh_mem->ACK = (success)? 0x01:0x00;
sem_post(SEM_BUS_Ack);
if(!connected) break;
}
wait();/* wait for next clock trigger */
}
raise(SIGINT);
}
/*=================== TLM Bus Access Function ====================*/
bool qsbridge::bus_read(uint32_t* data, uint32_t addr, uint32_t length)
{
bool success;
success = bus_b_access(false, (sc_dt::uint64)addr, reinterpret_cast<unsigned char*>(data), length);
if(!success) {
DB_PRINT("bus read 0x%X failed\n", addr);
}
return success;
}
bool qsbridge::bus_read64(uint64_t* data, uint64_t addr, uint32_t length)
{
bool success1;
bool success2;
uint32_t tmp;
success1 = bus_b_access(false, (sc_dt::uint64)addr, reinterpret_cast<unsigned char*>(&tmp), length);
*data = tmp;
success2 = bus_b_access(false, (sc_dt::uint64)addr + 4, reinterpret_cast<unsigned char*>(&tmp), length);
*data |= ((uint64_t)tmp) << 32;
if(success1 && success2)
return true;
else
{
DB_PRINT("bus read 64 0x%lX failed\n", addr);
return false;
}
}
bool qsbridge::bus_write(uint32_t data, uint32_t addr, uint32_t length)
{
bool success;
success = bus_b_access(true, (sc_dt::uint64)addr, reinterpret_cast<unsigned char*>(&data), length);
if(!success) {
DB_PRINT("bus write 0x%X:0x%X failed\n", addr, data);
}
return success;
}
bool qsbridge::bus_write64(uint64_t data, uint64_t addr, uint32_t length)
{
bool success1;
bool success2;
uint32_t tmp;
tmp = data;
success1 = bus_b_access(true, (sc_dt::uint64)addr, reinterpret_cast<unsigned char*>(&tmp), length);
tmp = data >> 32;
success2 = bus_b_access(true, (sc_dt::uint64)addr + 4, reinterpret_cast<unsigned char*>(&tmp), length);
if(success1 && success2)
return true;
else
{
DB_PRINT("bus write 64 0x%lX:0x%lX failed\n", addr, data);
return false;
}
}
```
:::
## QEMU 部分 (Virtual Hardware)
### 定義
即 QEMU-SystemC bridge QEMU 端
:::spoiler `qemu-6.2.0-rc2/hw/char/caslab_sysbridge.c`
```c=
#include "qemu/osdep.h"
#include "hw/sysbus.h"
#include "chardev/char-fe.h"
#include "qemu/log.h"
// added
#include "migration/vmstate.h"
#include "hw/qdev-properties-system.h"
#include "hw/qdev-core.h"
#include "hw/irq.h"
#include <fcntl.h> /* For O_* constants */
#include <sys/stat.h> /* For mode constants */
#include <sys/ipc.h>
#include <sys/shm.h>
#include <pthread.h>
#include <semaphore.h>
#define CASLAB_SYSBRG_DEBUG
#ifdef CASLAB_SYSBRG_DEBUG
#define DB_PRINT(...) \
do \
{ \
fprintf(stderr, "[DEBUG]: %s: ", __func__); \
fprintf(stderr, ##__VA_ARGS__); \
} while (0);
#else
#define DB_PRINT(...)
#endif
#define DeviceMemRange 0x1000
#define DeviceMemMask 0x0FFF
#define TYPE_CASLAB_SYSBRG "caslab_sysbridge"
#define CASLAB_SYSBRG(obj) OBJECT_CHECK(CASLabBridgeState, \
(obj), \
TYPE_CASLAB_SYSBRG)
#define CAS_SYSBRG_SMID 0x8000
#define CAS_SYSBRG_IRQSEM "/CASSYS_IRQ"
#define CAS_SYSBRG_DATASEM "/CASSYS_DATA"
#define CAS_SYSBRG_ACKSEM "/CASSYS_ACK"
#define CAS_SYSBUS_CMD_Read 0x00
#define CAS_SYSBUS_CMD_Write 0x01
#define CAS_SYSBUS_CMD_Reset 0x02
#define CAS_SYSBUS_CMD_Exit 0x04
typedef struct
{
uint32_t Address;
uint32_t Data;
uint32_t ACK; // For bus access ack/nak
uint32_t CMD;
uint32_t ISR;
} CAS_SYSBUS_DATA;
typedef struct
{
SysBusDevice parent_obj;
MemoryRegion iomem;
CharBackend chr;
qemu_irq irq;
const unsigned char *id;
sem_t *SEM_IRQ;
sem_t *SEM_BUS;
sem_t *SEM_BUS_Ack;
sem_t SEM_BUS_Key; // Unnamed semaphore，作為 shared memory 的 mutex lock。SystemC 部分因為不會造成 race 問題所以拿掉這個
bool _connected;
int shm_id;
volatile CAS_SYSBUS_DATA *sh_mem;
} CASLabBridgeState;
volatile bool PrgExit = false;
static pthread_t IRQ_Thread;
static CASLabBridgeState *_dev;
void *Thread_UpdateIRQ(void *);
static const unsigned char caslab_sysbridge_id_arm[8] =
{0x11, 0x10, 0x88, 0x08, 0x0d, 0xf0, 0x05, 0xb1};
static void caslab_sysbridge_deinit(CASLabBridgeState *s)
{
struct shmid_ds buf;
struct timespec wait_time = {0}; // 等待 exit 的最長時間
DB_PRINT("SYSBUS DEINIT\n");
clock_gettime(CLOCK_REALTIME, &wait_time); // CLOCK_REALRIME 表示 wall-clock time，將 wall-clock time 存進 wait_time
wait_time.tv_sec += 3; // wait 3s before force exit.
if (s->_connected)
{
s->_connected = false;
DB_PRINT("Disonnceting from HW.\n");
DB_PRINT("Invoking HW Bus Signal.\n")
s->sh_mem->CMD = CAS_SYSBUS_CMD_Exit;
while (sem_trywait(s->SEM_BUS_Ack) == 0)
;
sem_post(s->SEM_BUS);
sem_timedwait(s->SEM_BUS_Ack, &wait_time);
// reset sems.
while (sem_trywait(s->SEM_BUS) == 0)
;
while (sem_trywait(s->SEM_BUS_Ack) == 0)
;
while (sem_trywait(s->SEM_IRQ) == 0)
;
sem_close(s->SEM_BUS);
sem_close(s->SEM_BUS_Ack);
sem_close(s->SEM_IRQ);
}
shmdt((void *)s->sh_mem);
sem_destroy(&s->SEM_BUS_Key);
if (shmctl(s->shm_id, IPC_STAT, &buf) == 0)
shmctl(s->shm_id, IPC_RMID, &buf); // Delete shared memory.
}
/*
* 程式正常 exit/return 時會執行的 function
*/
static void ON_Module_Exit(void)
{
PrgExit = true;
if (_dev != NULL)
{
sem_post(_dev->SEM_IRQ); // 因為 Thread_UpdateIRQ() 內有 sem_wait(s->SEM_IRQ)，可能 thread 會在這邊 wait 導致 thread 無法正常退出，故新增此行
// pthread_join(IRQ_Thread, NULL);
pthread_cancel(IRQ_Thread);
caslab_sysbridge_deinit(_dev);
}
}
/*
* thread 主程式，用來作為 IRQ listener
*/
void *Thread_UpdateIRQ(void *mem)
{
CASLabBridgeState *s;
s = (CASLabBridgeState *)mem;
while (!PrgExit)
{
if (s->_connected)
{
uint32_t ISR;
DB_PRINT("Waiting Device IRQ Signal.\n");
sem_wait(s->SEM_IRQ);
if (PrgExit)
break;
if (s->sh_mem->CMD == CAS_SYSBUS_CMD_Exit)
{
sem_wait(&s->SEM_BUS_Key);
s->sh_mem->CMD = CAS_SYSBUS_CMD_Read;
sem_post(&s->SEM_BUS_Key);
s->_connected = false;
DB_PRINT("Device Disconnected.\n");
while (sem_trywait(s->SEM_BUS) == 0)
;
while (sem_trywait(s->SEM_BUS_Ack) == 0)
;
while (sem_trywait(s->SEM_IRQ) == 0)
;
continue;
}
ISR = s->sh_mem->ISR;
DB_PRINT("Received IRQ Signal from HW.(0x%02x)\n", ISR);
qemu_mutex_lock_iothread(); // 由於以下會使用到 internal I/O usage (ex: qemu_irq_raise())，任何非 QEMU main thread 的 thread 要用到 internal I/O 時必須用此 macro 保護
if (ISR & 0x01)
qemu_irq_raise(s->irq);
else
qemu_irq_lower(s->irq);
qemu_mutex_unlock_iothread(); // Done, now unlock it.
}
else
{
DB_PRINT("Waiting Device up...\n");
sem_post(s->SEM_BUS); // ┐
sem_wait(s->SEM_BUS_Ack); // ┘參考 qsbridge.cpp 內註解
s->_connected = true;
DB_PRINT("Device connected.\n");
if (PrgExit)
break;
}
}
DB_PRINT("IRQ Listener exited.\n");
return NULL;
}
static const VMStateDescription vmstate_caslab_sysbridge = {
.name = TYPE_CASLAB_SYSBRG,
.version_id = 1,
.minimum_version_id = 1,
.fields = (VMStateField[]){
// nothing needed to be saved here.
VMSTATE_END_OF_LIST(),
}};
static uint64_t caslab_sysbridge_read(void *opaque, hwaddr offset, unsigned size)
{
CASLabBridgeState *s = opaque;
uint64_t ret = 0;
hwaddr temp;
switch (offset & DeviceMemMask)
{
case 0xFE0 ... 0xFFC: // 以最末 32 bytes 作為裝置 ID
temp = (offset & DeviceMemMask) - 0xFE0;
ret = caslab_sysbridge_id_arm[temp >> 2];
break;
default:
if (s->_connected && offset < DeviceMemRange)
{
sem_wait(&s->SEM_BUS_Key); // Acquire the key to access BUS
s->sh_mem->CMD = CAS_SYSBUS_CMD_Read;
s->sh_mem->Address = offset;
sem_post(s->SEM_BUS);
sem_wait(s->SEM_BUS_Ack);
ret = s->sh_mem->Data;
sem_post(&s->SEM_BUS_Key); // Release the key
}
break;
}
return ret;
}
static void caslab_sysbridge_write(void *opaque, hwaddr offset, uint64_t value, unsigned size)
{
CASLabBridgeState *s = opaque;
if (offset >= DeviceMemRange)
{
fprintf(stderr, "Device ERROR: Overflow!!!\n");
return;
}
if (!s->_connected)
return;
sem_wait(&s->SEM_BUS_Key); // Acquire the key to access BUS
s->sh_mem->CMD = CAS_SYSBUS_CMD_Write;
s->sh_mem->Address = offset;
s->sh_mem->Data = (uint32_t)value;
sem_post(s->SEM_BUS);
sem_wait(s->SEM_BUS_Ack);
sem_post(&s->SEM_BUS_Key); // Release the key
}
static void caslab_sysbridge_reset(DeviceState *dev)
{
CASLabBridgeState *s = CASLAB_SYSBRG(dev);
if (!s->_connected)
return;
sem_wait(&s->SEM_BUS_Key); // Acquire the key to access BUS
s->sh_mem->CMD = CAS_SYSBUS_CMD_Reset;
sem_post(s->SEM_BUS);
sem_wait(s->SEM_BUS_Ack);
sem_post(&s->SEM_BUS_Key); // Release the key
}
static void caslab_sysbridge_realize(DeviceState *dev, Error **errp)
{
// Nothing to do here currently.
}
static const MemoryRegionOps caslab_sysbridge_ops = {
.read = caslab_sysbridge_read,
.write = caslab_sysbridge_write,
.endianness = DEVICE_NATIVE_ENDIAN,
.valid.min_access_size = 4,
.valid.max_access_size = 4,
};
static void caslab_sysbridge_init(Object *obj)
{
SysBusDevice *sbd = SYS_BUS_DEVICE(obj);
CASLabBridgeState *s = CASLAB_SYSBRG(obj);
int init_state = 5;
struct shmid_ds buf;
s->_connected = false;
s->shm_id = -1;
memory_region_init_io(&s->iomem, OBJECT(s), &caslab_sysbridge_ops, s,
TYPE_CASLAB_SYSBRG, DeviceMemRange);
sysbus_init_mmio(sbd, &s->iomem);
sysbus_init_irq(sbd, &s->irq);
do
{
if (0 != sem_init(&s->SEM_BUS_Key, 0, 1)) // Assign only 1 key to act like mutex
break;
init_state--;
// 和 qsbridge.cpp 共用 semaphore
s->SEM_BUS = sem_open(CAS_SYSBRG_DATASEM, O_CREAT, S_IRUSR | S_IWUSR, 0);
s->SEM_BUS_Ack = sem_open(CAS_SYSBRG_ACKSEM, O_CREAT, S_IRUSR | S_IWUSR, 0);
s->SEM_IRQ = sem_open(CAS_SYSBRG_IRQSEM, O_CREAT, S_IRUSR | S_IWUSR, 0);
if (SEM_FAILED == s->SEM_BUS || SEM_FAILED == s->SEM_BUS_Ack || SEM_FAILED == s->SEM_IRQ)
break;
init_state--;
// Reset sems
while (sem_trywait(s->SEM_BUS) == 0)
;
while (sem_trywait(s->SEM_BUS_Ack) == 0)
;
while (sem_trywait(s->SEM_IRQ) == 0)
;
s->shm_id = shmget(CAS_SYSBRG_SMID, sizeof(CAS_SYSBUS_DATA), 0666 | IPC_CREAT);
if (s->shm_id == -1)
break;
init_state--;
s->sh_mem = shmat(s->shm_id, NULL, 0);
if (s->sh_mem == (void *)-1)
break;
init_state--;
memset((void *)s->sh_mem, 0, sizeof(CAS_SYSBUS_DATA));
if (0 != pthread_create(&IRQ_Thread, NULL, Thread_UpdateIRQ, (void *)s))
break;
init_state--;
} while (0);
switch (init_state)
{
case 0:
DB_PRINT("SYSBUS INIT\n");
break;
case 1: // pthread_create failed
fprintf(stderr, "ERROR: pthread failed\n");
shmdt((void *)s->sh_mem);
s->sh_mem = NULL;
case 2: // shmat failed
fprintf(stderr, "ERROR: shmat failed\n");
if (shmctl(s->shm_id, IPC_STAT, &buf) == 0)
shmctl(s->shm_id, IPC_RMID, &buf);
case 3: // shmget failed
fprintf(stderr, "ERROR: shmget failed\n");
while (sem_trywait(s->SEM_BUS) == 0)
;
while (sem_trywait(s->SEM_BUS_Ack) == 0)
;
while (sem_trywait(s->SEM_IRQ) == 0)
;
sem_close(s->SEM_BUS);
sem_close(s->SEM_BUS_Ack);
sem_close(s->SEM_IRQ);
case 4: // shared semaphore failed
fprintf(stderr, "ERROR: semaphore failed\n");
sem_destroy(&s->SEM_BUS_Key);
case 5: // anonymous semaphore failed
fprintf(stderr, "ERROR: semaphore failed\n");
default:
exit(-1);
break;
}
_dev = s;
if (0 != atexit(ON_Module_Exit)) // 註冊正常 exit/return 時要執行的 function
fprintf(stderr, "failed to register on exit function, may cause memory violation when exit.\n");
}
static Property caslab_sysbridge_properties[] = {
DEFINE_PROP_CHR("chardev", CASLabBridgeState, chr),
DEFINE_PROP_END_OF_LIST(),
};
static void caslab_sysbridge_class_init(ObjectClass *oc, void *data)
{
DeviceClass *dc = DEVICE_CLASS(oc);
dc->realize = caslab_sysbridge_realize;
dc->vmsd = &vmstate_caslab_sysbridge;
dc->reset = caslab_sysbridge_reset;
dc->props_ = caslab_sysbridge_properties;
}
static const TypeInfo caslab_sysbridge_info = {
.name = TYPE_CASLAB_SYSBRG,
.parent = TYPE_SYS_BUS_DEVICE,
.instance_size = sizeof(CASLabBridgeState),
.instance_init = caslab_sysbridge_init,
.class_init = caslab_sysbridge_class_init};
static void caslab_sysbridge_register_types(void)
{
type_register_static(&caslab_sysbridge_info);
}
type_init(caslab_sysbridge_register_types)
```
:::
### 設定
[calculator 設定](https://hackmd.io/p4MBgIlaRbm7LBsd5Zh1gw?both#%E8%A8%AD%E5%AE%9A) 的 `qemu-6.2.0-rc2/hw/char/meson.build` 內 `caslab_calculator.c` 改為 `caslab_sysbridge.c`、`qemu-6.2.0-rc2/hw/arm/realview.c` 內 `caslab_calc` 改為 `caslab_sysbridge`
### Kernel Module
和 [calculator Kernel Module](https://hackmd.io/p4MBgIlaRbm7LBsd5Zh1gw?both#Kernel-Module) 相同
### Device Tree
和 [calculator Device Tree](https://hackmd.io/p4MBgIlaRbm7LBsd5Zh1gw?both#Device-Tree) 相同
Sign in with Wallet
Connect another wallet