---
# System prepended metadata

title: NCKU QEMU Lab 筆記
tags: [QEMU, SystemC]

---

> 此處為 [學長筆記](https://hackmd.io/@xswzaq44321/ryyFgMffyx) 內所有 command 及程式的解釋

> [改好的 source code](https://github.com/weber-lin-nthu/NCKU_Computer-Architecture_Qemu-SystemC-Co-Simulation)

# QEMU

## QEMU (`qemu-6.2.0-rc2/`)

```sh
# 在解壓縮 QEMU 後的 qemu-6.2.0-rc2，新增 build 資料夾後進去
../configure --target-list="arm-softmmu,arm-linux-user" --prefix=${HOME}/workspace/qemu-bin-6.2.0-rc2
```

* `target-list`: 只編譯指定的模擬器 target
    所有 target 可在 `configs/targets` 看到
    * `arm-softmmu`: 針對整個平台 (包含 CPU 與周邊硬體) 做模擬 (如 realview versatile family)，執行檔為 `qemu-bin-6.2.0-rc2/qemu-system-arm`
        realview versatile family 為 ARM 的一系列硬體開發平台
    * `arm-linux-user`: 只做 CPU 指令集架構的轉換模擬，執行檔為 `qemu-bin-6.2.0-rc2/qemu-arm`
* `prefix`: 設定安裝在此資料夾

```sh
make -j $(nproc)
make install
```

* `make -j $(nproc)`: 編譯 QEMU 原始碼，產生 `qemu-system-arm` 跟 `qemu-arm` 執行檔等檔案。 `-j $(nproc)` 為根據 CPU core 數量做平行處理加快編譯速度
* `make install`: 把那些編譯好的檔案複製到剛剛 configure 設定的 `prefix` 資料夾

## Cross Compile Toolchain (`arm-2014.05/`)

* `arm-2014.05/bin/` 內會放 cross compile toolchain 的執行檔。例如 `arm-2014.05/bin/arm-none-linux-gnueabi-gcc`:
    * `arm`: target 架構為 arm
    * `none`: 表示沒有特定廠商限制，為通用
    * `linux`: target OS 為 Linux
    * `gnueabi`
        * EABI (Embedded Application Binary Interface) 為 ARM 定義的 ABI 標準，包含:
            * 函式怎麼呼叫 (calling convention)
            * 資料怎麼排版（結構對齊、stack 對齊）
            * 參數怎麼傳遞（register 還是 stack）
            * 哪些暫存器由 caller/callee 負責保存
            * 浮點參數如何處理
        * GNU EABI 即為 GNU 針對 EABI 的一個實作

## Linux Kernel (`linux-4.14.85/`)

* 原本需先下 `make ARCH=arm menuconfig` 來以互動式介面修改 `linux-4.14.85/.config` 檔來設定哪些功能、驅動程式、架構支援會被編譯進 kernel，此 Lab 直接使用 RealView EB 最簡單的設定檔複製到資料夾內
    RealView EB (Emulation Baseboard) 為 ARM 官方的硬體開發板，用來模擬尚未量產的 ARM SoC
    * `linux-4.14.85/.config` 範例
        * `CONFIG_ARM=y`: 表示為 ARM 架構編譯的 kernel
        * `CONFIG_MIGHT_HAVE_PCI=y`: kernel 可能支援 PCI 裝置
        * `CONFIG_STACKTRACE_SUPPORT=y`: 啟用 stack 追蹤功能 (用來 debug crash)
        * `CONFIG_PGTABLE_LEVELS=2`: 使用 2 階段 page table (ARMv5 架構常見)

```sh
make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- zImage
```

* cross compile linux kernel 給 ARM 架構使用，產生可開機的 kernel 壓縮映像檔 (zImage)
    * `ARCH=arm`: target 架構為 arm
    * `CROSS_COMPILE=arm-none-linux-gnueabi-`: 使用的 cross compile toolchain 的 prefix
    * `zImage`: 要求產出一個壓縮過的 Linux kernel 映像檔，路徑為 `arch/arm/boot/zImage`

```sh
make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi- dtbs
```

* cross compile linux kernel 中的 Device Tree Blob 檔案 (dtb) (硬體描述檔)
    Device Tree 為一種文字化的硬體描述檔 (`.dts` (device tree source)、`.dtsi` (device tree source include，類似 C++ 的 header) 檔)，告訴 Linux:
    1. 有哪些硬體裝置 (CPU、RAM、UART、Ethernet ...)
    2. 它們的記憶體位置在哪裡
    3. 有哪些 driver 要綁定到哪些裝置
    
    編譯後會變成二進位 `.dtb` 檔（Device Tree Blob），在開機時由 bootloader (或 QEMU) 傳給 kernel

    * `dtbs`: 要求 kernel 編譯 dtb 檔案。會編譯出多個 dtb 檔，路徑為 `arch/arm/boot/dts/`，我們只會用到 `arch/arm/boot/dts/arm-realview-eb.dtb`

## initrd ramdisk (initrd) 及 BusyBox (`busybox-1.29.3/`)

* initrd ramdisk (initrd) 為在啟動階段被 Linux kernel 呼叫的臨時檔案系統，用於 root file system (RFS) 被掛載之前的準備工作
* Linux 首先要將 kernel 載入到主記憶體。initrd 通常被壓縮成 gzip，開機時由 bootloader (如 LILO、GRUB) 來告知 kernel initrd 的位置，使其被 kernel 存取，掛載成一個 loop device
* loop device (`/dev/loop*`) 為 Linux 的一種虛擬裝置，允許把一個普通的檔案當成一個 block device 來掛載
* [BusyBox](https://busybox.net/) 為將許多小版本的 UNIX 工具 (ex: `ls`, `mount`) 包裝成一個執行檔，非常適合拿來組一個 initrd
    * 同 [Linux kernel](https://hackmd.io/p4MBgIlaRbm7LBsd5Zh1gw?stext=1279%3A24%3A0%3A1749741786%3AjMkH5u&both=) 一樣需先下 `make ARCH=arm menuconfig` 修改設定來產生 .config，這裡也一樣直接複製提供好的設定檔
        * 有一個 config `CONFIG_LINUXRC=y` 表示要編譯出 linuxrc 檔
            linuxrc 為早期 linux kernel 掛載 initrd 之後第一個執行的程式 (比 `/sbin/init` 還早，其中一個原因為因為 `/sbin/init` 在 root file system 內，還沒被掛載)，現在的 linux kernel 不需要了
            
```sh
make ARCH=arm CROSS_COMPILE=arm-none-linux-gnueabi-
make install CROSS_COMPILE=arm-none-linux-gnueabi-
```

* 同 [Linux kernel](https://hackmd.io/p4MBgIlaRbm7LBsd5Zh1gw?both=&stext=1757%3A65%3A0%3A1749743167%3AafNm-V) 的參數，只是 BusyBox `make install` 預設的位置是 `_install/`
* 安裝完之後，`_install/` 內再自己加入 `dev/`, `lib/` (內放 [cross compile toolchain](https://hackmd.io/p4MBgIlaRbm7LBsd5Zh1gw?both=#Cross-Compile-Toolchain-arm-201405) 的函式庫 (`.so`) 檔 (在 `arm-2014.05/arm-none-linux-gnueabi/libc/lib/`)), `opt/`, `proc/`, `sys/`, `tmp/`, `var/`, `home/` 及提供的 `etc/` 資料夾，即成為一個 initrd。最後使用 `cpio` 指令將所有檔案打包成一個 cpio archive，並用 `gzip` 壓縮成 `.gz`
    * `etc/init.d/` 資料夾為早期 System V 系統在開機時用來啟動服務的所有 script。現代 Linux 皆已改用 systemd，但還是有這個資料夾，是為了向下相容

## 執行 QEMU

```sh
~/workspace/qemu-bin-6.2.0-rc2/bin/qemu-system-arm -M realview-eb -m 128M -cpu arm1136 -kernel zImage -initrd ./initrd.gz -nographic -serial mon:stdio -dtb arm-realview-eb.dtb
```

* `-M`: 指定要模擬的開發版
* `-m`: 指定模擬機的記憶體大小
* `-cpu`: 指定使用的 CPU 型號
* `-kernel`: kernel 映像檔路徑
* `-initrd`: initrd 路徑。**之後模擬機開機後的 root file system 即為 initrd**，因為在嵌入式環境通常不用再掛載外部磁碟，故也不需要真正的 root file system
* `-nographic`: 使用純文字模式
* `-serial mon:stdio`: 讓 QEMU 的 monitor (mon, 控制介面) 和模擬機的 serial port **共用 host 的 terminal 的 stdio**。這樣可以在同一個 terminal 透過 **切換 (按 ctrl-a c)** 來對模擬機的 serial port 做 I/O 或對 QEMU 的 monitor 做 I/O
    QEMU 的 monitor 為用來控制和觀察模擬機狀態的介面，屬於 QEMU
    模擬機的 serial port 是因為模擬機沒有顯示器，所以跟 Linux 互動可以透過 serial port
* `-dtb`: Device Tree Blob 路徑

## Virtual Hardware

此範例為定義一個 calculator 硬體計算單元 (character device)，透過 memory-mapped I/O (MMIO) 操作，並可支援 QEMU monitor 及 VMState 機制

### 定義

放在 `qemu-6.2.0-rc2/hw/char/`，表示為一個 character device

:::spoiler `caslab_calculator.h`
```cpp=
#ifndef CASLAB_CALCULATOR_H
#define CASLAB_CALCULATOR_H
#include "hw/sysbus.h"
#include "chardev/char-fe.h"
#include "hw/qdev-core.h"
#include "qapi/error.h"

#define TYPE_CASLAB_CALC "caslab_calc"
#define CASLAB_CALC(obj) OBJECT_CHECK(CASLabCalcState, \
								(obj), \
								TYPE_CASLAB_CALC);  // OBJECT_CHECK() 為 QEMU 提供的 macro，為將 pointer 轉型並檢查型別是否正確
#define CASLAB_CALC_RegCount	6

/*
 * 儲存此裝置的 state
 */
typedef struct {
    SysBusDevice parent_obj;  // 使用 C 模擬繼承關係，繼承自 SysBusDevice，表示此裝置是一個在 SysBus 上的裝置
    MemoryRegion iomem;  // 定義此裝置的 MMIO address 及大小
    uint32_t Reg[6];         // 定義此裝置有 6 個 register (分別為 control, operator, operand1, operand2, result, status)
    CharBackend chr;         // 為 character device 與 host 通訊的管道
    qemu_irq irq;            // IRQ 線
    const unsigned char *id; // 裝置 ID
} CASLabCalcState;

/*
 * 創建裝置
 */
static inline DeviceState* caslab_calc_create(hwaddr addr,
											qemu_irq irq,
											Chardev *chr)
{
	DeviceState *dev;
	SysBusDevice *s;
	dev = qdev_new(TYPE_CASLAB_CALC);  // 建立名為 caslab_calc 的 QEMU 裝置
	s = SYS_BUS_DEVICE(dev);  // 將裝置轉型為 SysBusDevice parent class
	qdev_prop_set_chr(dev, "chardev", chr);  // 設定 QEMU property "chardev"。在 .c 檔有以 DEFINE_PROP_CHR 定義了 chardev 的 property。QEMU property 為裝置初始化時由 QEMU 傳給模擬裝置的設定值
	qdev_realize_and_unref(dev, NULL, &error_fatal);  // 正式實體化 (realize) 裝置
	sysbus_mmio_map(s, 0, addr);  // 把裝置的第 0 個 MMIO 區域 map 到系統 bus 的 addr 位址
	sysbus_connect_irq(s, 0, irq);  // 把裝置的第 0 條 IRQ 線連接到 CPU 的 irq interrupt port
	return dev;
}

#endif
```
:::

:::spoiler `caslab_calculator.c`
```c=
#include "qemu/osdep.h"
#include "hw/sysbus.h"
#include "qemu/log.h"
// added
#include "migration/vmstate.h"
#include "hw/qdev-properties-system.h"
#include "hw/qdev-core.h"
#include "hw/irq.h"

#include "caslab_calculator.h"

//#define CASLAB_CALC_DEBUG

#ifdef CASLAB_CALC_DEBUG
#define DB_PRINT(...)						\
	do										\
	{										\
		fprintf(stderr, "[DEBUG]: %s: ", __func__);\
		fprintf(stderr, ##__VA_ARGS__);		\
	} while(0);
#else
#define DB_PRINT(...)
#endif

#define	CALC_REG_CONTROL	0x00
#define	CALC_REG_OPERATOR	0x01
#define	CALC_REG_OPERAND1	0x02
#define	CALC_REG_OPERAND2	0x03
#define	CALC_REG_RESULT		0x04
#define	CALC_REG_STATUS		0x05

/*
 * 控制暫存器 (CALC_REG_CONTROL) 的各種操作指令
 */
#define CALC_CTRL_GO		0x01  // 開始運算 (呼叫 caslab_calc_op())
#define CALC_CTRL_ABORT		0x02  // 終止進行中的運算 (如果狀態是 CALC_STATUS_BUSY，會立刻變為 CALC_STATUS_IDLE 並發出 interrupt)
#define CALC_CTRL_CLRI		0x04  // 清除中斷 (clear IRQ, 代表 guest 程式已處理完結果)

#define CALC_STATUS_IDLE	0x00
#define CALC_STATUS_BUSY	0x01

#define CALC_OP_ADD			0x01
#define CALC_OP_SUB			0x02
#define CALC_OP_MUL			0x03
#define CALC_OP_DIV			0x04

static void caslab_calc_reset(DeviceState *dev); //forward decleration

static void caslab_calc_op(CASLabCalcState *s)
{
	uint32_t op;
	uint32_t src1, src2;
	uint32_t result = 0;
	if(s == NULL)
		return;
	s->Reg[CALC_REG_STATUS] = CALC_STATUS_BUSY;

	op		= s->Reg[CALC_REG_OPERATOR];
	src1	= s->Reg[CALC_REG_OPERAND1];
	src2	= s->Reg[CALC_REG_OPERAND2];

	DB_PRINT("Doing OP %u with value %u, %u\n", op, src1, src2);
	switch(op) {
	case CALC_OP_ADD:
		result = src1 + src2;
		break;
	case CALC_OP_SUB:
		result = src1 - src2;
		break;
	case CALC_OP_MUL:
		result = src1 * src2;
		break;
	case CALC_OP_DIV:
		if(src2 == 0)
			fprintf(stderr, "Calculator: Invalid operand_2 !!\n");
		else
			result = src1 / src2;
		break;
	default:
		result = 0;
		break;
	}
	s->Reg[CALC_REG_STATUS] = CALC_STATUS_IDLE;
	s->Reg[CALC_REG_RESULT] = result;
	qemu_irq_raise(s->irq);  // 觸發 IRQ，同等於 qemu_set_irq(s->irq, 1);
}

/*
 * 此裝置的 ID
 */
static const unsigned char caslab_calc_id_arm[8] = 
	{0x11, 0x10, 0x88, 0x08, 0x0d, 0xf0, 0x05, 0xb1};

/*
 * 根據讀取的位址 (offset) 判斷要回傳裝置 ID 還是 register 值
 */
static uint64_t caslab_calc_read(void *opaque, hwaddr offset, unsigned size)
{
	CASLabCalcState *s = opaque;
	uint64_t ret = 0;
	offset >>= 2;  // 因為 register 是 32 bit，所以要把以 32 bit 為單位的位址 (0x0, 0x4, 0x8, ...) 轉為 array 的 index (0, 1, 2, ...) 時就要 shift 2 bit
	switch(offset) {
		case 0x3F8:  // 0x3F8 是一般 legacy UART (COM1) 的位址，這邊不是拿來模擬 UART，而是把此區段當成裝置 ID 區
			ret = caslab_calc_id_arm[0];
			break;
		case 0x3F9:
			ret = caslab_calc_id_arm[1];
			break;
		case 0x3FA:
			ret = caslab_calc_id_arm[2];
			break;
		case 0x3FB:
			ret = caslab_calc_id_arm[3];
			break;
		case 0x3FC:
			ret = caslab_calc_id_arm[4];
			break;
		case 0x3FD:
			ret = caslab_calc_id_arm[5];
			break;
		case 0x3FE:
			ret = caslab_calc_id_arm[6];
			break;
		case 0x3FF:
			ret = caslab_calc_id_arm[7];
			break;

		case CALC_REG_OPERAND1:
		case CALC_REG_OPERAND2:
		case CALC_REG_RESULT:
		case CALC_REG_STATUS:
			ret = s->Reg[offset];
			break;

		default:
			break;
	};
	DB_PRINT("Read value %lu @ 0x%02lx\n", ret, offset<<2);
	return ret;
}

static void caslab_calc_write(void *opaque, hwaddr offset, uint64_t value, unsigned size)
{
	CASLabCalcState *s = opaque;
	DB_PRINT("Write value %lu @ 0x%02lx\n", value, offset);
	offset >>= 2;
	if(offset >= CASLAB_CALC_RegCount)
		return;
	else
	{
		switch(offset)
		{
			case CALC_REG_CONTROL:
				if(CALC_STATUS_IDLE == s->Reg[CALC_REG_STATUS]) {
					if( value == CALC_CTRL_GO ) {
						DB_PRINT("CASlab Calculator commit\n");
						caslab_calc_op(s);
					} else if ( value == CALC_CTRL_CLRI ) {
						qemu_set_irq(s->irq, 0);
					}
				} else if(CALC_STATUS_BUSY == s->Reg[CALC_REG_STATUS]) {
					if ( value == CALC_CTRL_ABORT ) {
						//Cancel operations
						s->Reg[CALC_REG_STATUS] = CALC_STATUS_IDLE;
						qemu_set_irq(s->irq, 1);
					}
				}
				break;
			case CALC_REG_OPERATOR://Writable register
			case CALC_REG_OPERAND1:
			case CALC_REG_OPERAND2:
				if(s->Reg[CALC_REG_STATUS] == CALC_STATUS_IDLE)
					s->Reg[offset] = value;
				break;
			default: //Read only register
				break;
		}
	}
}

/*
 * 在裝置被 instantiate 時會被呼叫
 */
static void caslab_calc_realize(DeviceState *dev, Error **errp)
{
	//Nothing here curreltly
}

static void caslab_calc_reset(DeviceState *dev)
{
	CASLabCalcState *s = CASLAB_CALC(dev);
	memset(s->Reg, 0, CASLAB_CALC_RegCount * sizeof(*(s->Reg)));
	qemu_set_irq(s->irq, 0);
}

/*
 * 表示這個模擬裝置的 MMIO 存取操作規則 (當有 guest 程式對此裝置的記憶體做讀寫時，QEMU 就會參考這個來決定怎麼處理讀寫行為)
 */
static const MemoryRegionOps caslab_calc_ops = {
	.read		= caslab_calc_read,
	.write		= caslab_calc_write,
	.endianness	= DEVICE_NATIVE_ENDIAN,
	.valid.min_access_size = 4,  // 允許 guest 程式存取的最小資料長度
	.valid.max_access_size = 4  // 允許 guest 程式存取的最大資料長度
};

static void caslab_calc_init(Object *obj)
{
    // 型別轉換
	SysBusDevice *sbd = SYS_BUS_DEVICE(obj);
	CASLabCalcState *s = CASLAB_CALC(obj);

	memory_region_init_io(&s->iomem, OBJECT(s), &caslab_calc_ops, s,
						TYPE_CASLAB_CALC, 0x1000);  // 初始化一塊 MMIO 區塊，大小為 4 KB
	sysbus_init_mmio(sbd, &s->iomem);  // 把此 MMIO 區塊掛載到 SysBus，讓 bus 可以存取到此塊位址
	sysbus_init_irq(sbd, &s->irq);  // 初始化 IRQ 腳位，註冊一條中斷線給此裝置
	DB_PRINT("INIT\n");
}

/*
 * 此部分為 optional，為 QEMU 定義此裝置的快照狀態描述 (VM migration state)
 */
static const VMStateDescription vmstate_caslab_calc = {
	.name 				= TYPE_CASLAB_CALC,
	.version_id			= 1,
	.minimum_version_id	= 1,
	.fields	= (VMStateField[]) {  // 需要保存的欄位
		VMSTATE_UINT32_ARRAY(Reg, CASLabCalcState,
							CASLAB_CALC_RegCount),  // 儲存 CASLabCalcState 中的 Reg 這個 uint32 array，array 長度為 CASLAB_CALC_RegCount
		VMSTATE_END_OF_LIST()
	}
};

/*
 * 定義一個 character device property，名稱為 "chardev"
 * 
 * 常見的使用方式為:
 * 1. 在 QEMU 啟動時先使用 -chardev 參數來建立一個 character device
 *     > ex: 
 *     > -chardev socket,id=mysock,path=/tmp/mysock,server,nowait
 * 2. 再使用 -device 參數將該 character device 指定給 caslab_calc 裝置
 *     > ex:
 *     > -device caslab_calc,chardev=mysock
 *     > 此設定會將 id 為 mysock 的 character device 綁定到 caslab_calc 裝置的 "chardev" property，進而對應到 CasLabCalcState.chr 欄位 (因為此段程式碼有設定好了)。這樣之後 caslab_calc 裝置就可以透過 chr 欄位來做字元 I/O ，實現與外部的通訊功能。但此 Lab 沒有用到這個
 */
static Property caslab_calc_properties[] = {
	DEFINE_PROP_CHR("chardev", CASLabCalcState, chr),
	DEFINE_PROP_END_OF_LIST(),
};

/*
 * 「裝置類別 (device class)」的初始化
 * (caslab_calc_init() 是針對 instance device 做初始化，這個是針對 class 做初始化)
 */
static void caslab_calc_class_init(ObjectClass *oc, void *data)
{
    // 型別轉換
	DeviceClass *dc = DEVICE_CLASS(oc);
	
    // 註冊
	dc->realize = caslab_calc_realize;
	dc->vmsd 	= &vmstate_caslab_calc;
	dc->reset	= caslab_calc_reset;
	dc->props_	= caslab_calc_properties;
}

/*
 * 定義一個新的「裝置型別 (device type)」
 * device type 為針對 device class 的描述
 */
static const TypeInfo caslab_calc_info = {
	.name			= TYPE_CASLAB_CALC,
	.parent			= TYPE_SYS_BUS_DEVICE,  // 定義裝置是掛在 SysBus 上
	.instance_size	= sizeof(CASLabCalcState),  // instance 大小
	.instance_init	= caslab_calc_init,  // 註冊
	.class_init		= caslab_calc_class_init,  // 註冊
};

static void caslab_calc_register_types(void)
{
	type_register_static(&caslab_calc_info);
}

type_init(caslab_calc_register_types)
```
:::

### 設定

#### `qemu-6.2.0-rc2/hw/char/meson.build`

```meson
# ...

softmmu_ss.add(files('caslab_calculator.c'))  # Add

# ...
```

表示使用 [softmmu](https://hackmd.io/p4MBgIlaRbm7LBsd5Zh1gw?both=&stext=270%3A7%3A0%3A1749910537%3ARXND8E) 模擬器時，ss (source set) 新增需要編譯此檔案

#### `qemu-6.2.0-rc2/hw/arm/realview.c`

```c
// ...

    for (n = 0; n < 64; n++) {
        pic[n] = qdev_get_gpio_in(dev, n);
    }
    sysbus_create_simple("caslab_calc", 0x80000000, pic[30]);  // Add

// ...
```

建立一個簡單的 System Bus 裝置 "caslab_calc"，此裝置的實體位址為 0x80000000，並設定 PIC (在 ARM 稱為 GIC) 第 30 號 interrupt 線連接至此裝置 (則 IRQ number 為 30)

#### `linux-4.14.85/.config`

使用 `make ARCH=arm menuconfig` 修改 config "Enable loadable module support" 為開啟

### Kernel Module

此 kernel module 的功能為 calculator 的 driver

:::spoiler `calc_drv.c` (編譯成 `.ko` 來使用)
```c=
#include <linux/init.h>
#include <linux/module.h>
#include <linux/types.h>
#include <linux/fs.h>
#include <linux/cdev.h>
#include <linux/ioctl.h>
#include <linux/wait.h>
#include <linux/mm.h>
#include <linux/uaccess.h>
#include <linux/vmalloc.h>
#include <linux/of.h>
#include <linux/of_irq.h>
#include <asm/atomic.h>
#include <asm/io.h>
#include <asm/page.h>

#include <linux/device.h>
#include <asm/irq.h>
#include <linux/sched.h>
#include <linux/interrupt.h>
#include <linux/irqflags.h>

// 和 device 定義 (caslab_calculator.c) 內相同 ///////////////
#define CALC_REG_CONTROL	0x00
#define CALC_REG_OPERATOR	0x04
#define CALC_REG_OPERAND1	0x08
#define CALC_REG_OPERAND2	0x0C
#define CALC_REG_RESULT		0x10
#define CALC_REG_STATUS		0x14

#define CALC_STATUS_IDLE	0x00

#define CALC_CTRL_GO		0x01
#define CALC_CTRL_ABORT		0x02
#define CALC_CTRL_CLRI		0x04
#define CALC_CTRL_RESET		0x08

#define CALC_OP_ADD			0x01
#define CALC_OP_SUB			0x02
#define CALC_OP_MUL			0x03
#define CALC_OP_DIV			0x04
////////////////////////////////////////////////////////////

// Some CMD number is invaild(used by linux kernel)
// For detail read linux kernel document (Documentation/ioctl/ioctl-number.txt)
// and http://man7.org/linux/man-pages/man2/ioctl_list.2.html
#define CMD_STATUS			0x00
#define CMD_OP_SET			0x03
#define CMD_CTRL			0x04

#define BUFFER_SIZE			20

#define DRIVER_NAME			"caslab_calc_hw"
#define DEVICE_NAME			"caslab-calc"
#define DEVICE_COMP			"caslab,calc"  // device 的 compatible property 名稱，定義在 linux-4.14.85/arch/arm/boot/dts/arm-realview-eb.dtsi。每個 device node 都有一個 compatible property，用來描述該裝置的類型、廠商、相容性等資訊
// Can be found within linux kernel Documentation/admin-guide/devices.txt
// 252 is LOCAL/EXPERIMENTAL USE
#define CASCALC_MAJOR		252
#define CASCALC_MINOR		0
#define CASCALC_BASE		0x80000000
#define CASCALC_MEMREGION	0x1000
#define CASCALC_IRQ			30

MODULE_AUTHOR("DUN-JIE CHEN");
MODULE_LICENSE("GPL");

static DECLARE_WAIT_QUEUE_HEAD(wq);  // DECLARE_WAIT_QUEUE_HEAD 為 linux 提供的 macro，定義 wq 為一個 driver 內的 waiting queue 的 head，用來放執行此 driver 的 user process，等待 interrupt 發生再繼續
static int flag = 0;  // 0: 裝置尚未完成運算或 interrupt 尚未到來，process 還不能醒
                      // 1: 裝置已完成運算，interrupt 已發生，process 可被喚醒

static unsigned int irqnum		= CASCALC_IRQ;

static int cascalc_major	= CASCALC_MAJOR;
static int cascalc_minor	= CASCALC_MINOR;

typedef struct {
	unsigned int buffer[BUFFER_SIZE];
} DevicePrivate;  // 每個使用此裝置的 process 都有一個此空間
static void __iomem *base;  // 為 CASCALC_BASE (physical address) 的 virtual address

/*
 * 透過 dev_compatible_name 字串在 device tree 中找到該節點，並取出它的 IRQ number
 */
static int get_dut_irq(const char* dev_compatible_name)
{
	struct device_node* dev_node;  // 作業系統筆記 p54
	int irq = -1;
    
	dev_node = of_find_compatible_node(NULL, NULL, dev_compatible_name);  // 透過 compatible 名稱找出對應的 device node
	if(!dev_node)
		return -1;
    
	irq = irq_of_parse_and_map(dev_node, 0);
    
	of_node_put(dev_node);  //  把剛剛 reference 的 device node 結構釋放 (降低 reference count)
	
    return irq;
}

static DevicePrivate* allocate_memory_region(void)
{
	DevicePrivate* _prv;
	_prv = (DevicePrivate*)vmalloc(sizeof(DevicePrivate));
	return _prv;

}

static void free_allocated_memory(DevicePrivate *_prv)
{
	vfree(_prv);
	_prv = NULL;
}

static int calc_open(struct inode *inode, struct file *file)
{
	file->private_data = allocate_memory_region();
	if(file->private_data == NULL)
		return -ENOMEM;
	return 0;
}

static int calc_release(struct inode *inode, struct file *file)
{
	free_allocated_memory(file->private_data);
	return 0;
}

static ssize_t calc_read(struct file *filp, char *buff, size_t len, loff_t *off)
{
	int i;
	DevicePrivate *prv_data = (DevicePrivate*)filp->private_data;

	/* Only result is readable memory address */
	if(len > 4)
		len = 4;

	for( i = 0; i < len/4; i++ )
		prv_data->buffer[i] = readl(base + CALC_REG_RESULT + i * 4);  // 從 MMIO 區域讀取 4 byte (l)

	i = copy_to_user(buff, prv_data->buffer, len);  // copy_to_user() 為 Linux 提供的 API，把 kernel space 的 buffer (prv_data->buffer) 複製到 user space 的 buffer (buff)

	return len;

}

static ssize_t calc_write(struct file *filp, const char *buff, size_t len, loff_t *off)
{
	int i;
	DevicePrivate *prv_data = (DevicePrivate*)filp->private_data;

	/* Two operands are writeable memory address */
	if ( len > 8 )
		len = 8;
	
	i = copy_from_user(prv_data->buffer, buff, len);
	for(i = 0; i < len/4; i++)
		writel(prv_data->buffer[i], base + CALC_REG_OPERAND1 + i*4);

	return len;
}

/*
 * 較新版 kernel 從 ioctl 改成 unlocked_ioctl 了 (user program 呼叫還是用 ioctl())
 * 可以把 ioctl 理解成一般操作 (read/write/open/...) 外的雜項
 */
static long calc_unlocked_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
{
	long ret = 0;
	unsigned int tmp;
	int i;
	printk("IOCTL CMD = %u, value = %lu\n", cmd, arg);
	switch(cmd)
	{
		case CMD_STATUS:  // 查看 status
			tmp = readl(base + CALC_REG_STATUS);
			i = copy_to_user((void*)arg, &tmp, sizeof(unsigned int));
		case CMD_OP_SET:  // 設定 operator
			if(arg >= CALC_OP_ADD && arg <= CALC_OP_DIV)
				writel(arg, base + CALC_REG_OPERATOR);
			else
				printk("Invaild OP %lu\n", arg);
			break;
		case CMD_CTRL:  // 控制硬體行為
			tmp = readl(base + CALC_REG_STATUS);
			if(tmp == CALC_STATUS_IDLE) {  // 裝置處於 idle，啟動計算
				writel(arg, base + CALC_REG_CONTROL);  // 寫入 control register 之後，根據 caslab_calculator.c 的定義，會讓 hardware 開始計算
				wait_event_interruptible(wq, flag != 0);  // 將呼叫此 ioctl 的 user process 放入 waiting queue wq，並等待 hardware 計算，hardware 計算完之後會發出 IRQ，CPU 被 interrupt，執行 cascalc_interrupt()，把 flag 設為 1、並執行 wake_up_interruptible() 把 user process 從 wq 移出。此時 (flag != 0) 為 true，就可以繼續往下執行了
				flag = 0;
			} else {
				if(arg == CALC_CTRL_ABORT || arg == CALC_CTRL_RESET) {
					writel(arg, base + CALC_REG_CONTROL);
					flag = 0;
				}
			}
			break;
		default:
			printk("Invaild CMD %u\n", cmd);
			ret = -ENOTTY;
			break;
	}
	return ret;
}

/*
 * struct cdev 為 linux kernel 中管理 character device 的核心資料結構，在 driver 初始化時 (cascalc_init) 與 file_operations 一起註冊，使裝置能透過 /dev/ 介面被使用
 */
static struct cdev cascalc_cdev = {
	.kobj	= { .name = DRIVER_NAME },  // kernel object (kobject)，目前僅供內部使用或 debug
	.owner	= THIS_MODULE  // 指定此裝置由目前這個 kernel module 擁有，可防止使用中被卸載
};

static struct file_operations cascalc_fops ={
	.owner				= THIS_MODULE,
	.read				= calc_read,
	.write				= calc_write,
	.unlocked_ioctl		= calc_unlocked_ioctl,
	.open				= calc_open,
	.release			= calc_release
};

/*
 * 即 ISR，由 linux kernel 執行
 */
static irqreturn_t cascalc_interrupt(int irq, void *dev_id, struct pt_regs *regs)
{
    // 參考 calc_unlocked_ioctl 的註解
    
	if (irq == irqnum) {
		printk("received interrupt from %s\n", DRIVER_NAME);
		writel(CALC_CTRL_CLRI, base + CALC_REG_CONTROL);
		printk("Release interrupt \n");
		flag = 1;
		wake_up_interruptible(&wq);
	}
	return IRQ_HANDLED;
}

/*
 * 執行 insmod 時會呼叫此函數做初始化
 */
static int __init cascalc_init(void) {
	dev_t dev = 0;
	int result;
    
	if(CASCALC_MAJOR) {  // 有指定 major number
		dev = MKDEV(cascalc_major, cascalc_minor);  // linux kernel 提供的 macro，把 major number 跟 minor number 組合成 dev_t type
		result = register_chrdev_region( dev, 1, DRIVER_NAME);  // 註冊裝置號，註冊 1 個裝置
	} else {  // 沒有指定 major number
		result = alloc_chrdev_region( &dev, cascalc_minor, 1, DRIVER_NAME);  // 動態分配一個 major number (存入 dev)，註冊裝置號，註冊 1 個裝置
		cascalc_major = MAJOR(dev);
	}
	if(result < 0) {
		printk(KERN_ERR DRIVER_NAME "Can't allocate major number %d for %s\n", cascalc_major, DEVICE_NAME);
		return -EAGAIN;
	}
    
	cdev_init( &cascalc_cdev, &cascalc_fops);  // 註冊 struct cdev 及 operations
    
	result = cdev_add( &cascalc_cdev, dev, 1);  // 加入 linux kernel，正式成為可用的 character device
	if(result < 0) {
		printk(KERN_ERR DRIVER_NAME "Can't add device %d\n", dev);
		return -EAGAIN;
	}

	/* allocate memory */
	base = ioremap_nocache(CASCALC_BASE , CASCALC_MEMREGION);  // 把 physical address CASCALC_BASE map 成 virtual address，才能讓 kernel 做存取
	if(base == NULL)
		return -ENXIO;

	irqnum = get_dut_irq(DEVICE_COMP);
	if(irqnum == -1) {
		printk(KERN_ERR DRIVER_NAME " Can't request IRQ, Device not found\n");
		return -ENXIO;
	}
	
	result = request_irq(irqnum, (void *)cascalc_interrupt, 0, DEVICE_NAME, NULL);  // 註冊 ISR
	if(result) {
		printk(KERN_ERR DRIVER_NAME " Can't request IRQ %d (return %d)\n", irqnum, result);
		return -EFAULT;
	}

	return 0;
}

static void __exit cascalc_exit(void) {
	dev_t dev;
	dev = MKDEV(cascalc_major, cascalc_minor);
	unregister_chrdev_region( dev, 1);
	free_irq(irqnum, NULL);
	iounmap((unsigned long *) base);
	printk("CASLab_Calc driver exit.\n");
}

module_init(cascalc_init);
module_exit(cascalc_exit);
```
:::

### Device Tree

把主要的 device 定義放在 `.dtsi`，`.dts` 則用來表示這塊板子要啟用哪些裝置，以及針對這些裝置提供實際的使用設定。因為 `arm-realview-eb.dts` 內有 include `arm-realview-eb.dtsi`，所以編譯出的 `.dtb` 會包含 `arm-realview-eb.dtsi` 的所有 device

:::spoiler `linux-4.14.85/arch/arm/boot/dts/arm-realview-eb.dtsi`
```dtsi=
    // ...
    
    // Add
    cascalc: cascalc@80000000 {  // 表示有一個 device，實體位址為 80000000
        compatible = "caslab,calc";  // 參考 driver 中的註解
        reg = <0x80000000 0x1000>;  // 裝置的記憶體範圍
        clocks = <&pclk>;  //  裝置的 clock source 為 pclk (此檔案有定義)
        clock-names = "apb_pclk";  // 指定此裝置每個 clock source 的名稱。因為一個裝置的 clock source 可以有多個，這樣可以在 driver 使用這些名字來 access clock source
    };
};
```
:::

:::spoiler `linux-4.14.85/arch/arm/boot/dts/arm-realview-eb.dts`
```dts=
// ...

// Add
&cascalc{  // 指向 .dtsi 內定義的 cascalc
    interrupt-parent = <&intc>;  // 表示此裝置的 interrupt 來源為 intc (此檔案有定義為 interrupt controller)
    interrupts = <0 30 IRQ_TYPE_LEVEL_HIGH>;
};  // 0: SPI (Shared Peripheral Interrupt)，為 ARM GIC interrupt 的一種類型，表示 GIC 可以分發此 interrupt 到任何一顆 CPU core
    // 30: IRQ number
    // IRQ_TYPE_LEVEL_HIGH: interrupt 訊號保持為高準位時視為觸發 interrupt 中，回到低準位時才表示結束 interrupt
```
:::

## User Program

使用 [Virtual Hardware](https://hackmd.io/p4MBgIlaRbm7LBsd5Zh1gw?both#Virtual-Hardware) 中定義好的 caslab_calc device，做所有 operation

:::spoiler `calc.c`
```c
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/ioctl.h>

#define DevPath "/dev/calculator"  // 在此範例中會手動下 mknod /dev/calculator c 252 0 來在 /dev/ 建立一個 device node (作業系統筆記 p54)

// 和 device 定義 (caslab_calculator.c) 內相同 ///////////////
#define CALC_CTRL_GO		0x01
#define CALC_CTRL_ABORT		0x02
#define CALC_CTRL_CLRI		0x04
#define CALC_CTRL_RESET		0x08

#define CALC_OP_ADD			0x01
#define CALC_OP_SUB			0x02
#define CALC_OP_MUL			0x03
#define CALC_OP_DIV			0x04

#define CMD_STATUS			0x00
#define CMD_OP_SET			0x03
#define CMD_CTRL			0x04
////////////////////////////////////////////////////////////

static const unsigned int OP_LIST[4] = {
				CALC_OP_ADD,
				CALC_OP_SUB,
				CALC_OP_MUL,
				CALC_OP_DIV };

int main(int argc, char* argv[]) {
	int i;
	int buffer[2] = {0};
	int result[4] = {0};
	int calc_fd = 0;

	if(argc < 3) {
		printf("Usage: \n");
		printf("  %s <operand1> <operand2>\n", argv[0]);
		return 0;
	} else {
        // argv[1] 為 operand1、argv[2] 為 operand 2
		buffer[0] = atoi(argv[1]);
		buffer[1] = atoi(argv[2]);
	}
    
	calc_fd = open(DevPath, O_RDWR);
	if(calc_fd == 0) {
		perror("Error opening device\n");
		return 1;
	}
    
	write(calc_fd, (void*)buffer, sizeof(unsigned int) * 2);
	
    for(i = 0; i < 4; i++) {
		ioctl(calc_fd, CMD_OP_SET, OP_LIST[i]);
		ioctl(calc_fd, CMD_CTRL, CALC_CTRL_GO);
		read(calc_fd, (void*)&(result[i]), sizeof(int));
	}

	printf("%d + %d = %d\n", buffer[0], buffer[1], result[0]);
	printf("%d - %d = %d\n", buffer[0], buffer[1], result[1]);
	printf("%d * %d = %d\n", buffer[0], buffer[1], result[2]);
	printf("%d / %d = %d\n", buffer[0], buffer[1], result[3]);

	close(calc_fd);

	return 0;
}
```
:::

# QEMU-SystemC co-simulation

* 要做 QEMU-SystemC co-simulation 時才會用到
* 是把 QEMU-SystemC bridge 的 **QEMU 端** (`qemu-6.2.0-rc2/hw/char/caslab_sysbridge.c`) 視為一 virtual hardware，替代掉上面的 `caslab_calculator.h`、`caslab_calculator.c`；**SystemC 端** (`SystemC_Module/qsysbridge`) 使用另一 terminal 獨立執行

## SystemC 部分 (`SystemC_Module/`)

### calculator

:::spoiler `calculator.h`
```cpp=
#ifndef _H_CALCULATOR_
#define _H_CALCULATOR_
#include <systemc.h>
#include <ahb_slave_if.h>
#include <stdint.h>

#define CASLAB_CALC_RegCount	6
#define CAL_MEM_Region			0x1000

#define	CALC_REG_CONTROL	0x00
#define	CALC_REG_OPERATOR	0x01
#define	CALC_REG_OPERAND1	0x02
#define	CALC_REG_OPERAND2	0x03
#define	CALC_REG_RESULT		0x04
#define	CALC_REG_STATUS		0x05

#define CALC_CTRL_IDLE		0x00
#define CALC_CTRL_GO		0x01
#define CALC_CTRL_ABORT		0x02
#define CALC_CTRL_CLRI		0x04

#define CALC_STATUS_IDLE	0x00
#define CALC_STATUS_BUSY	0x01

#define CALC_OP_ADD			0x01
#define CALC_OP_SUB			0x02
#define CALC_OP_MUL			0x03
#define CALC_OP_DIV			0x04


class calculator:public ahb_slave_if, public sc_module
{
	public:
		sc_in_clk clk;
		sc_out<bool> irq_out;
		sc_in<bool> rst_in;

		sc_uint<32> Reg[CASLAB_CALC_RegCount];
		sc_uint<8> state;

		SC_HAS_PROCESS(calculator);
		calculator(sc_module_name name, uint32_t mapping_size);
		~calculator(void);

		void run();
		bool read (uint32_t*, uint32_t, int);
		bool write (uint32_t, uint32_t, int);
		virtual bool local_access (bool write, uint32_t addr, uint32_t& data, unsigned int length);

	private:
		void caslab_calc_op(void);
		const static unsigned char caslab_calc_id_arm[];
};
#endif

```
:::

:::spoiler `calculator.cpp`
```cpp=
#include "calculator.h"

using namespace std;

#ifdef CASLAB_CALC_DEBUG
#define DB_PRINT(...)                   \
    do                                  \
    {                                   \
        fprintf(stderr, "[CALC]: ");    \
        fprintf(stderr, ##__VA_ARGS__); \
    } while (0);
#else
#define DB_PRINT(...)
#endif

const unsigned char calculator::caslab_calc_id_arm[] = {0x11, 0x10, 0x88, 0x08, 0x0d, 0xf0, 0x05, 0xb1};

calculator::calculator(sc_module_name name, uint32_t mapping_size) : ahb_slave_if(mapping_size), sc_module(name)
{
    SC_THREAD(run);
    sensitive << clk.pos();
    reset_signal_is(rst_in, true);
}

calculator::~calculator(void)
{
}

void calculator::caslab_calc_op(void)
{
    uint32_t i;
    uint32_t op;
    uint32_t src1, src2;
    uint32_t result = 0;

    op = Reg[CALC_REG_OPERATOR];
    src1 = Reg[CALC_REG_OPERAND1];
    src2 = Reg[CALC_REG_OPERAND2];

    switch (op)
    {
    case CALC_OP_ADD:
        result = src1 + src2;
        wait();
        break;
    case CALC_OP_SUB:
        result = src1 - src2;
        wait();
        break;
    case CALC_OP_MUL:
        result = src1 * src2;
        for (i = 0; i < 4; i++)
            wait();
        break;
    case CALC_OP_DIV:
        if (src2 == 0)
            fprintf(stderr, "Calculator: Invalid operand_2 !!\n");
        else
        {
            result = src1 / src2;
            for (i = 0; i < 13; i++)
                wait();
        }
        break;
    default:
        break;
    }
    Reg[CALC_REG_RESULT] = result;
}

void calculator::run()
{
    // Reset registers
    for (unsigned i = 0; i < CASLAB_CALC_RegCount; i++)
        Reg[i] = 0x00;
    // Clear IRQ
    irq_out = false;
    state = 0; // 用來當作 state machine

    while (1)
    {
        wait();
        if (Reg[CALC_REG_STATUS] == CALC_STATUS_IDLE)
        {
            switch (Reg[CALC_REG_CONTROL])
            {
            case CALC_CTRL_GO:
                irq_out = false;
                state = 0;
                Reg[CALC_REG_STATUS] = CALC_STATUS_BUSY;
                break;
            case CALC_CTRL_CLRI:
                irq_out = false;
                DB_PRINT("Reset IRQ.\n");
                break;
            default:
                break;
            }
        }
        else
        { // Run the state machine
            if (Reg[CALC_REG_CONTROL] == CALC_CTRL_ABORT)
            {
                state = 0;
                Reg[CALC_REG_STATUS] = CALC_STATUS_IDLE;
            }
            else
            {
                switch (state)
                {
                case 0:
                    caslab_calc_op();
                    state = state + 1;
                    break;
                case 1:
                    irq_out = true;
                    Reg[CALC_REG_STATUS] = CALC_STATUS_IDLE;
                    state = 0;
                    DB_PRINT("Done, raising IRQ.\n");
                    break;
                default:
                    state = 0;
                    break;
                }
            }
        }
        Reg[CALC_REG_CONTROL] = CALC_CTRL_IDLE;
    }
}

bool calculator::read(uint32_t *data, uint32_t addr, int length)
{
    bool success = true;
    DB_PRINT("Reading data from address 0x%08x\n", addr);
    if (addr < (CASLAB_CALC_RegCount << 2)) // addr 是要讀取 register
        *data = Reg[addr >> 2];
    else // addr 是要讀取裝置 ID
    {
        switch (addr >> 2)
        {
        case 0x3F8 ... 0x3FF:
            *data = caslab_calc_id_arm[(addr >> 2) - 0x3F8];
            break;
        default:
            *data = 0x00;
            success = false;
        }
    }
    return success;
}

bool calculator::write(uint32_t data, uint32_t addr, int length)
{
    bool success = true;
    DB_PRINT("Writting data [0x%08x] to address 0x%08x\n", data, addr);
    if (addr < (CASLAB_CALC_RegCount << 2))
    {
        switch (addr >> 2)
        {
        case CALC_REG_OPERATOR:
        case CALC_REG_OPERAND1:
        case CALC_REG_OPERAND2:
        case CALC_REG_CONTROL:
            Reg[addr >> 2] = data;
            break;
        default: // Read only.
            success = false;
            break;
        }
    }
    else
        success = false;

    return success;
}

bool calculator::local_access(bool write, uint32_t addr, uint32_t &data, unsigned int length)
{
    bool success = true;
    uint32_t local_address = addr & get_address_mask();

    if (write)
        success = this->write(data, local_address, length);
    else
        success = this->read(&data, local_address, length);

    return success;
}

```
:::

### qemu_sc_bridge

即 QEMU-SystemC bridge SystemC 端

:::spoiler `qsbridge.h`
```cpp=
#ifndef _H_QSBRIDGE_
#define _H_QSBRIDGE_
#include <semaphore.h>
#include <systemc.h>
#include <ahb_master_if.h>
#include <stdint.h>

#define CAS_SYSBRG_SMID 0x8000
#define CAS_SYSBRG_IRQSEM "/CASSYS_IRQ"
#define CAS_SYSBRG_DATASEM "/CASSYS_DATA"
#define CAS_SYSBRG_ACKSEM "/CASSYS_ACK"

#define CAS_SYSBUS_CMD_Read 0x00
#define CAS_SYSBUS_CMD_Write 0x01
#define CAS_SYSBUS_CMD_Reset 0x02
#define CAS_SYSBUS_CMD_Exit 0x04

typedef struct
{
    uint32_t Address;
    uint32_t Data;
    uint32_t ACK; // For bus access ack/nak
    uint32_t CMD;
    uint32_t ISR;
} CAS_SYSBUS_DATA;

class qsbridge : public sc_module, public ahb_master_if
{
public:
    sc_in_clk clk;
    sc_in<bool> irq_in; // 代收 calculator 的 interrupt
    sc_out<bool> rst_out;

    SC_HAS_PROCESS(qsbridge);
    qsbridge(sc_module_name name);
    ~qsbridge();

    void run();

protected:
    bool bus_read(uint32_t *data, uint32_t addr, uint32_t length);
    bool bus_read64(uint64_t *data, uint64_t addr, uint32_t length);
    bool bus_write(uint32_t data, uint32_t addr, uint32_t length);
    bool bus_write64(uint64_t data, uint64_t addr, uint32_t length);

private:
    sem_t *SEM_IRQ;
    sem_t *SEM_BUS;     // 表示 QEMU 端 bridge 向 SystemC 端 bridge 發出各種指令 (CAS_SYSBUS_CMD_Read, CAS_SYSBUS_CMD_Write, CAS_SYSBUS_CMD_Reset, CAS_SYSBUS_CMD_Exit)
    sem_t *SEM_BUS_Ack; // 表示:
                        // 1. SystemC 端 bridge 已準備好
                        // 2. SystemC 向 QEMU 表示命令處理完畢
    // sem_t SEM_BUS_Lock;
    int shm_id;
    volatile CAS_SYSBUS_DATA *sh_mem;
    bool connected;
};

#endif

```
:::

:::spoiler `qsbridge.cpp`
```cpp=
#include "qsbridge.h"
#include <signal.h>
#include <fcntl.h>		/* For O_* constants */
#include <sys/ipc.h>
#include <sys/shm.h>

using namespace std;

#define CASLAB_SYSBRG_DEBUG

#ifdef CASLAB_SYSBRG_DEBUG
#define DB_PRINT(...)					\
	do									\
	{									\
		fprintf(stderr, "[QSBRIDGE]: ");\
		fprintf(stderr, ##__VA_ARGS__);	\
	} while(0);
#else
#define DB_PRINT(...)
#endif


qsbridge::qsbridge(sc_module_name name): sc_module(name)
{
	int init_state = 3;
	shm_id = -1;
	connected = false;
	do {
		SEM_IRQ = sem_open(CAS_SYSBRG_IRQSEM, O_CREAT, S_IRUSR | S_IWUSR, 0);
		SEM_BUS = sem_open(CAS_SYSBRG_DATASEM, O_CREAT, S_IRUSR | S_IWUSR, 0);
		SEM_BUS_Ack = sem_open(CAS_SYSBRG_ACKSEM, O_CREAT, S_IRUSR | S_IWUSR, 0);
		if(SEM_FAILED == SEM_IRQ || SEM_FAILED == SEM_BUS || SEM_FAILED == SEM_BUS_Ack) {
			fprintf(stderr, "ERROR: semaphore failed\n");
			break;
		}
		init_state--;
		//Clear semaphore (non-blocking 把 semaphore 值減到 0 為止)
		while(sem_trywait(SEM_IRQ) == 0);
		while(sem_trywait(SEM_BUS_Ack) == 0);

		shm_id = shmget(CAS_SYSBRG_SMID, sizeof(CAS_SYSBUS_DATA), 0666 | IPC_CREAT);
		if(shm_id == -1){
			fprintf(stderr, "ERROR: shmget failed\n");
			break;
		}
		init_state--;

		sh_mem = (volatile CAS_SYSBUS_DATA*)shmat(shm_id, NULL, 0);
		if(sh_mem == (void*)-1) {
			fprintf(stderr, "ERROR: shmat failed\n");
			break;
		}
		memset((void*)sh_mem, 0, sizeof(CAS_SYSBUS_DATA));
		init_state--;

	} while(0);

	if(init_state != 0) {
		struct shmid_ds buf;
		switch(init_state) {
		case 1://shmat failed
			sh_mem = NULL;
			if(shm_id != -1 && shmctl(shm_id, IPC_STAT, &buf) == 0)
				shmctl(shm_id, IPC_RMID, &buf);//Delete shared memory.
		case 2://shmget failed
			sem_close(SEM_IRQ);
			while(sem_trywait(SEM_BUS) == 0);
			sem_close(SEM_BUS);
			sem_close(SEM_BUS_Ack);
		case 3://sem_open failed
			SEM_IRQ = NULL;
			SEM_BUS = NULL;
			SEM_BUS_Ack = NULL;
			exit(-1);
			break;
		default://Should not happen
			exit(0);
			break;
		}
	}

	SC_THREAD(run);
	sensitive << clk.pos();
	dont_initialize();
}

qsbridge::~qsbridge()
{
	DB_PRINT("qsbridge deinit.\n");
	if(SEM_BUS != NULL) {
		while(sem_trywait(SEM_BUS) == 0);
		sem_close(SEM_BUS);
	}
	if(SEM_BUS_Ack != NULL) {
		while(sem_trywait(SEM_BUS_Ack) == 0);
		sem_close(SEM_BUS_Ack);
	}
	if(SEM_IRQ != NULL) {
		if(sh_mem != NULL && connected) {
			DB_PRINT("Sending exit signal(0x%02x) to QEMU\n", CAS_SYSBUS_CMD_Exit);
			sh_mem->CMD = CAS_SYSBUS_CMD_Exit;
			sem_post(SEM_IRQ);
		}
		sem_close(SEM_IRQ);
	}
	if(sh_mem != NULL)
		shmdt((void*) sh_mem);
	if(shm_id != -1) {
		struct shmid_ds buf;
		if(shmctl(shm_id, IPC_STAT, &buf) == 0)
			shmctl(shm_id, IPC_RMID, &buf);//Delete shared memory.
	}
	
}

void qsbridge::run()
{
	bool success = false;
	bool _previous_irq = false;
	printf("Waiting QEMU UP....\n");
	sem_wait(SEM_BUS);  // QEMU 準備好後在 caslab_sysbridge.c 的 Thread_UpdateIRQ() post SEM_BUS
	sem_post(SEM_BUS_Ack);
	printf("Connected to QEMU.\n");
	connected = true;

	while(connected) {
		/* At first, we handle the IRQ */
		if(_previous_irq != irq_in) {
			if(irq_in)
				sh_mem->ISR |= 0x01;
			else
				sh_mem->ISR &= (uint32_t)(-1) ^ 0x01;  // 清除 LSb。(0xFFFFFFFF ^ 0x00000001 = 0xFFFFFFFE)
			sem_post(SEM_IRQ);
			_previous_irq = irq_in;
		}

		/* And then, we process the memory access */
		if(sem_trywait(SEM_BUS) == 0) {//There's bus command (non-blocking 對 SEM_BUS 減 1，如果原本 semaphore 值 > 0 則會回傳 0)
			uint32_t temp;
			switch(sh_mem->CMD) {
			case CAS_SYSBUS_CMD_Read:
				DB_PRINT("BUS Read, read from 0x%08x\n", sh_mem->Address);
				success = this->bus_read(&temp, sh_mem->Address, 4);
				sh_mem->Data = temp;
				break;
			case CAS_SYSBUS_CMD_Write:
				DB_PRINT("BUS Write, Write 0x%04x to 0x%08x\n", sh_mem->Data, sh_mem->Address);
				success = this->bus_write(sh_mem->Data, sh_mem->Address, 4);
				break;
			case CAS_SYSBUS_CMD_Reset:
				DB_PRINT("BUS Reset\n");
				rst_out = true;
				wait();//Wait 1 cycle
				rst_out = false;
				success = true;
				break;
			case CAS_SYSBUS_CMD_Exit:
				DB_PRINT("Simulation end\n");
				success = true;
				connected = false;
				break;
			}
			sh_mem->ACK = (success)? 0x01:0x00;
			sem_post(SEM_BUS_Ack);
			if(!connected) break;
		}
		
		wait();/* wait for next clock trigger */
	}
	raise(SIGINT);
}

/*=================== TLM Bus Access Function ====================*/
bool qsbridge::bus_read(uint32_t* data, uint32_t addr, uint32_t length)
{
    bool success;
    success = bus_b_access(false, (sc_dt::uint64)addr, reinterpret_cast<unsigned char*>(data), length);

    if(!success) {
        DB_PRINT("bus read 0x%X failed\n", addr);
	}

	return success;
}

bool qsbridge::bus_read64(uint64_t* data, uint64_t addr, uint32_t length)
{
    bool success1;
    bool success2;
    uint32_t tmp;

    success1 = bus_b_access(false, (sc_dt::uint64)addr, reinterpret_cast<unsigned char*>(&tmp), length);
    *data = tmp;
    success2 = bus_b_access(false, (sc_dt::uint64)addr + 4, reinterpret_cast<unsigned char*>(&tmp), length);
    *data |= ((uint64_t)tmp) << 32;

    if(success1 && success2)
        return true;
    else
    {
        DB_PRINT("bus read 64 0x%lX failed\n", addr);
        return false;
    }
}

bool qsbridge::bus_write(uint32_t data, uint32_t addr, uint32_t length)
{
    bool success;
    success = bus_b_access(true, (sc_dt::uint64)addr, reinterpret_cast<unsigned char*>(&data), length);

    if(!success) {
        DB_PRINT("bus write 0x%X:0x%X failed\n", addr, data);
	}
	return success;
}

bool qsbridge::bus_write64(uint64_t data, uint64_t addr, uint32_t length)
{
    bool success1;
    bool success2;

    uint32_t tmp;

    tmp = data;
    success1 = bus_b_access(true, (sc_dt::uint64)addr, reinterpret_cast<unsigned char*>(&tmp), length);

    tmp = data >> 32;
    success2 = bus_b_access(true, (sc_dt::uint64)addr + 4, reinterpret_cast<unsigned char*>(&tmp), length);

    if(success1 && success2)
        return true;
    else
    {
        DB_PRINT("bus write 64 0x%lX:0x%lX failed\n", addr, data);
        return false;
    }
}

```
:::

## QEMU 部分 (Virtual Hardware)

### 定義

即 QEMU-SystemC bridge QEMU 端

:::spoiler `qemu-6.2.0-rc2/hw/char/caslab_sysbridge.c`
```c=
#include "qemu/osdep.h"
#include "hw/sysbus.h"
#include "chardev/char-fe.h"
#include "qemu/log.h"
// added
#include "migration/vmstate.h"
#include "hw/qdev-properties-system.h"
#include "hw/qdev-core.h"
#include "hw/irq.h"

#include <fcntl.h>    /* For O_* constants */
#include <sys/stat.h> /* For mode constants */
#include <sys/ipc.h>
#include <sys/shm.h>
#include <pthread.h>
#include <semaphore.h>

#define CASLAB_SYSBRG_DEBUG

#ifdef CASLAB_SYSBRG_DEBUG
#define DB_PRINT(...)                               \
    do                                              \
    {                                               \
        fprintf(stderr, "[DEBUG]: %s: ", __func__); \
        fprintf(stderr, ##__VA_ARGS__);             \
    } while (0);
#else
#define DB_PRINT(...)
#endif

#define DeviceMemRange 0x1000
#define DeviceMemMask 0x0FFF
#define TYPE_CASLAB_SYSBRG "caslab_sysbridge"
#define CASLAB_SYSBRG(obj) OBJECT_CHECK(CASLabBridgeState, \
                                        (obj),             \
                                        TYPE_CASLAB_SYSBRG)

#define CAS_SYSBRG_SMID 0x8000
#define CAS_SYSBRG_IRQSEM "/CASSYS_IRQ"
#define CAS_SYSBRG_DATASEM "/CASSYS_DATA"
#define CAS_SYSBRG_ACKSEM "/CASSYS_ACK"

#define CAS_SYSBUS_CMD_Read 0x00
#define CAS_SYSBUS_CMD_Write 0x01
#define CAS_SYSBUS_CMD_Reset 0x02
#define CAS_SYSBUS_CMD_Exit 0x04

typedef struct
{
    uint32_t Address;
    uint32_t Data;
    uint32_t ACK; // For bus access ack/nak
    uint32_t CMD;
    uint32_t ISR;
} CAS_SYSBUS_DATA;

typedef struct
{
    SysBusDevice parent_obj;
    MemoryRegion iomem;
    CharBackend chr;
    qemu_irq irq;
    const unsigned char *id;

    sem_t *SEM_IRQ;
    sem_t *SEM_BUS;
    sem_t *SEM_BUS_Ack;
    sem_t SEM_BUS_Key; // Unnamed semaphore，作為 shared memory 的 mutex lock。SystemC 部分因為不會造成 race 問題所以拿掉這個
    bool _connected;
    int shm_id;
    volatile CAS_SYSBUS_DATA *sh_mem;
} CASLabBridgeState;

volatile bool PrgExit = false;
static pthread_t IRQ_Thread;
static CASLabBridgeState *_dev;
void *Thread_UpdateIRQ(void *);

static const unsigned char caslab_sysbridge_id_arm[8] =
    {0x11, 0x10, 0x88, 0x08, 0x0d, 0xf0, 0x05, 0xb1};

static void caslab_sysbridge_deinit(CASLabBridgeState *s)
{
    struct shmid_ds buf;
    struct timespec wait_time = {0}; // 等待 exit 的最長時間
    DB_PRINT("SYSBUS DEINIT\n");
    clock_gettime(CLOCK_REALTIME, &wait_time); // CLOCK_REALRIME 表示 wall-clock time，將 wall-clock time 存進 wait_time
    wait_time.tv_sec += 3;                     // wait 3s before force exit.

    if (s->_connected)
    {
        s->_connected = false;
        DB_PRINT("Disonnceting from HW.\n");
        DB_PRINT("Invoking HW Bus Signal.\n")
        s->sh_mem->CMD = CAS_SYSBUS_CMD_Exit;
        while (sem_trywait(s->SEM_BUS_Ack) == 0)
            ;
        sem_post(s->SEM_BUS);
        sem_timedwait(s->SEM_BUS_Ack, &wait_time);
        // reset sems.
        while (sem_trywait(s->SEM_BUS) == 0)
            ;
        while (sem_trywait(s->SEM_BUS_Ack) == 0)
            ;
        while (sem_trywait(s->SEM_IRQ) == 0)
            ;
        sem_close(s->SEM_BUS);
        sem_close(s->SEM_BUS_Ack);
        sem_close(s->SEM_IRQ);
    }

    shmdt((void *)s->sh_mem);
    sem_destroy(&s->SEM_BUS_Key);
    if (shmctl(s->shm_id, IPC_STAT, &buf) == 0)
        shmctl(s->shm_id, IPC_RMID, &buf); // Delete shared memory.
}

/*
 * 程式正常 exit/return 時會執行的 function
 */
static void ON_Module_Exit(void)
{
    PrgExit = true;
    if (_dev != NULL)
    {
        sem_post(_dev->SEM_IRQ); // 因為 Thread_UpdateIRQ() 內有 sem_wait(s->SEM_IRQ)，可能 thread 會在這邊 wait 導致 thread 無法正常退出，故新增此行
        // pthread_join(IRQ_Thread, NULL);
        pthread_cancel(IRQ_Thread);
        caslab_sysbridge_deinit(_dev);
    }
}

/*
 * thread 主程式，用來作為 IRQ listener
 */
void *Thread_UpdateIRQ(void *mem)
{
    CASLabBridgeState *s;
    s = (CASLabBridgeState *)mem;

    while (!PrgExit)
    {
        if (s->_connected)
        {
            uint32_t ISR;
            DB_PRINT("Waiting Device IRQ Signal.\n");
            sem_wait(s->SEM_IRQ);
            if (PrgExit)
                break;
            if (s->sh_mem->CMD == CAS_SYSBUS_CMD_Exit)
            {
                sem_wait(&s->SEM_BUS_Key);
                s->sh_mem->CMD = CAS_SYSBUS_CMD_Read;
                sem_post(&s->SEM_BUS_Key);
                s->_connected = false;
                DB_PRINT("Device Disconnected.\n");
                while (sem_trywait(s->SEM_BUS) == 0)
                    ;
                while (sem_trywait(s->SEM_BUS_Ack) == 0)
                    ;
                while (sem_trywait(s->SEM_IRQ) == 0)
                    ;
                continue;
            }

            ISR = s->sh_mem->ISR;
            DB_PRINT("Received IRQ Signal from HW.(0x%02x)\n", ISR);

            qemu_mutex_lock_iothread(); // 由於以下會使用到 internal I/O usage (ex: qemu_irq_raise())，任何非 QEMU main thread 的 thread 要用到 internal I/O 時必須用此 macro 保護
            if (ISR & 0x01)
                qemu_irq_raise(s->irq);
            else
                qemu_irq_lower(s->irq);
            qemu_mutex_unlock_iothread(); // Done, now unlock it.
        }
        else
        {
            DB_PRINT("Waiting Device up...\n");
            sem_post(s->SEM_BUS);     // ┐
            sem_wait(s->SEM_BUS_Ack); // ┘參考 qsbridge.cpp 內註解
            s->_connected = true;
            DB_PRINT("Device connected.\n");
            if (PrgExit)
                break;
        }
    }
    DB_PRINT("IRQ Listener exited.\n");
    return NULL;
}

static const VMStateDescription vmstate_caslab_sysbridge = {
    .name = TYPE_CASLAB_SYSBRG,
    .version_id = 1,
    .minimum_version_id = 1,
    .fields = (VMStateField[]){
        // nothing needed to be saved here.
        VMSTATE_END_OF_LIST(),
    }};

static uint64_t caslab_sysbridge_read(void *opaque, hwaddr offset, unsigned size)
{
    CASLabBridgeState *s = opaque;
    uint64_t ret = 0;
    hwaddr temp;
    switch (offset & DeviceMemMask)
    {
    case 0xFE0 ... 0xFFC: // 以最末 32 bytes 作為裝置 ID
        temp = (offset & DeviceMemMask) - 0xFE0;
        ret = caslab_sysbridge_id_arm[temp >> 2];
        break;
    default:
        if (s->_connected && offset < DeviceMemRange)
        {
            sem_wait(&s->SEM_BUS_Key); // Acquire the key to access BUS

            s->sh_mem->CMD = CAS_SYSBUS_CMD_Read;
            s->sh_mem->Address = offset;
            sem_post(s->SEM_BUS);
            sem_wait(s->SEM_BUS_Ack);
            ret = s->sh_mem->Data;

            sem_post(&s->SEM_BUS_Key); // Release the key
        }
        break;
    }
    return ret;
}

static void caslab_sysbridge_write(void *opaque, hwaddr offset, uint64_t value, unsigned size)
{
    CASLabBridgeState *s = opaque;
    if (offset >= DeviceMemRange)
    {
        fprintf(stderr, "Device ERROR: Overflow!!!\n");
        return;
    }
    if (!s->_connected)
        return;

    sem_wait(&s->SEM_BUS_Key); // Acquire the key to access BUS

    s->sh_mem->CMD = CAS_SYSBUS_CMD_Write;
    s->sh_mem->Address = offset;
    s->sh_mem->Data = (uint32_t)value;
    sem_post(s->SEM_BUS);
    sem_wait(s->SEM_BUS_Ack);

    sem_post(&s->SEM_BUS_Key); // Release the key
}

static void caslab_sysbridge_reset(DeviceState *dev)
{
    CASLabBridgeState *s = CASLAB_SYSBRG(dev);

    if (!s->_connected)
        return;

    sem_wait(&s->SEM_BUS_Key); // Acquire the key to access BUS
    s->sh_mem->CMD = CAS_SYSBUS_CMD_Reset;
    sem_post(s->SEM_BUS);
    sem_wait(s->SEM_BUS_Ack);
    sem_post(&s->SEM_BUS_Key); // Release the key
}

static void caslab_sysbridge_realize(DeviceState *dev, Error **errp)
{
    // Nothing to do here currently.
}

static const MemoryRegionOps caslab_sysbridge_ops = {
    .read = caslab_sysbridge_read,
    .write = caslab_sysbridge_write,
    .endianness = DEVICE_NATIVE_ENDIAN,
    .valid.min_access_size = 4,
    .valid.max_access_size = 4,
};

static void caslab_sysbridge_init(Object *obj)
{
    SysBusDevice *sbd = SYS_BUS_DEVICE(obj);
    CASLabBridgeState *s = CASLAB_SYSBRG(obj);
    int init_state = 5;
    struct shmid_ds buf;
    s->_connected = false;
    s->shm_id = -1;

    memory_region_init_io(&s->iomem, OBJECT(s), &caslab_sysbridge_ops, s,
                          TYPE_CASLAB_SYSBRG, DeviceMemRange);

    sysbus_init_mmio(sbd, &s->iomem);
    sysbus_init_irq(sbd, &s->irq);
    do
    {
        if (0 != sem_init(&s->SEM_BUS_Key, 0, 1)) // Assign only 1 key to act like mutex
            break;
        init_state--;

        // 和 qsbridge.cpp 共用 semaphore
        s->SEM_BUS = sem_open(CAS_SYSBRG_DATASEM, O_CREAT, S_IRUSR | S_IWUSR, 0);
        s->SEM_BUS_Ack = sem_open(CAS_SYSBRG_ACKSEM, O_CREAT, S_IRUSR | S_IWUSR, 0);
        s->SEM_IRQ = sem_open(CAS_SYSBRG_IRQSEM, O_CREAT, S_IRUSR | S_IWUSR, 0);
        if (SEM_FAILED == s->SEM_BUS || SEM_FAILED == s->SEM_BUS_Ack || SEM_FAILED == s->SEM_IRQ)
            break;
        init_state--;

        // Reset sems
        while (sem_trywait(s->SEM_BUS) == 0)
            ;
        while (sem_trywait(s->SEM_BUS_Ack) == 0)
            ;
        while (sem_trywait(s->SEM_IRQ) == 0)
            ;

        s->shm_id = shmget(CAS_SYSBRG_SMID, sizeof(CAS_SYSBUS_DATA), 0666 | IPC_CREAT);
        if (s->shm_id == -1)
            break;
        init_state--;

        s->sh_mem = shmat(s->shm_id, NULL, 0);
        if (s->sh_mem == (void *)-1)
            break;
        init_state--;

        memset((void *)s->sh_mem, 0, sizeof(CAS_SYSBUS_DATA));
        if (0 != pthread_create(&IRQ_Thread, NULL, Thread_UpdateIRQ, (void *)s))
            break;
        init_state--;
    } while (0);

    switch (init_state)
    {
    case 0:
        DB_PRINT("SYSBUS INIT\n");
        break;
    case 1: // pthread_create failed
        fprintf(stderr, "ERROR: pthread failed\n");
        shmdt((void *)s->sh_mem);
        s->sh_mem = NULL;
    case 2: // shmat failed
        fprintf(stderr, "ERROR: shmat failed\n");
        if (shmctl(s->shm_id, IPC_STAT, &buf) == 0)
            shmctl(s->shm_id, IPC_RMID, &buf);
    case 3: // shmget failed
        fprintf(stderr, "ERROR: shmget failed\n");
        while (sem_trywait(s->SEM_BUS) == 0)
            ;
        while (sem_trywait(s->SEM_BUS_Ack) == 0)
            ;
        while (sem_trywait(s->SEM_IRQ) == 0)
            ;
        sem_close(s->SEM_BUS);
        sem_close(s->SEM_BUS_Ack);
        sem_close(s->SEM_IRQ);
    case 4: // shared semaphore failed
        fprintf(stderr, "ERROR: semaphore failed\n");
        sem_destroy(&s->SEM_BUS_Key);
    case 5: // anonymous semaphore failed
        fprintf(stderr, "ERROR: semaphore failed\n");
    default:
        exit(-1);
        break;
    }

    _dev = s;
    if (0 != atexit(ON_Module_Exit)) // 註冊正常 exit/return 時要執行的 function
        fprintf(stderr, "failed to register on exit function, may cause memory violation when exit.\n");
}

static Property caslab_sysbridge_properties[] = {
    DEFINE_PROP_CHR("chardev", CASLabBridgeState, chr),
    DEFINE_PROP_END_OF_LIST(),
};

static void caslab_sysbridge_class_init(ObjectClass *oc, void *data)
{
    DeviceClass *dc = DEVICE_CLASS(oc);

    dc->realize = caslab_sysbridge_realize;
    dc->vmsd = &vmstate_caslab_sysbridge;
    dc->reset = caslab_sysbridge_reset;
    dc->props_ = caslab_sysbridge_properties;
}

static const TypeInfo caslab_sysbridge_info = {
    .name = TYPE_CASLAB_SYSBRG,
    .parent = TYPE_SYS_BUS_DEVICE,
    .instance_size = sizeof(CASLabBridgeState),
    .instance_init = caslab_sysbridge_init,
    .class_init = caslab_sysbridge_class_init};

static void caslab_sysbridge_register_types(void)
{
    type_register_static(&caslab_sysbridge_info);
}

type_init(caslab_sysbridge_register_types)

```
:::

### 設定

[calculator 設定](https://hackmd.io/p4MBgIlaRbm7LBsd5Zh1gw?both#%E8%A8%AD%E5%AE%9A) 的 `qemu-6.2.0-rc2/hw/char/meson.build` 內 `caslab_calculator.c` 改為 `caslab_sysbridge.c`、`qemu-6.2.0-rc2/hw/arm/realview.c` 內 `caslab_calc` 改為 `caslab_sysbridge`

### Kernel Module

和 [calculator Kernel Module](https://hackmd.io/p4MBgIlaRbm7LBsd5Zh1gw?both#Kernel-Module) 相同

### Device Tree

和 [calculator Device Tree](https://hackmd.io/p4MBgIlaRbm7LBsd5Zh1gw?both#Device-Tree) 相同