---
# System prepended metadata

title: Linux OS 2023 Final Project

---

# Linux OS 2023 Final Project
#### 組員： 112552011柯竣凱, 112552003陳柏家, 112552029莫兆全

# **Project Requirements**
1. The user address space of a Linux thread consists of the following segments.
- text segment
- data segment (global variables with initial values)
- BSS segment (global variables without initial values)
- heap segment (memory area allocated through function malloc())
- libraries
- stack segment
2. In this project, you need to add some new system calls first. The functionality and number of the new system calls are determined by you.
3. Write a multi-thread program with three threads (main thread, thread 1, and thread 2) and the new system calls to check which segments of a thread are shared by which other thread(s).
4. You do not need to calculate the size, star address, and end address of each thread segment. However, if you can obtain the the size, star address, and end address of each thread segment, we will add 30 extra points to your project 1 grade.
5. Write a report to describe the results created by your multi-thread program. Your report should contain a figure like the following one to summarize your results.


# **Kernel 與 OS 版本**
#####    Kernel版本: 5.15.74
#####    OS版本: Ubuntu 22.04

# **Kernel 編譯過程**
#### 透過Virtualbox安裝Linux 22.04 64bit
#### 安裝編譯需要的依賴
```shell=
sudo apt install -y python3 build-essential libncurses-dev \
                    libssl-dev  libelf-dev bison flex dwarves \
                    bc cpio zstd
```

#### 下載 Kernel 5.15.74 並解壓縮
```shell=
wget https://cdn.kernel.org/pub/linux/kernel/v3.x/linux-5.15.74.tar.xz
tar -xf linux-5.15.74.tar.xz
cd linux-5.15.74
```
#### 準備 Kernel 設定文件
##### 複製當前環境設定文件
```shell=
cp /boot/config-$(uname -r) .config
```
##### 將CONFIG_SYSTEM_TRUSTED_KEYS清空
- 因為我們編譯過程有遇到憑證問題，所以將CONFIG_SYSTEM_TRUSTED_KEYS清空
```shell=
- CONFIG_SYSTEM_TRUSTED_KEYS="debian/certs/benh@debian.org.cert.pem"
- CONFIG_DEBUG_INFO_BTF="Y"
+ CONFIG_SYSTEM_TRUSTED_KEYS=""
+ CONFIG_DEBUG_INFO_BTF="N"
```
#### 編譯與安裝 Kernel
```shell=
make -j$(nproc)
sudo make modules_install -j$(nproc)
sudo make install -j$(nproc)
sudo update-grub
```
#### 設定grub開機選單
```shell=
vim /etc/default/grub
```
```shell=
- GRUB_HIDDEN_TIMEOUT=0
- GRUB_TIMEOUT=0
+ # GRUB_HIDDEN_TIMEOUT=0
+ GRUB_TIMEOUT=3
```
#### 重新開機，並選擇編譯好的5.15.74版本Kernel

# **新增 Syscall 過程**
1. 新增 `v2p.c`, `get_segments.c`
3. 調整 Makefile
```shell=
obj-y := v2p.o
obj-y := get_segments.o
```
3. 調整 System Call 頭文件: `include/linux/syscalls.h`
```shell=
+ asmlinkage unsigned long sys_get_segments(unsigned long start_stack, void * __user addr);
+ asmlinkage unsigned long sys_v2p(unsigned long *initial, unsigned long *result);
```
4. 新增 System Call Table: `arch/x86/syscalls/syscalls_64.tbl`
```shell=
+ 548     common  get_segments            sys_get_segments
+ 549     common  v2p                     sys_v2p
```
5. [**編譯並安裝**](#-Kernel編譯過程-)


# **Kernel Space code (syscall)**
#### get_segments(548): 用來取得目前線程的各個 segment 虛擬位址
![image](https://hackmd.io/_uploads/r1rqpjZ86.png)
```clike=
/* import head file */
#include <linux/kernel.h>
#include <linux/syscalls.h>

/* define struct */
struct Segment
{
    unsigned long start;
    unsigned long end;
};

struct Segments
{
    struct Segment code;
    struct Segment data;
    struct Segment bss;
    struct Segment heap;
    struct Segment lib;
    struct Segment stack;

    unsigned long start_code;
    unsigned long end_code;
    unsigned long start_data;
    unsigned long end_data;
    unsigned long start_brk;
    unsigned long brk;
    unsigned long mmap_base;
    unsigned long start_stack;
};

/* find segment by traverse vma */
void find_segment(struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, struct Segment *segment, char include_start)
{   
    char continuous = 0;
    char is_start_in_range;
    while (vma) {
        is_start_in_range = include_start ? vma->vm_end >= start_addr : vma->vm_end > start_addr;
        if (!continuous && is_start_in_range) {
            segment->start = vma->vm_start;
            continuous = 1;
        }

        if (continuous && vma->vm_end >= end_addr) {
            segment->end = vma->vm_end;
            break;
        }

        vma = vma->vm_next;
    }
}

/* get segment and find the start and end, then return to user space */
SYSCALL_DEFINE2(get_segments, unsigned long, start_stack, void * __user, addr)
{
    struct Segments segments;
    struct mm_struct *mm = current->mm;

    start_stack = start_stack ? start_stack : mm->start_stack;
    
    //code segment
    segments.start_code = mm->start_code;
    segments.end_code = mm->end_code;
    //data segment
    segments.start_data = mm->start_data;
    segments.end_data = mm->end_data;
    //heap segment
    segments.start_brk = mm->start_brk;
    segments.brk = mm->brk;
    //lib segment
    segments.mmap_base = mm->mmap_base;
    //stack segment
    segments.start_stack = start_stack;

    find_segment(mm->mmap, mm->start_data,    mm->end_data,        &segments.data,  1);
    find_segment(mm->mmap, mm->start_code,    segments.data.start, &segments.code,  1);
    find_segment(mm->mmap, mm->end_data,      mm->end_data,        &segments.bss,   1);
    find_segment(mm->mmap, mm->start_brk,     mm->brk,             &segments.heap,  0);
    find_segment(mm->mmap, segments.heap.end, mm->mmap_base,       &segments.lib,   0);
    find_segment(mm->mmap, start_stack,       start_stack,         &segments.stack, 0);

    return copy_to_user(addr, &segments, sizeof(struct Segments));
}
```
#### Source Code 說明
- find_segment: traverse vma去取得起始跟結束位址
    - 這個函數在VMA列表中尋找一個記憶體區段。
    - 根據給定的地址範圍，它設置提供的 segment 結構的 start 和 end。 
        - while (vma) 循環遍歷每個 VMA。is_start_in_range 邏輯檢查 VMA 的 end 是否在指定範圍內。
- 拿到mm_struct取得各個segment的起始與結束地址
    - text start and end
        - mm->start_code
        - mm->start_data
    - lib start and end
        - segments.heap.end
        - mm->mmap_base
    - data start and end
        - mm->start_data
        - mm->end_data
    - heap start and end
        - mm->start_brk
        - mm->brk
    - stack start and end
        - mm->start_stack
- 將更新後的Segments複製回user space
    - copy_to_user(addr, &segments, sizeof(struct Segments));

#### v2p(549):用來將虛擬記憶體位址轉為實體記憶體位址
![Screenshot 2024-01-01 at 10.48.18 PM](https://hackmd.io/_uploads/HJEaLIlOT.png)
```clike=
#include <linux/kernel.h>
#include <linux/string.h>
#include <linux/uaccess.h>
#include <linux/init_task.h>
#include <linux/syscalls.h>

SYSCALL_DEFINE2(v2p, unsigned long *, initial, unsigned long *, result)
{
	pgd_t *pgd;
	p4d_t *p4d;
	pud_t *pud;
	pmd_t *pmd;
	pte_t *pte;
   
	int i = 0;
	unsigned long page_addr = 0;
	unsigned long page_offset = 0;
	unsigned long *vir_adds = kmalloc(1 * sizeof(unsigned long), GFP_KERNEL);
	unsigned long *phy_adds = kmalloc(1 * sizeof(unsigned long), GFP_KERNEL);
	copy_from_user(vir_adds, initial, 1 * sizeof(unsigned long));

	pgd = pgd_offset(current->mm, *(vir_adds + i));
	printk("pgd_val = 0x%lx\n", pgd_val(*pgd));
	printk("pgd_index = %lu\n", pgd_index(*(vir_adds + i)));
	if (pgd_none(*pgd)) {
		printk("not mapped in pgdn");
	}

	p4d = p4d_offset(pgd, *(vir_adds + i));
	printk("p4d_val = 0x%lx\n", p4d_val(*p4d));
	printk("p4d_index = %lu\n", p4d_index(*(vir_adds + i)));
	if (p4d_none(*p4d)) {
		printk("not mapped in p4d");
	}

	pud = pud_offset(p4d, *(vir_adds + i));
	printk("pud_val = 0x%lx\n", pud_val(*pud));
	printk("pud_index = %lu\n", pud_index(*(vir_adds + i)));
	if (pud_none(*pud)) {
		printk("not mapped in pudn");
	}

	pmd = pmd_offset(pud, *(vir_adds + i));
	printk("pmd_val = 0x%lx\n", pmd_val(*pmd));
	printk("pmd_index = %lu\n", pmd_index(*(vir_adds + i)));
	if (pmd_none(*pmd)) {
		printk("not mapped in pmdn");
	}

	pte = pte_offset_kernel(pmd, *(vir_adds + i));
	printk("pte_val = 0x%lx\n", pte_val(*pte));
	printk("pte_index = %lu\n", pte_index(*(vir_adds + i)));
	if (pte_none(*pte)) {
		printk("not mapped in pten");
	}

	page_addr = pte_val(*pte) & PAGE_MASK; 
	page_offset = *(vir_adds + i) & ~PAGE_MASK;
	*(phy_adds + i) = page_addr | page_offset;

	printk("page_addr = %lx\n", page_addr);
	printk("page_offset = %lx\n", page_offset);
	printk("vaddr =%lx, paddr = %lx\n", *(vir_adds + i), *(phy_adds + i));

	copy_to_user(result, phy_adds, 1 * sizeof(unsigned long));
	kfree(vir_adds);
	kfree(phy_adds);

	return 0;
}
```
#### Source Code 說明
- Copy virtual address from user space to kernel space
    - copy_from_user(va,vaddr,sizeof(unsigned long));
- 拿到當前process的virtual address的pgd, pud, pmd, pte
    - pgd = pgd_offset(current->mm, paddr);
    - p4d = p4d_offset(pgd, vaddr);
    - pud = pud_offset(p4d, vaddr);
    - pmd = pmd_offset(pud, vaddr);
    - pte = pte_offset_kernel(pmd, vaddr);
- 從pte中拿到physical address的start_index
    - p_index = pte_val(*pte) & PAGE_MASK;
- 從virtual address中取得physical address的offset
    - p_offset = t1 & ~PAGE_MASK;
- 組合physical address
    - t2 = p_index | p_offset;
- Copy physical address from kernel space to user space
    - copy_to_user(paddr,pa,sizeof(unsigned long));

# **User Space code (Test code)** 
#### Reference diagram
![image](https://hackmd.io/_uploads/rJ79RIK_6.png)

![image](https://hackmd.io/_uploads/ryY5AUY_a.png)

#### Test Source Code
```clike=
#include <syscall.h>
#include <sys/types.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>
#include <semaphore.h>

sem_t s1, s2;
int i = 0;
int bss_val;
int data_val = 12345;
int code_val(){return 0;}
char printArray [6][6] = {"stack", "heap", "lib", "bss", "data", "code"};
struct ThreadSegments thread_1_segs_addr;
struct ThreadSegments thread_2_segs_addr;
struct ThreadSegments main_thread_segs_addr;

struct ThreadSegments
{   
    // {stack, heap, lib, bss, data, code}  	
    unsigned long vir_addrs[6];
};

struct Segment
{
    unsigned long start;
    unsigned long end;
};

struct Segments
{
    struct Segment code;
    struct Segment data;
    struct Segment bss;
    struct Segment heap;
    struct Segment lib;
    struct Segment stack;

    unsigned long start_code;
    unsigned long end_code;
    unsigned long start_data;
    unsigned long end_data;
    unsigned long start_brk;
    unsigned long brk;
    unsigned long mmap_base;
    unsigned long start_stack;
};

void get_seg_start_end_addr(void *start_stack)
{    
    struct Segments segments;
    unsigned long *virtual_address_start;
    unsigned long *virtual_address_end;
    unsigned long physical_address_start;
    unsigned long physical_address_end;
    
    //get segments
    syscall(548, (unsigned long)start_stack, &segments);

    //code    
    virtual_address_start = &segments.code.start;
    virtual_address_end = &segments.code.end;
    syscall(549, &virtual_address_start, &physical_address_start);              
    syscall(549, &virtual_address_end, &physical_address_end);              
    printf("segment:code vir_addr: start:%lx-end:%lx", segments.code.start, segments.code.end);
    printf(", phy_addr: start:%lx-end:%lx \n", physical_address_start, physical_address_end);

    //data
    virtual_address_start = &segments.data.start;
    virtual_address_end = &segments.data.end;
    syscall(549, &virtual_address_start, &physical_address_start);
    syscall(549, &virtual_address_end, &physical_address_end);
    printf("segment:data vir_addr: start:%lx-end:%lx", segments.data.start, segments.data.end);
    printf(", phy_addr: start:%lx-end:%lx \n", physical_address_start, physical_address_end);

    //bss
    virtual_address_start = &segments.bss.start;
    virtual_address_end = &segments.bss.end;
    syscall(549, &virtual_address_start, &physical_address_start);
    syscall(549, &virtual_address_end, &physical_address_end);
    printf("segment:bss vir_addr: start:%lx-end:%lx", segments.bss.start, segments.bss.end);
    printf(", phy_addr: start:%lx-end:%lx \n", physical_address_start, physical_address_end);
                
    //heap
    virtual_address_start = &segments.heap.start;
    virtual_address_end = &segments.heap.end;
    syscall(549, &virtual_address_start, &physical_address_start);
    syscall(549, &virtual_address_end, &physical_address_end);
    printf("segment:heap vir_addr: start:%lx-end:%lx", segments.heap.start, segments.heap.end);
    printf(", phy_addr: start:%lx-end:%lx \n", physical_address_start, physical_address_end);
     
    //lib
    virtual_address_start = &segments.lib.start;
    virtual_address_end = &segments.lib.end;
    syscall(549, &virtual_address_start, &physical_address_start);
    syscall(549, &virtual_address_end, &physical_address_end);
    printf("segment:lib vir_addr: start:%lx-end:%lx", segments.lib.start, segments.lib.end);
    printf(", phy_addr: start:%lx-end:%lx \n", physical_address_start, physical_address_end);
    
    //stack
    virtual_address_start = &segments.stack.start;
    virtual_address_end = &segments.stack.end;
    syscall(549, &virtual_address_start, &physical_address_start);
    syscall(549, &virtual_address_end, &physical_address_end);
    printf("segment:stack vir_addr: start:%lx-end:%lx", segments.stack.start, segments.stack.end);
    printf(", phy_addr: start:%lx-end:%lx \n", physical_address_start, physical_address_end);
}

void thread_variable_init(struct ThreadSegments *thread_segs_addr, int *stack_val)
{
     thread_segs_addr->vir_addrs[0] = (unsigned long)stack_val;
     thread_segs_addr->vir_addrs[1] = (unsigned long)malloc(sizeof(int));
     thread_segs_addr->vir_addrs[2] = (unsigned long)&printf;
     thread_segs_addr->vir_addrs[3] = (unsigned long)&bss_val;
     thread_segs_addr->vir_addrs[4] = (unsigned long)&data_val;
     thread_segs_addr->vir_addrs[5] = (unsigned long)code_val;
}

void check_same_segment()
{
     printf("Segments are 'shared' resources by each threads\n");
     for (i = 0; i < 6; i++) {    
         if(thread_1_segs_addr.vir_addrs[i] == thread_2_segs_addr.vir_addrs[i] 
             && thread_2_segs_addr.vir_addrs[i] == main_thread_segs_addr.vir_addrs[i]) {
	     printf("segment:%s\n", printArray[i]);
	 }
     }	     
     
     printf("Segments are 'not shared' resources by each threads\n");
     for (i = 0; i < 6; i++) {  
         if(thread_1_segs_addr.vir_addrs[i] != thread_2_segs_addr.vir_addrs[i] 
             || thread_2_segs_addr.vir_addrs[i] != main_thread_segs_addr.vir_addrs[i]) {
             printf("segment:%s\n", printArray[i]);
         }
     }
}

void *thread_1 ()
{    
     sem_wait(&s1);
     int stack_val_1 = 54321;
     unsigned long physical_address;
     unsigned long virtual_address;
     void *stack_addr;
     size_t stack_size;
     pthread_attr_t attr;
     //struct ThreadSegments thread_1_segs_addr;

     printf("========================= Thread1 !=========================\n");
     thread_variable_init(&thread_1_segs_addr, &stack_val_1);
     // 1. basic: check which segments are shared resources by each threads
     printf("1. basic: check which segments are shared resources by each threads\n");
     for (i = 5; i >= 0; i--) {
         virtual_address = thread_1_segs_addr.vir_addrs[i];
	 syscall(549, &virtual_address, &physical_address);
	 printf("segment:%s, vir_addr:%lx, phy_addr:%lx \n", printArray[i], thread_1_segs_addr.vir_addrs[i], physical_address);
     }
     pthread_getattr_np(pthread_self(), &attr);
     pthread_attr_getstack(&attr, &stack_addr, &stack_size);
     // 2. bonus: get the start address and end address of each segment
     printf("2. bonus: get the start address and end address of each segment\n");
     get_seg_start_end_addr(stack_addr);
     printf("========================= Thread1 !=========================\n");
     printf("--------------------------------------------------------------------------------------------------------\n");
     sem_post(&s2);

     return NULL;	      
}

void *thread_2 ()
{    
     sem_wait(&s2);
     int stack_val_2 = 54321;
     unsigned long physical_address;
     unsigned long virtual_address;
     void *stack_addr;
     size_t stack_size;
     pthread_attr_t attr;
     //struct ThreadSegments thread_2_segs_addr;

     printf("========================= Thread2 !=========================\n");
     thread_variable_init(&thread_2_segs_addr, &stack_val_2);
     // 1. basic: check which segments are shared resources by each threads
     printf("1. basic: check which segments are shared resources by each threads\n");
     for (i = 5; i >= 0; i--) {
         virtual_address = thread_2_segs_addr.vir_addrs[i];
	 syscall(549, &virtual_address, &physical_address);
	 printf("segment:%s, vir_addr:%lx, phy_addr:%lx \n", printArray[i], thread_2_segs_addr.vir_addrs[i], physical_address);
     }	
     pthread_getattr_np(pthread_self(), &attr);
     pthread_attr_getstack(&attr, &stack_addr, &stack_size);
     // 2. bonus: get the start address and end address of each segment
     printf("2. bonus: get the start address and end address of each segment\n");
     get_seg_start_end_addr(stack_addr);
     printf("========================= Thread2 !=========================\n");
     printf("--------------------------------------------------------------------------------------------------------\n");
     return NULL;
}

int main ()
{       	
    sem_init(&s1, 0, 0);
    sem_init(&s2, 0, 0);	    
    pthread_t p1, p2;
    pthread_create(&p1,NULL,thread_1,NULL);
    pthread_create(&p2,NULL,thread_2,NULL);
    
    // check same segment 
    int stack_val_main = 54321;
    unsigned long physical_address;
    unsigned long *virtual_address;
    //struct ThreadSegments main_thread_segs_addr;
    
    printf("========================= Main !=========================\n");
    thread_variable_init(&main_thread_segs_addr, &stack_val_main);
    // 1. basic: check which segments are shared resources by each threads
    printf("1. basic: check which segments are shared resources by each threads\n");
    for (i = 5; i >= 0; i--) {
	virtual_address = main_thread_segs_addr.vir_addrs[i];	
	syscall(549, &virtual_address, &physical_address);
	printf("segment:%s, vir_addr:%lx, phy_addr:%lx \n", printArray[i], main_thread_segs_addr.vir_addrs[i], physical_address);
    }
    // 2. bonus: get the start address and end address of each segment
    printf("2. bonus: get the start address and end address of each segment\n");
    get_seg_start_end_addr(NULL);
    printf("========================= Main !=========================\n"); 
    printf("--------------------------------------------------------------------------------------------------------\n");
    sem_post(&s1);
    pthread_join(p1,NULL);
    pthread_join(p2,NULL);

    printf("1. basic: check which segments are shared resources by each threads\n"); 
    check_same_segment();
    return 0;
} 
```

#### 輸出
```txt=
========================= Main !=========================
1. basic: check which segments are shared resources by each threads
segment:code, vir_addr:55975cb472b9, phy_addr:1579d32b9 
segment:data, vir_addr:55975cb4a020, phy_addr:800000016af60020 
segment:bss, vir_addr:55975cb4a0e4, phy_addr:800000016af600e4 
segment:lib, vir_addr:7fd246e116f0, phy_addr:10e4b96f0 
segment:heap, vir_addr:55975e7178f0, phy_addr:800000012a23c8f0 
segment:stack, vir_addr:7ffdcc045b74, phy_addr:800000016af3ab74 
2. bonus: get the start address and end address of each segment
segment:code vir_addr: start:55975cb46000-end:55975cb49000, phy_addr: start:800000016af3aab0-end:800000016af3aab8 
segment:data vir_addr: start:55975cb49000-end:55975cb4b000, phy_addr: start:800000016af3aac0-end:800000016af3aac8 
segment:bss vir_addr: start:55975cb4a000-end:55975cb4b000, phy_addr: start:800000016af3aad0-end:800000016af3aad8 
segment:heap vir_addr: start:55975e717000-end:55975e738000, phy_addr: start:800000016af3aae0-end:800000016af3aae8 
segment:lib vir_addr: start:7fd245dac000-end:7fd247027000, phy_addr: start:800000016af3aaf0-end:800000016af3aaf8 
segment:stack vir_addr: start:7ffdcc027000-end:7ffdcc048000, phy_addr: start:800000016af3ab00-end:800000016af3ab08 
========================= Main !=========================

========================= Thread1 !=========================
1. basic: check which segments are shared resources by each threads
segment:code, vir_addr:55975cb472b9, phy_addr:1579d32b9 
segment:data, vir_addr:55975cb4a020, phy_addr:800000016af60020 
segment:bss, vir_addr:55975cb4a0e4, phy_addr:800000016af600e4 
segment:lib, vir_addr:7fd246e116f0, phy_addr:10e4b96f0 
segment:heap, vir_addr:7fd240000b70, phy_addr:800000013956eb70 
segment:stack, vir_addr:7fd246dacdec, phy_addr:800000013f340dec 
2. bonus: get the start address and end address of each segment
segment:code vir_addr: start:55975cb46000-end:55975cb49000, phy_addr: start:800000013f340d20-end:800000013f340d28 
segment:data vir_addr: start:55975cb49000-end:55975cb4b000, phy_addr: start:800000013f340d30-end:800000013f340d38 
segment:bss vir_addr: start:55975cb4a000-end:55975cb4b000, phy_addr: start:800000013f340d40-end:800000013f340d48 
segment:heap vir_addr: start:55975e717000-end:55975e738000, phy_addr: start:800000013f340d50-end:800000013f340d58 
segment:lib vir_addr: start:7fd240000000-end:7fd247027000, phy_addr: start:800000013f340d60-end:800000013f340d68 
segment:stack vir_addr: start:7fd2465ae000-end:7fd246db1000, phy_addr: start:800000013f340d70-end:800000013f340d78 
========================= Thread1 !=========================

========================= Thread2 !=========================
1. basic: check which segments are shared resources by each threads
segment:code, vir_addr:55975cb472b9, phy_addr:1579d32b9 
segment:data, vir_addr:55975cb4a020, phy_addr:800000016af60020 
segment:bss, vir_addr:55975cb4a0e4, phy_addr:800000016af600e4 
segment:lib, vir_addr:7fd246e116f0, phy_addr:10e4b96f0 
segment:heap, vir_addr:7fd240000c90, phy_addr:800000013956ec90 
segment:stack, vir_addr:7fd2465abdec, phy_addr:800000016e99adec 
2. bonus: get the start address and end address of each segment
segment:code vir_addr: start:55975cb46000-end:55975cb49000, phy_addr: start:800000016e99ad20-end:800000016e99ad28 
segment:data vir_addr: start:55975cb49000-end:55975cb4b000, phy_addr: start:800000016e99ad30-end:800000016e99ad38 
segment:bss vir_addr: start:55975cb4a000-end:55975cb4b000, phy_addr: start:800000016e99ad40-end:800000016e99ad48 
segment:heap vir_addr: start:55975e717000-end:55975e738000, phy_addr: start:800000016e99ad50-end:800000016e99ad58 
segment:lib vir_addr: start:7fd240000000-end:7fd247027000, phy_addr: start:800000016e99ad60-end:800000016e99ad68 
segment:stack vir_addr: start:7fd245dad000-end:7fd2465ad000, phy_addr: start:800000016e99ad70-end:800000016e99ad78 
========================= Thread2 !=========================
1. basic: check which segments are shared resources by each threads
Segments are 'shared' resources by each threads
segment:lib
segment:bss
segment:data
segment:code
Segments are 'not shared' resources by each threads
segment:stack
segment:heap
```

# **Results分析**
#### 輸出說明
- 我們建立了3個Thread(Main Thread、Thread1、Thread2)，查看他們之間共用的情況，有共用的部分我們使用綠色將其標示出來。
- Thread 之間共用 Lib、Bss、Data、Code 段記憶體位置，不共用 Stack、Heap 段記憶體位置。
    - 理論上應該只有stack是不共享，但是我們的結果中heap也是各個thread不共享，查詢一些資料發現，可能是glibc 中的 malloc，會在 heap 上實現自己的管理策略，這可能會讓你覺得 heap 沒有被共享。例如，它們可能會為不同線程分配不同的內存區域以減少鎖的競爭，但這些區域仍然位於共享的 heap 中。
    - 所以雖然看起來在記憶體位址不同，但是整體來說不同thread還是共用common heap才對。
- 我們也找出了每個segment的實體記憶體起始、結束位址以及segment大小。
- Figure for results
    - Findings
        - 我們的Stack of threads排序是往下長，跟原圖上面從下往上長有差異
        - Library的位址在heap跟stack之間，但是在sub-threads上面，跟原圖有差異
![Screenshot 2024-01-06 at 3.45.31 PM](https://hackmd.io/_uploads/SJN4HFL_T.png)

| Segment | Address Type | Main Thread                               | Thread-1                                 | Thread-2                                 |
|---------|--------------|-------------------------------------------|------------------------------------------|------------------------------------------|
| Code    | Virtual      | <span style="color:green">`0x55975cb472b9`</span> | <span style="color:green">`0x55975cb472b9`</span> | <span style="color:green">`0x55975cb472b9`</span> |
|         | Physical     | <span style="color:green">`0x1579d32b9`</span>    | <span style="color:green">`0x1579d32b9`</span>    | <span style="color:green">`0x1579d32b9`</span>    |
| Data    | Virtual      | <span style="color:green">`0x55975cb4a020`</span> | <span style="color:green">`0x55975cb4a020`</span> | <span style="color:green">`0x55975cb4a020`</span> |
|         | Physical     | <span style="color:green">`0x800000016af60020`</span> | <span style="color:green">`0x800000016af60020`</span> | <span style="color:green">`0x800000016af60020`</span> |
| BSS     | Virtual      | <span style="color:green">`0x55975cb4a0e4`</span> | <span style="color:green">`0x55975cb4a0e4`</span> | <span style="color:green">`0x55975cb4a0e4`</span> |
|         | Physical     | <span style="color:green">`0x800000016af600e4`</span> | <span style="color:green">`0x800000016af600e4`</span> | <span style="color:green">`0x800000016af600e4`</span> |
| Lib     | Virtual      | <span style="color:green">`0x7fd246e116f0`</span> | <span style="color:green">`0x7fd246e116f0`</span> | <span style="color:green">`0x7fd246e116f0`</span> |
|         | Physical     | <span style="color:green">`0x10e4b96f0`</span> | <span style="color:green">`0x10e4b96f0`</span> | <span style="color:green">`0x10e4b96f0`</span> |
| Heap    | Virtual      | `0x55975e7178f0`                          | `0x7fd240000b70`                         | `0x7fd240000c90`                         |
|         | Physical     | `0x800000012a23c8f0`                      | `0x800000013956eb70`                     | `0x800000013956ec90`                     |
| Stack   | Virtual      | `0x7ffdcc045b74`                          | `0x7fd246dacdec`                         | `0x7fd2465abdec`                         |
|         | Physical     | `0x800000016af3ab74`                      | `0x800000013f340dec`                     | `0x800000016e99adec`                     |


#### Main Thread start, end address and address size
| Segment | Address Type | Segment Size (bytes) | Start Address           | End Address             |
|---------|--------------|----------------------|-------------------------|-------------------------|
| Code    | Virtual      | `12288`              | `0x55975cb46000`        | `0x55975cb49000`        |
|         | Physical     |                      | `0x800000016af3aab0`    | `0x800000016af3aab8`    |
| Data    | Virtual      | `8192`               | `0x55975cb49000`        | `0x55975cb4b000`        |
|         | Physical     |                      | `0x800000016af3aac0`    | `0x800000016af3aac8`    |
| BSS     | Virtual      | `4096`               | `0x55975cb4a000`        | `0x55975cb4b000`        |
|         | Physical     |                      | `0x800000016af3aad0`    | `0x800000016af3aad8`    |
| Heap    | Virtual      | `135168`             | `0x55975e717000`        | `0x55975e738000`        |
|         | Physical     |                      | `0x800000016af3aae0`    | `0x800000016af3aae8`    |
| Lib     | Virtual      | `12386304`           | `0x7fd245dac000`        | `0x7fd247027000`        |
|         | Physical     |                      | `0x800000016af3aaf0`    | `0x800000016af3aaf8`    |
| Stack   | Virtual      | `135168`             | `0x7ffdcc027000`        | `0x7ffdcc048000`        |
|         | Physical     |                      | `0x800000016af3ab00`    | `0x800000016af3ab08`    |

#### Thread1 start, end address and address size
| Segment | Address Type | Segment Size (bytes) | Start Address           | End Address             |
|---------|--------------|----------------------|-------------------------|-------------------------|
| Code    | Virtual      | `12288`              | `0x55975cb46000`        | `0x55975cb49000`        |
|         | Physical     |                      | `0x800000013f340d20`    | `0x800000013f340d28`    |
| Data    | Virtual      | `8192`               | `0x55975cb49000`        | `0x55975cb4b000`        |
|         | Physical     |                      | `0x800000013f340d30`    | `0x800000013f340d38`    |
| BSS     | Virtual      | `4096`               | `0x55975cb4a000`        | `0x55975cb4b000`        |
|         | Physical     |                      | `0x800000013f340d40`    | `0x800000013f340d48`    |
| Heap    | Virtual      | `135168`             | `0x55975e717000`        | `0x55975e738000`        |
|         | Physical     |                      | `0x800000013f340d50`    | `0x800000013f340d58`    |
| Lib     | Virtual      | `70930432`           | `0x7fd240000000`        | `0x7fd247027000`        |
|         | Physical     |                      | `0x800000013f340d60`    | `0x800000013f340d68`    |
| Stack   | Virtual      | `8400896`            | `0x7fd2465ae000`        | `0x7fd246db1000`        |
|         | Physical     |                      | `0x800000013f340d70`    | `0x800000013f340d78`    |


#### Thread-2 start, end address and address size
| Segment | Address Type | Segment Size (bytes) | Start Address           | End Address             |
|---------|--------------|----------------------|-------------------------|-------------------------|
| Code    | Virtual      | `12288`              | `0x55975cb46000`        | `0x55975cb49000`        |
|         | Physical     |                      | `0x800000016e99ad20`    | `0x800000016e99ad28`    |
| Data    | Virtual      | `8192`               | `0x55975cb49000`        | `0x55975cb4b000`        |
|         | Physical     |                      | `0x800000016e99ad30`    | `0x800000016e99ad38`    |
| BSS     | Virtual      | `4096`               | `0x55975cb4a000`        | `0x55975cb4b000`        |
|         | Physical     |                      | `0x800000016e99ad40`    | `0x800000016e99ad48`    |
| Heap    | Virtual      | `135168`             | `0x55975e717000`        | `0x55975e738000`        |
|         | Physical     |                      | `0x800000016e99ad50`    | `0x800000016e99ad58`    |
| Lib     | Virtual      | `70930432`           | `0x7fd240000000`        | `0x7fd247027000`        |
|         | Physical     |                      | `0x800000016e99ad60`    | `0x800000016e99ad68`    |
| Stack   | Virtual      | `8400896`            | `0x7fd245dad000`        | `0x7fd2465ad000`        |
|         | Physical     |                      | `0x800000016e99ad70`    | `0x800000016e99ad78`    |

# **Kernel 編譯過程與碰到的問題**
#### **1. load BTF form vmlinux : Invalid argument**

![image](https://hackmd.io/_uploads/Sk3uPUkI6.png)

vim .config 修改 CONFIG_DEBUG_INFO_BTF Y->N

#### **2. fatal error: linux/compiler-gcc5.h错误**
copy 目前當前使用kernel的compiler-gcc.h去取代編譯kernel的compliler-gcc5.h
reference: 
https://blog.csdn.net/u014525494/article/details/53573298
https://hackmd.io/@eugenechou/H1LGA9AiB

#### **3. fatal error: linux/compiler-gcc5.h错误**
No rule to make target ‘debian/canonical-certs.pem‘, needed by ‘certs/x509_certificate_list‘
解法:修改.config文件，把CONFIG_SYSTEM_TRUSTED_KEYS，改成空值。

#### **4. reboot後Grub沒跳出來**
有三種可能原因 
a.not build finish
b.not setting etc/default/grub
```
開啟/etc/default/grub文件，註解GRUB_HIDDEN_TIMEOUT=0
並把 GRUB_TIMEOUT 設成3，意思是等待3秒。
接著執行指令 sudo update-grub 啟動的時候就會顯示grub選單了
3秒內不選擇，則會自動進入系統。
```
c.not copy source code to /usr/src
reference:https://blog.csdn.net/sfe1012/article/details/53113271


#### **5.[v2p] not mapped in pmd**
我們曾經在部分情況轉換virtual memory到physical memory的時候遇到mapping不到pmd的錯誤。
- call by value -> call by reference會導致mapping不到physical address
- Testing code and syscall need match

#### **6. pthread error**
test.c:(.text+0x138): undefined reference to sem_wait
test.c:(.text+0x253): undefined reference to sem_init
test.c:(.text+0x2b7): undefined reference to pthread_create
test.c:(.text+0x3e3): undefined reference to sem_post
test.c:(.text+0x40b): undefined reference to pthread_join
collect2: error: ld returned 1 exit status
**rootcause** :
在編譯的時候需要加上額外的參數-lpthread，因為
#include <pthread.h> 標頭在Linux預設Import Library中沒有，
需要使用函式庫libpthread.a進行編譯連結。 
**solution** : 
gcc -o file.out file.c -lpthread

# **Reference** 
##### 
https://hackmd.io/@edJH8dxaRPOaOfJQPav3XQ/BkFMEu4ws#Results
https://blog.csdn.net/u014525494/article/details/53573298
https://hackmd.io/@eugenechou/H1LGA9AiB
https://www.csie.ntu.edu.tw/~sprout/algo2016/homework/week3.pdf