# Virtual Address to Physicall Address, Threads Memory Sharing (Linux OS Project 1)
## Environment
- VM Manager: KVM
- VM : Ubuntu 22.04.1
- VM Kernel: Linux 6.2.0-36-generic
## Set Up Kernel
### Install dependencies
```bash
sudo apt install -y \
python3 \
build-essential \ # provide utils for buiding and compiling, including make, compiler...
libncurses-dev \ # provide cli header file for character user interface developement
libssl-dev \ # openssl for encryption/decryption
libelf-dev \ # header file for ELF repos
bison \ # yacc, tool for building semantic analyser
flex \ # lex, tool for building parser
dwarves \ # tools for analysing processes
bc \ # calculator for high precision calculation
cpio \ # similar to tar, tounch/fetch files
zstd \ # efficient compress/decompress tool (Zstandard algorithm)
git \
fakeroot \ # running root commands in fakeroot env
xz-utils \ # efficient compress/decompress tool
```
### Fetch kernel
- fetch Kernel from providers such as: https://cdn.kernel.org/pub/linux/kernel/
```bash
wget https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.15.74.tar.gz # fetch linux-6.2 kernel
tar xvzf ./linux-5.15.74.tar.gz # decompress
```
> recommand a stable version, notice that higher version number doesn't mean newer version
### Kernel config
- two methods to generate .config file
- inherit original kernel config
```bash
cd ./linux-6.2 # enter fetched kernel folder
cp /boot/config-$(uname -r) .config # uname -r gives you your current kernel version
make oldconfig # update new kernel by inheriting old config in .config
# <enter> to skip every setting is fine
```
- generate a new .config
```bash
make menuconfig
# you could just leave the menu once GUI pops up
```
> IMPORTANT!
> you should edit .config where:
> CONFIG_SYSTEM_TRUSTED_KEYS="debian/> canonical-certs.pem"
> ===> CONFIG_SYSTEM_TRUSTED_KEYS = ""
>
> AND
>
> CONFIG_SYSTEM_REVOCATION_KEYS="debian/canonical-revoked-certs.pem"
> ===> CONFIG_SYSTEM_REVOCATION_KEYS=""
>
> if you don't do this, some modules will fail to make since they require keys to build, legend has it spent tons of hours in vain to de this bug
- compile and install kernel
```bash
sudo make -j $(nproc) # j claims the num of logical core to run, nproc returns your total logical cores of your machine
echo $? # if it returns 0 means the build was successful
sudo make modules_install -j $(nproc) # install kernel module
sudo make install -j $(nproc) # install kernel
```
> - sudo make -j 8 tooks me 1.5 hour to build up with Intel Core i7 -8565U
>
> - remember to use sudo or it may occur error
>
> - and KVM used 100% CPU resource of my Kali linux host
## Use Custom Kernel
- It will show on grub menu => Advance Option , then you can choose your kernel
- To show up grub menu every boot, edit /etc/default/grub:
- Set GRUB_TIMEOUT=0 to GRUB_TIMEOUT=5
- Comment GRUB_HIDDEN_TIMEOUT=0 and GRUB_TIMEOUT_STYLE=hidden with #
# Add System call
## architecture:
```bash=
linux_kernel
├─ custom
│ ├─ my_get_physical_address.c # core func
. └─ Makefile # let kernel know how to build up /custom
.
.
```
### custom/Makefile
```bash=
obj-y := my_get_physical_address.o
```
- **/custom** folder stands for a kernel module, which can be dynamic loaded into kernel, giving kernel the availability to extend its funciton
- /custom/Makefile is a config file for building up **/custom** module
- obj-y specifies which source code should be compiled to object file and linked to the module
### /Makefile
```bash=
core-y += custom/
```
- add /custom module to kernel dependencies
## Add to Syscall Table
### /arch/x86/entry/syscalls/syscall_64.tbl
## ASMlinkage (Additional)
### /include/linux/syscalls.h
add at the end of file before #endif
```clike
asmlinkage void* sys_my_get_physical_address(void *vaddr);
```
- the purpose of asmlinkage is to make x64's function compatible with x86's,
- the principle of asmlinkage is to ask C function directyly get parameters from stack instead of registers, since x86 does not read parameters from registers while x64 does. It push registers' values into stack when calling fuctions in x64.
# my_get_physical_address
## Principle:
### page concept
- given virtual address, we can get its physicall address via page table.
- the concept of page table is spliting memory space into **blocks**, and every block has its own index, so we can order a block by telling the kernel "I want Block 5" instead of a verbose address
- but how to locate a specific address in a block? It comes to **offset**. Blockes contains a set of addresses, which can be labeled as number 1, 2, 3.... Order cam be formed as "I want the address of number 4 of Block 2"
- So, a virtual address can be split into
```bash=
+-----------------+-----------------+
| Page Index | Offset |
+-----------------+-----------------+
64 0
```
- above can be though of a 1 level page table
### how about 2 level page table?
```bash=
+--------------+---------------+-----------------+
| PT1_INDEX | PT2_INDEX | Offset |
+--------------+---------------+-----------------+
64 0
```
- every address of PT1 stores the head(base) address of a block of PT2 blocks
- PT1(base) + PT1_INDEX stores the value of PT2's base address, PT2(base)+PT2_INDEX stores the address of a block, and a block + Offset get a physicall address
### n level page table?
- same concept of 2 level page table, but more remapping
A 4 level page table (blue field in the pic is not used)
### structure of 5 level paging
#### source: https://lenovopress.lenovo.com/lp1468.pdf
- Check how many levels of current kernel supports
```bash
$ cat /boot/config-$(uname -r) | grep -E 'X86_5LEVEL|PGTABLE_LEVELS'
CONFIG_PGTABLE_LEVELS=5
CONFIG_X86_5LEVEL=y
```
- relative mocros
```clike
/* /usr/src/linux-5.15.74/include/asm-generic/page.h */
#define PAGE_SHIFT 12
#ifdef __ASSEMBLY__
#define PAGE_SIZE (1 << PAGE_SHIFT)
#else
#define PAGE_SIZE (1UL << PAGE_SHIFT)
#endif
#define PAGE_MASK (~(PAGE_SIZE-1))
```
>[In linux-5.15.74, \_\_ASSEMBLY\_\_ is not defined](https://elixir.bootlin.com/linux/v5.15.74/A/ident/__ASSEMBLY__), so during compilation, PAGE_SIZE is (1UL << PAGE_SHIFT). When implementing a custom system call, unsigned long is used for computation
- relative struct
```c
/* /usr/src/linux-5.15.74/include/asm-generic/page.h */
typedef struct { unsigned long pgd; } pgd_t; // Global Dir
typedef struct { unsigned long pmd[16]; } pmd_t; // Middle Dir
typedef struct { unsigned long pte; } pte_t; // Offset
```
```c
/* /usr/src/linux-5.15.74/include/asm-generic/pgtable-nop4d.h */
#define __PAGETABLE_P4D_FOLDED 1
typedef struct { pgd_t pgd; } p4d_t; // P4 Dir
```
```c
/* /usr/src/linux-5.15.74/include/asm-generic/pgtable-nopud.h */
#define __PAGETABLE_PUD_FOLDED 1
typedef struct { p4d_t p4d; } pud_t; // Upper Dir
```
- currently we have 5 level page table, which add p4d between PGD and PUD
## Code
### header
```c
#include <linux/kernel.h>
#include <linux/syscalls.h>
```
- kernel dependencies,
- when define a syscall with SYSCALL_DEFINE1, it uses linux/syscalls.h
- when some insturctions like current->mm, pgd_offset, pgd_val, pgd_index ... , it uses linux/kernel.h
### define
```c
SYSCALL_DEFINE1(my_get_physical_address, unsigned long, vaddr)
{
return vaddr2paddr(vaddr);
}
```
- SYSCLL_DEFINE{variable num} is a macro defined in syscalls.h
- first parameter is the name of this syscall
- second and third parameter defines a vairable, corresponding to type, variable name
- Digging into SYSCALL_DEFINEx, we notice that it add **asmlinkage** to syscalls, so we can skip the step of adding asmlinkage in /include/linux/syscalls.h mentioned previously.
>```cpp=
>#define __SYSCALL_DEFINEx(x, name, ...) \
> asmlinkage long sys##name(__MAP(x,__SC_DECL,__VA_ARGS__)) \
> __attribute__((alias(__stringify(SyS##name)))); \
> static inline long SYSC##name(__MAP(x,__SC_DECL,__VA_ARGS__)); \
> asmlinkage long SyS##name(__MAP(x,__SC_LONG,__VA_ARGS__)); \
> asmlinkage long SyS##name(__MAP(x,__SC_LONG,__VA_ARGS__)) \
> { \
> long ret = SYSC##name(__MAP(x,__SC_CAST,__VA_ARGS__)); \
> __MAP(x,__SC_TEST,__VA_ARGS__); \
> __PROTECT(x, ret,__MAP(x,__SC_ARGS,__VA_ARGS__)); \
> return ret; \
> } \
> static inline long SYSC##name(__MAP(x,__SC_DECL,__VA_ARGS__))
>
### variable
```c
pgd_t *pgd;
p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
pte_t *pte;
unsigned long paddr = 0;
unsigned long page_addr = 0;
unsigned long page_offset = 0;
```
- using unsigned long to store address, honestly long type is enough cos long has 8 bytes = 64 bits, it can be found that syscalls usually returns in long type(if negative values means fail), but since addresses here wouldn't be negative so unsigned long.
- (page_addr) concat (page_offset) = paddr(physical address)
### get Page Content
```c
pgd = pgd_offset(current->mm, vaddr);
printk("[v2p] pgd_val = 0x%lx\n", pgd_val(*pgd));
printk("[v2p] pgd_index = %lu\n", pgd_index(vaddr));
if (pgd_none(*pgd)) {
printk("[v2p] not mapped in pgd\n");
return -1;
}
```
- pgd_offset is function in kernel to retrieve address in pgd table, which has the value of pud base address
- it calculated by get pgd table base address from current-mm (current process's memory mapping), and get pgd index from first few bits of virtual address(it depends on page table size)
- notice that if the process doesn't not use that much tables, for example, **pgd -> pmd -> pte**, it can still get **p4d** and **pud** but the value of **pud** and **p4d** is the same as **pmd** , **pgd->p4d=pud=pmd->pte**
- if the address is not mapped, it will return -1
- the output of printk can be found in /var/log/syslog
## source code
```c
#include <linux/kernel.h>
#include <linux/syscalls.h>
static unsigned long vaddr2(unsigned long vaddr)
{
pgd_t *pgd;
p4d_t *p4d;
pud_t *pud;
pmd_t *pmd;
pte_t *pte;
unsigned long paddr = 0;
unsigned long page_addr = 0;
unsigned long page_offset = 0;
pgd = pgd_offset(current->mm, vaddr);
printk("[v2p] pgd_val = 0x%lx\n", pgd_val(*pgd));
printk("[v2p] pgd_index = %lu\n", pgd_index(vaddr));
if (pgd_none(*pgd)) {
printk("[v2p] not mapped in pgd\n");
return NULL;
}
p4d = p4d_offset(pgd, vaddr);
printk("[v2p] p4d_val = 0x%lx\n", p4d_val(*p4d));
printk("[v2p] p4d_index = %lu\n", p4d_index(vaddr));
if (p4d_none(*p4d)) {
printk("[v2p] not mapped in p4d\n");
return NULL;
}
pud = pud_offset(p4d, vaddr);
printk("[v2p] pud_val = 0x%lx\n", pud_val(*pud));
printk("[v2p] pud_index = %lu\n", pud_index(vaddr));
if (pud_none(*pud)) {
printk("[v2p] not mapped in pud\n");
return NULL;
}
pmd = pmd_offset(pud, vaddr);
printk("[v2p] pmd_val = 0x%lx\n", pmd_val(*pmd));
printk("[v2p] pmd_index = %lu\n", pmd_index(vaddr));
if (pmd_none(*pmd)) {
printk("[v2p] not mapped in pmd\n");
return NULL;
}
pte = pte_offset_kernel(pmd, vaddr);
printk("[v2p] pte_val = 0x%lx\n", pte_val(*pte));
printk("[v2p] pte_index = %lu\n", pte_index(vaddr));
if (pte_none(*pte)) {
printk("[v2p] not mapped in pte\n");
return NULL;
}
/* Page frame physical address mechanism | offset */
page_addr = pte_val(*pte) & PAGE_MASK;
page_offset = vaddr & ~PAGE_MASK;
paddr = page_addr | page_offset;
printk("[v2p] page_addr = %lx, page_offset = %lx\n", page_addr, page_offset);
printk("[v2p] vaddr = %lx, paddr = %lx\n", vaddr, paddr);
return paddr;
}
SYSCALL_DEFINE1(my_get_physical_address, unsigned long, vaddr)
{
return vaddr2paddr(vaddr);
}
```
## test the validity of each level of the page table
- The source code has definitions for these types in multiple files. According to the code navigation, p4d_t and pud_t come from pgtable-nop4d.h and pgtable-nopud.h respectively, and both __PAGETABLE_P4D_FOLDED and __PAGETABLE_PUD_FOLDED are set to 1
```c
/* v2p_dmesg.c */
#include <stdio.h>
#include <pthread.h>
#include <string.h>
#include <sys/syscall.h> /* Definition of SYS_* constants */
#include <unistd.h>
// extern int main();
void * my_get_physical_addresses(void *vaddr_p) {
unsigned long t = syscall(450, vaddr_p);
return t;
}
int main() {
int i = 0;
printf("%p\n", my_get_physical_addresses(&i));
return 0;
}
```
```bash=
# Result of dmesg
[ 147.123348] [v2p] pgd_val = 0x8000000074ecc067
[ 147.123352] [v2p] pgd_index = 255
[ 147.123368] [v2p] p4d_val = 0x8000000074ecc067
[ 147.123368] [v2p] p4d_index = 0
[ 147.123369] [v2p] pud_val = 0x6353d067
[ 147.123369] [v2p] pud_index = 496
[ 147.123369] [v2p] pmd_val = 0x74c3d067
[ 147.123370] [v2p] pmd_index = 140
[ 147.123370] [v2p] pte_val = 0x8000000082a02867
[ 147.123371] [v2p] pte_index = 309
[ 147.123371] [v2p] page_addr = 8000000082a02000, page_offset = 3f4
[ 147.123371] [v2p] vaddr = 7ffc119353f4, paddr = 8000000082a023f4
```
- **p4d_val** and **pgd_val** are consistent, **p4d_index** equals 0
- P4D is currently not in use, it exists only for compatibility
## test correctness
```c=
#include <stdio.h>
#include <sys/syscall.h>
#include <unistd.h>
int a = 123;
int main(){
int b = 321;
long pb = syscall(449, &b);
long pa = syscall(449, &a);
printf("vaddr of a: %p\n", &a);
printf("paddr of a: 0x%lx\n", pa);
printf("vaddr of b: %p\n", &b);
printf("paddr of b: 0x%lx\n", pb);
return 0;
}
```
- Successfully get our desired result
- a (0x555555558010) is a initialized global variable, as we can see from vmmap, it truly lies in data segment
```text
0x555555558000 0x555555559000 rw-p 1000 3000 /home/jasoff/demo/a.out
```
- b (0x7fffffffdf24) is a initialized local variable in fuction, which should and it does be in stack segment
```text
0x7ffffffde000 0x7ffffffff000 rw-p 21000 0 [stack]
```
> ps: remember to include not only syscalls.h but unistd.h, unistd.h also defines some utils to syscall
---
# result
## example code:
```clike=
#include <stdio.h>
#include <pthread.h>
#include <string.h>
#include <sys/syscall.h> /* Definition of SYS_* constants */
#include <unistd.h>
extern void *func1(void *);
extern void *func2(void *);
extern int main();
//void * my_get_physical_addresses(void *); // your syscall
struct data_
{
int id;
char name[16];
};
typedef struct data_ sdata;
static __thread sdata tx; //thread local variable
int a = 123; //global variable
void hello(int pid)
{
int b = 10; //local varialbe
b = b + pid;
//global variable
printf("In thread %d \nthe value of gloable varialbe a is %d, the offset of the logical address of a is %p, ", pid, a, &a);
printf("the physical address of global variable a is %p\n", my_get_physical_addresses(&a));
//local variable
printf("the value of local varialbe b is %d, the offset of the logical address of b is %p, ", b, &b);
printf("the physical address of local variable b is %p\n", my_get_physical_addresses(&b));
//thread local variable
printf("the offset of the logical address of thread local varialbe tx is %p, ", &tx);
printf("the physical address of thread local variable tx is %p\n", my_get_physical_addresses(&tx));
//function
printf("the offset of the logical address of function hello is %p, ", hello);
printf("the physical address of function hello is %p\n", my_get_physical_addresses(hello));
printf("the offset of the logical address of function func1 is %p, ", func1);
printf("the physical address of function func1 is %p\n", my_get_physical_addresses(func1));
printf("the offset of the logical address of function func2 is %p, ", func2);
printf("the physical address of function func2 is %p\n", my_get_physical_addresses(func2));
printf("the offset of the logical address of function main is %p, ", main);
printf("the physical address of function main is %p\n", my_get_physical_addresses(main));
//library function
printf("the offset of the logical address of library function printf is %p, ", printf);
printf("the physical address of library function printf is %p\n", my_get_physical_addresses(printf));
printf("====================================================================================================================\n");
}
void *func1(void *arg)
{
char *p = (char *)arg;
int pid;
pid = syscall(__NR_gettid);
tx.id = pid;
strcpy(tx.name, p);
printf("I am thread with ID %d executing func1().\n", pid);
hello(pid);
while (1)
{
//printf("(%d)(%s)\n",tx.id,tx.name) ;
sleep(1);
}
}
void *func2(void *arg)
{
char *p = (char *)arg;
int pid;
pid = syscall(__NR_gettid);
tx.id = pid;
strcpy(tx.name, p);
printf("I am thread with ID %d executing func2().\n", pid);
hello(pid);
while (1)
{
//printf("(%d)(%s)\n",tx.id,tx.name) ;
sleep(2);
}
}
int main()
{
pthread_t id[2];
char p[2][16];
strcpy(p[0], "Thread1");
pthread_create(&id[0], NULL, func1, (void *)p[0]);
strcpy(p[1], "Thread2");
pthread_create(&id[1], NULL, func2, (void *)p[1]);
int pid;
pid = syscall(__NR_gettid);
tx.id = pid;
strcpy(tx.name, "MAIN");
printf("I am main thread with ID %d.\n", pid);
hello(pid);
while (1)
{
//printf("(%d)(%s)\n",tx.id,tx.name) ;
sleep(5);
}
}
```
## example code result:
A little messy...
## example code(labeled) result:
- it run simultaneously, so the output will not follow the order of main, thread 1, tread 2, and the result differs every time.
- the process flow can be control by **mutex lock**
### mutual lock
```c
pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
```
- get a usable mutex lock
```c
pthread_mutex_lock(&mutex);
```
- check if the mutex is locked ? wait til it unlocked
- once it unlocked, get it, and lock it
```c
pthread_mutex_lock(&mutex);
```
- unlocked the mutex
```c
pthread_join(thread, NULL);
```
- wait until the thread finished
```c
pthread_mutex_destroy(&mutex);
```
- make mutex lock uninitialed(not usable)
#### concept
1. let main have the mutex
2. main creates thread 1, who will wait for the lock
3. main creates thread 2, who will also wait for the lock
4. main doing stuff
5. main unlock
6. main wait for thread1, thread2
7. main destroy the mutex
## example code(with mutex lock) result
## example code(with mutex lock & formatting) result
Prefect :D
## custom heap check
### track the memory address allocated by malloc
### overview
### main thread (0x55555555a510)(at heap)
### thread 1 (0x7ffff0000b70)(at anon_7ffff0000)(violate)
### thread 2 (0x7ffff0000b90)(at anon_7ffff0000)(violate)
### Rule Violation!!!!
- we know that threads will share heap segment, but the test above show that threads' malloc were asigned to an anonymous segment next to heap
### Dig in Deeper
- Clue1: by setting more break points in pwndbg, we notice that, the anonymous reflection area would not spawn until start calling thread
- Clue2: by erase the malloc related commands, the anonymous segment vanished
### it's about MALLOC
- We studied about malloc's mechanism, a suspect named **mmap** occurs.
- Normally, malloc uses sbrk,brk to allocat spaces in heap segment. But if the requested space larger than mmap_threshold(128 KB), malloc will turn to use mmap, and mmap can reflect addresses by **anonymous reflection**
- Dejavu??
- but what we malloc is just a size of char, it should not use mmap...
### Weird mmap
- Found few related studies on the net, we tried to run another malloc allocator called jemalloc, but nothing changed
### Fuck You GLIBC
- The answer is, **glibc's malloc uses multiple malloc "arenas", and gives each new thread its own arena, no matter what you set M_MMAP_MAX to**
- glibc ignore mmap threshold, and allocat threads their own arena with mutiple malloc arenas
- to avoid this circustance,
```cpp
#include <malloc.h>
int main(){
mallopt(M_ARENA_MAX, 1);
}
```
## custom heap result
### overview
### main thread (0x55555555a510)(at heap)
### thread 1 (0x55555555a7c0)(at heap)
### thread 2 (0x55555555a7e0)(at heap)
# Memory Layout
## vm_area_struct(VMA), mm_struct, segment
(task_struct.mm -> mm_struct.mmap -> vm_area_struct)
- task_struct: all info about the process
- mm_struct: all info about the process's space
- VMA: info about the chunck made of virtual addresses
- Segments is composed of vm_area_struct, therefore, we have to add another syscalls (or in tricky way, echo '/proc/self/maps')
## Syscalls to Find Segment
- here is a script on the net, let's, dig it deeper:
```c
#include <linux/kernel.h>
#include <linux/syscalls.h>
struct Segment
{
unsigned long start;
unsigned long end;
};
struct Segments
{
struct Segment text;
struct Segment data;
struct Segment bss;
struct Segment heap;
struct Segment mm;
struct Segment stack;
unsigned long start_code;
unsigned long end_code;
unsigned long start_data;
unsigned long end_data;
unsigned long start_brk;
unsigned long brk;
unsigned long mmap_base;
unsigned long start_stack;
};
void find_segment(struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, struct Segment *segment, char include_start)
{
char continuous = 0;
char is_start_in_range;
while (vma) {
is_start_in_range = include_start ? vma->vm_end >= start_addr : vma->vm_end > start_addr;
if (!continuous && is_start_in_range) {
segment->start = vma->vm_start;
continuous = 1;
}
if (continuous && vma->vm_end >= end_addr) {
segment->end = vma->vm_end;
break;
}
vma = vma->vm_next;
}
}
SYSCALL_DEFINE2(get_segments, unsigned long, start_stack, void * __user, addr)
{
struct Segments segments;
struct mm_struct *mm = current->mm;
start_stack = start_stack ? start_stack : mm->start_stack;
segments.start_code = mm->start_code;
segments.end_code = mm->end_code;
segments.start_data = mm->start_data;
segments.end_data = mm->end_data;
segments.start_brk = mm->start_brk;
segments.brk = mm->brk;
segments.mmap_base = mm->mmap_base;
segments.start_stack = start_stack;
find_segment(mm->mmap, mm->start_data, mm->end_data, &segments.data, 1);
find_segment(mm->mmap, mm->start_code, segments.data.start, &segments.text, 1);
find_segment(mm->mmap, mm->end_data, mm->end_data, &segments.bss, 1);
find_segment(mm->mmap, mm->start_brk, mm->brk, &segments.heap, 0);
find_segment(mm->mmap, segments.heap.end, mm->mmap_base, &segments.mm, 0);
find_segment(mm->mmap, start_stack, start_stack, &segments.stack, 0);
return copy_to_user(addr, &segments, sizeof(struct Segments));
}
```
1.
```c
struct mm_struct *mm = current->mm;
```
- it first get mm_struct of current process
- **current** is a pointer to current process, defined in **arch/x86/include/asm/current.h**
2.
```c
void find_segment(struct vm_area_struct *vma, unsigned long start_addr, unsigned long end_addr, struct Segment *segment, char include_start)
{
char continuous = 0;
char is_start_in_range;
while (vma) {
is_start_in_range = include_start ? vma->vm_end >= start_addr : vma->vm_end > start_addr;
if (!continuous && is_start_in_range) {
segment->start = vma->vm_start;
continuous = 1;
}
if (continuous && vma->vm_end >= end_addr) {
segment->end = vma->vm_end;
break;
}
vma = vma->vm_next;
}
}
```
- by providing the start and the end address of the segment(provided by mm_struct), it search the vma area that can fully contain these two.
- **WARNING!!** It works fine, but if the end segment address provided by mm_struct exceeds its VMA end segment, it will include two VMA. That's why this script getting DATA segment covered with BSS segment, cos the mm->end_data is not the same as the end of DATA's vma. We'll talk about it later.
## MMAP
- segment between heap and stack
- grows like stack(from high address to low address)
- be used to reflect a part of a file (cos we can't load whole of file into memory), therefore linking library can be found here
- if malloc a huge size of space (>128KB), the allocator will turn to use mmap
- Threads' stacks are allocated to MMAP
## Code to Show Memory Layout
```c
#include <malloc.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
extern char __executable_start[];
extern char __etext[];
extern char __data_start[];
extern char _edata[];
extern char __bss_start[];
extern char _end[];
pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
const char k1 = 'a';
char var1 = 'a';
char var2;
static __thread char stx; // thread local variable
struct Segment {
unsigned long start;
unsigned long end;
};
struct Segments {
struct Segment text;
struct Segment data;
struct Segment bss;
struct Segment heap;
struct Segment mm;
struct Segment stack;
unsigned long start_code;
unsigned long end_code;
unsigned long start_data;
unsigned long end_data;
unsigned long start_brk;
unsigned long brk;
unsigned long mmap_base;
unsigned long start_stack;
};
unsigned long v2p(unsigned long vaddr) {
return syscall(601, vaddr); // sys_v2p
}
void print_info(void *start_stack) {
struct Segments segments;
void *dynamic = malloc(sizeof(char));
syscall(600, (unsigned long)start_stack, &segments); // sys_get_segments
if (segments.stack.end < segments.mm.end) {
// thread's stack
printf("-----------------\n");
printf("| |\n");
printf("| STACK(main's) |\n");
printf("| |\n");
printf("-----------------\n");
printf("| |\n");
printf("----------------- VMA:0x%lx(0x%lx)\n", segments.mm.end,
v2p(segments.mm.end));
printf("| |\n");
printf("| | -library function(printf):%p(%lx)\n",
printf, v2p((unsigned long)printf));
printf("| MMAP |\n");
printf("| | |thread stack end: 0x%lx(0x%lx)\n",
segments.stack.end, v2p(segments.stack.end));
printf("| |\n");
printf("| | -__thread var: "
"%p(0x%lx)\n",
&stx, v2p((unsigned long)&stx));
printf("| |\n");
printf("| |\n");
printf("| |\n");
printf("| | |thread stack start: 0x%lx(0x%lx)\n",
segments.stack.start, v2p(segments.stack.start));
printf("| |\n");
printf("----------------- VMA:0x%lx(0x%lx)\n", segments.mm.start,
v2p(segments.mm.start));
} else {
printf("----------------- VMA:0x%lx(0x%lx)\n", segments.stack.end,
v2p(segments.stack.end));
printf("| |\n");
printf("| STACK(main's) |\n");
printf("| |\n");
printf("----------------- VMA:0x%lx(0x%lx)\n", segments.stack.start,
v2p(segments.stack.start));
printf("| |\n");
printf("----------------- VMA:0x%lx(0x%lx)\n", segments.mm.end,
v2p(segments.mm.end));
printf("| |\n");
printf("| | -library function(printf):%p(%lx)\n",
printf, v2p((unsigned long)printf));
printf("| MMAP |\n");
printf("| | -__thread var: "
"%p(0x%lx)\n",
&stx, v2p((unsigned long)&stx));
printf("| |\n");
printf("| |\n");
printf("----------------- VMA:0x%lx(0x%lx)\n", segments.mm.start,
v2p(segments.mm.start));
}
printf("| |\n");
printf("| |\n");
printf("----------------- VMA:0x%lx(0x%lx)\n", segments.heap.end,
v2p(segments.heap.end));
printf("| | |brk_end: 0x%lx(0x%lx)\n", segments.brk,
v2p(segments.brk));
printf("| |\n");
printf("| |\n");
printf("| HEAP |\n");
printf("| |\n");
printf("| | -malloc var: %p(0x%lx)\n", dynamic,
v2p((unsigned long)dynamic));
printf("| |\n");
printf("| | |brk_start: 0x%lx(0x%lx)\n",
segments.start_brk, v2p(segments.start_brk));
printf("----------------- VMA:0x%lx(0x%lx)\n", segments.bss.end,
v2p(segments.bss.end));
printf("| |\n");
printf("| BSS(DATA) |\n");
printf("| | -uninitialized global variable: "
"%p(0x%lx)\n",
&var2, v2p((unsigned long)&var2));
printf("| |\n");
printf("| | |data_end: 0x%lx(0x%lx)\n",
segments.end_data, v2p(segments.end_data));
printf("| | -initialized global variable: "
"%p(0x%lx)\n",
&var1, v2p((unsigned long)&var1));
printf("----------------- VMA:0x%lx(0x%lx)\n", segments.bss.start,
v2p(segments.bss.start));
printf("| |\n");
printf("| DATA |\n");
printf("| |\n");
printf("| | |data_start: 0x%lx(0x%lx)\n",
segments.start_data, v2p(segments.start_data));
printf("| |\n");
printf("----------------- VMA:0x%lx(0x%lx)\n", segments.text.end,
v2p(segments.text.end));
printf("| |\n");
printf("| | -const global variable: %p(0x%lx)\n",
&k1, v2p((unsigned long)&k1));
printf("| |\n");
printf("| | |code_end: 0x%lx(0x%lx)\n",
segments.end_code, v2p(segments.end_code));
printf("| |\n");
printf("| TEXT |\n");
printf("| |\n");
printf("| | -code function: %p(0x%lx)\n",
print_info, v2p((unsigned long)print_info));
printf("| |\n");
printf("| | |code_start: 0x%lx(0x%lx)\n",
segments.start_code, v2p(segments.start_code));
printf("| |\n");
printf("----------------- VMA:0x%lx(0x%lx)\n", segments.text.start,
v2p(segments.text.start));
printf("\n");
}
void *thread_fn(void *name) {
pthread_mutex_lock(&mutex);
pthread_attr_t attr;
void *stack_addr;
size_t stack_size;
printf("\n========== %s ==========\n", (char *)name);
pthread_getattr_np(pthread_self(), &attr);
pthread_attr_getstack(&attr, &stack_addr, &stack_size);
print_info(stack_addr);
pthread_mutex_unlock(&mutex);
return NULL;
}
int main() {
mallopt(M_ARENA_MAX, 1);
pthread_t t1;
pthread_t t2;
pthread_mutex_lock(&mutex);
pthread_create(&t1, NULL, thread_fn, (void *)"Thread-1");
pthread_create(&t2, NULL, thread_fn, (void *)"Thread-2");
printf("========== Main Thread ==========\n");
print_info(NULL);
pthread_mutex_unlock(&mutex);
pthread_join(t1, NULL);
pthread_join(t2, NULL);
pthread_mutex_destroy(&mutex);
return 0;
}
```
## RESULT
### main thread
### thread 1
### thread 2
### virtual memory address
- const global variable in TEXT segment but after mm->end_code
- intialized global variable at the beginning of mm->start_data
- Library at MMAP segment
- Thread stack at MMAP segment
- Thread local storages also at MMAP segment
## Threads' memory sharing
- They share TEXT, DATA, BSS, HEAP, MMAP
- They only have their independent STACK (threads' stack are in MMAP after Library)
# Troubleshooting
## mm->end_data is not as same as DATA's vm_area_struct.end
- mm->end_data = _edata
- _edata is the symbol representing end of DATA labeled by **linker**
- process of linking:
1. after compiling, source code is turned into object file, including text segment and data segment
2. When linking, linker will handle with symbol table, and calculate the start and end of segments, labeling them with _sdata, _edata...
3. When executing, memory descriptor will be set, ex. mm->end_data = _edata
4. variables which should be put in DATA segment will follow mm->end_data
(linker script dealing .data)
- variable will follow mm->end_data, that's why var1 belows data_end and out of DATA segment
- when **objdump -h a.out**
- linker labeled .data with size of 0x11, In coincidence, mm->end_data is just 0x11 more than DATA's vm_area_struct.end
- highly doubt that the issue comes from the linker
### experiment
#### 1 initialized global variable
- the linker set the size of .data 0x10 + 1 (=0X11)
- the mm->end_data is 0x10 + 1 (=0X11) far away than VMA(DATA)
- VMA(DATA) remains the size of 0x1000
- VMA(BSS) remains the size of 0x1000, but its available space is squeezed to 0x1000 - 0x11
#### 10 initialized global variables
- the linker set the size of .data 0x10 + 10 (=0X1a)
- the mm->end_data is 0x10 + 10 (=0X1a) far away than VMA(DATA)
- VMA(DATA) remains the size of 0x1000
- VMA(BSS) remains the size of 0x1000, but its available space is squeezed to 0x1000 - 0x1a
#### 4096 initialized global variables
- the linker set the size of .data 0x20 + 4096 (=0X1020)
- the mm->end_data is 0x20 + 4096 (=0X1020) far away than VMA(DATA)
- VMA(DATA) remains the size of 0x1000
- VMA(BSS) extends its size to 0x1000 + 4096 (=0x2000), since the initialized global varialbes fills up its original area
#### 40960 initialized global variables
- the linker set the size of .data 0x20 + 40960 (=0Xa020)
- the mm->end_data is 0x20 + 40960 (=0Xa020) far away than VMA(DATA)
- VMA(DATA) remains the size of 0x1000
- VMA(BSS) extends its size to 0x1000 + 40960 (=0xb000), since the initialized global varialbes fills up its original area
# Reference
## Compile Kernel
- https://meetonfriday.com/posts/5523c739/
### syscall:
- https://blog.csdn.net/rikeyone/article/details/91047118
https://www.twblogs.net/a/5b7df5dd2b7177683854947e
## 5 level page:
- https://tinylab.org/lwn-717293/
- https://blog.csdn.net/forDreamYue/article/details/78887035
- https://hackmd.io/@LJP/rkxtGgoIO
- https://blog.csdn.net/weixin_38537730/article/details/104252794
## thread mechanism:
- https://stackoverflow.com/questions/10706466/how-does-malloc-work-in-a-multithreaded-environment
## malloc:
- https://www.cnblogs.com/LoyenWang/p/12037658.html
- https://zhuanlan.zhihu.com/p/645312749
- https://stackoverflow.com/questions/41178216/how-to-turn-off-mmap-usage-for-malloc-in-multithread
## VMA
- https://zhuanlan.zhihu.com/p/567231858
- https://zhuanlan.zhihu.com/p/592342280
## mm->end_data, _edata
- http://wen00072.github.io/blog/2014/12/22/rtenv-linker-script-explained/
- https://blog.csdn.net/jasonactions/article/details/111753936
Google Drive
Gist
Clipboard
Sign in with Wallet
