# System call protection study
###### tags: `security` `system call`

## Approach study: call stack integrety

On the meeting at 1st March, this approach was proposed. Therefore, we are gonna look into the possibility and limitation of this approach.

## Core idea
The core idea behind this method is to **verify the call stack integrety**, and determine if the process is compromised.

To verify the call stack integrety, the approach states that given a call stack, if we can perform **backtrace back to the entry point** (`_start` in `crt0.o`), **this call stack is sound**

## Design

1. for all programs in the system, we **assume they are all compiled with gnu toolchain without bringing flags like: `-fno-unwind-tables`** ([see](https://gcc.gnu.org/onlinedocs/gcc-4.2.4/gcc/Code-Gen-Options.html))
2. upon a process calls a sensitive system call, the mechanism is enforced
3. retreive the call stack, and use library, like [libunwind](https://github.com/libunwind/libunwind) to perform backtrace
4. if we can backtrace back to the entry point, we consider the process is uncompromised, and give this system call permission

- [ ] backtrace to `_start` states the integrety is not compromised

- [ ] a ROP chain wreck the call stack, therefore we cannot backtrace to `_start`

## Pros and Cons

### Pros
* quick for implementation
* backtrace can invoke [libunwind](https://github.com/libunwind/libunwind)
* don't need to generate more information during compilation

### Cons
* no efficient for runtime: O(N) for each system call
* N: the number of the call frames
* hight false negative rate
* this approach based on some assumptions, invalidated if any one of them not held:
* The ROP chain definitely compromises the call stack
* programs always have `.en_frame` section
* cannot deal with attack methods like, dll injection, on which the backtrace can still be performed back to the entry point
* similiar to [stack canary](https://developers.redhat.com/articles/2022/06/02/use-compiler-flags-stack-protection-gcc-and-clang#stack_canary)
* exception handler (setjump, jump)