System call protection study

tags: security system call

Approach study: call stack integrety

On the meeting at 1st March, this approach was proposed. Therefore, we are gonna look into the possibility and limitation of this approach.

Core idea

The core idea behind this method is to verify the call stack integrety, and determine if the process is compromised.

To verify the call stack integrety, the approach states that given a call stack, if we can perform backtrace back to the entry point (_start in crt0.o), this call stack is sound

Design

1. for all programs in the system, we assume they are all compiled with gnu toolchain without bringing flags like: -fno-unwind-tables (see)
2. upon a process calls a sensitive system call, the mechanism is enforced
3. retreive the call stack, and use library, like libunwind to perform backtrace
4. if we can backtrace back to the entry point, we consider the process is uncompromised, and give this system call permission

• backtrace to _start states the integrety is not compromised
  

• a ROP chain wreck the call stack, therefore we cannot backtrace to _start
  

Pros and Cons

Pros

• quick for implementation
  • backtrace can invoke libunwind
  • don't need to generate more information during compilation

Cons

• no efficient for runtime: O(N) for each system call
  • N: the number of the call frames
• hight false negative rate
  • this approach based on some assumptions, invalidated if any one of them not held:
    • The ROP chain definitely compromises the call stack
    • programs always have .en_frame section
  • cannot deal with attack methods like, dll injection, on which the backtrace can still be performed back to the entry point
• similiar to stack canary
• exception handler (setjump, jump)