## Runtime Security ## & ## Observability --- ## About me https://rewanthtammana.com/ * Independent consultant & SME at Uptycs * Ex Senior Security Architect at Emirates NBD * Containers, Cloud & Kubernetes security * Open source contributor --- ### API server bypass attack Let's look at a tricky scenario. I've reported it to Kubernetes team long back. They said it's a feature, not a bug. If exploited, it can be drastic. Recently, Kubernetes team released official blog post on the same. https://kubernetes.io/docs/concepts/security/api-server-bypass-risks/#static-pods --- ### DNS hijacking  [Reference](https://www.imperva.com/learn/application-security/dns-hijacking-redirection/) --- ### Runtime security Runtime security is often an after thought but its essential to safeguard against real-time attacks & ensure continous protection of sensitive data & infrastructure. --- ### Common threats * Unauthorized access * Privilege escalation * Data breaches & exfiltration * Denial of service attacks - Fork Bomb attack - how identify if limits aren't set? * Malware and ransomware infections * Misconfigurations & vulnerabilities --- ### Observability To secure something, its important for us to observe it in the first place. The more information we have, the better chances of securing systems. --- ### User space Everything that exists on your OS. The files, programs, scripts, etc. Binaries like `ls` run in user space & connects with kernel space to get information. --- ### Kernel space Kernel provides abstraction for security, hardware, etc.  [Reference](https://www.redhat.com/en/blog/architecting-containers-part-1-why-understanding-user-space-vs-kernel-space-matters) --- ### User vs Kernel space   --- ### User vs Kernel space  If you add the values, 0.13 + 0.01 = 0.04. But the total time is 4.205. What happened in the mean time? user - amount of time in user-mode system - amount of time in kernel What else is there in between? --- ### User vs Kernel space  [Reference](https://www.redhat.com/en/blog/architecting-containers-part-1-why-understanding-user-space-vs-kernel-space-matters) --- ### strace `strace ls` `strace -c ls` `strace -e trace=write ls` Lists all the system calls triggered by the process. --- ### Tracing | Trace Type | Description | |---|---| | Tracepoints | Embedded in kernel functions | | eBPF | Lies inside kernel & allows you to run programs | | K(ernel) Probes | Allows to write handlers that can be executed before or after a particular kernel function | | Linux auditing system | Auditd & log file | [Reference](https://www.confluera.com/post/tracing-linux-fast-compatible-complete) --- ### Tracing  [Reference](https://www.confluera.com/post/tracing-linux-fast-compatible-complete) --- ### BPF We won't dive into BPF for this presentation. Berkeley Packet Filter allows users to run a code safely inside the operating system. Originally used for packet filtering but later got extended. The program you write is converted into a format that can be executed in kernel space. --- ### BPF  [Reference](https://www.sobyte.net/post/2023-01/py-bpf/) --- ## BPF BPF is fast, powerful and flexible for deep observability of both Linux kernel & user space. For ex, to count events, stack traces, etc. Observability into user space from kernel space is possible because kernel can control & observe user mode. --- ### BPF Hello World ``` Python code -> C -> BPF bytecode ``` ```bash strace -c ls ``` ```python= from __future__ import print_function from bcc import BPF prog = r""" int helloworld(void *ctx) { bpf_trace_printk("Hello, World!\n"); return 0; } """ bpf = BPF(text=prog) bpf.attach_kprobe(event=bpf.get_syscall_fnname("execve"), fn_name="helloworld") try: bpf.trace_print(fmt="{1} {5}") # bpf.trace_fields except KeyboardInterrupt: exit() ``` --- ### Tetragon * Application level - Function calls & traces * Kernel level - System calls, namespaces, file I/O, network, etc * Real time security - Prevent. If not, detect. --- ### Tetragon * Network traffic * File operations * System activities * Privilege escalations --- ### eBPF  --- ### Tetragon * Tetragon can hook with ebpf programs that looks for things at syscall handling layer. * It supports in kernel events filtering * Running metrics at kernel level is much effective than context switching between kernel & user space --- ### Tetragon * Raw system call visibility and ability to trace arbitary events - tracingpolicy * Ex, look for volume mounts * Identifying malicious practices just with system calls is a nightmare & network monitoring really helps to understand the system & communication --- ### Hands on demo * List dangerous syscalls like `CAP_SYS_ADMIN` * Resolve domain names - `DNS hijacking` * Identify Kubernetes API server bypass attack * Bypass Tetragon prevent check --- ## Follow me https://www.linkedin.com/in/rewanthtammana https://twitter.com/rewanthtammana