Falco Installation # Analyzing Container Behavior - Falco Installation Date: Nov 2023 OS: Ubuntu 22.04 Kubernetes: 1.26.4 Containerd: 1.6.24 Calico: 3.26.3 Node: 1 x control + 2 x workers Falco version: 0.36.2 **Lab setup:** ``` cloud_user@k8s-control:~$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s-control Ready control-plane 3d9h v1.26.4 172.31.22.65 <none> Ubuntu 22.04.3 LTS 6.2.0-1013-aws containerd://1.6.24 k8s-worker1 Ready <none> 3d8h v1.26.4 172.31.21.187 <none> Ubuntu 22.04.3 LTS 6.2.0-1009-aws containerd://1.6.24 k8s-worker2 Ready <none> 3d8h v1.26.4 172.31.25.96 <none> Ubuntu 22.04.3 LTS 6.2.0-1013-aws containerd://1.6.24 ``` ``` cloud_user@k8s-control:~$ kubectl version --short Flag --short has been deprecated, and will be removed in the future. The --short output will become the default. Client Version: v1.26.4 Kustomize Version: v4.5.7 Server Version: v1.26.4 ``` ``` cloud_user@k8s-control:~$ kubelet --version Kubernetes v1.26.4 ``` ``` cloud_user@k8s-control:~$ kubeadm version kubeadm version: &version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.4", GitCommit:"f89670c3aa4059d6999cb42e23ccb4f0b9a03979", GitTreeState:"clean", BuildDate:"2023-04-12T12:12:17Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"} ``` **Falco Installation Steps:** Reference: https://falco.org/docs/install-operate/installation/ On all the worker nodes, install Falco 1. Trust the falcosecurity GPG key ``` curl -fsSL https://falco.org/repo/falcosecurity-packages.asc | \ sudo gpg --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg ``` 2. Configure the apt repository ``` echo "deb [signed-by=/usr/share/keyrings/falco-archive-keyring.gpg] https://download.falco.org/packages/deb stable main" | \ sudo tee -a /etc/apt/sources.list.d/falcosecurity.list ``` 3. Update the package list sudo apt-get update -y 4. Install some required dependencies that are needed to build the kernel module and the BPF probe. I will use "Kmod" driver later ``` sudo apt install -y dkms make linux-headers-$(uname -r) # If you use the falco-driver-loader to build the BPF probe locally you need also clang toolchain sudo apt install -y clang llvm # You can install also the dialog package if you want it sudo apt install -y dialog ``` 5. Install the Falco package ``` sudo apt-get install -y falco ``` 6. Installation with dialog The dialog will be prompted only if you install "dialog binary" . - Choose "2. Kmod" ![image](https://hackmd.io/_uploads/S1TwyFh4T.png) - Choose "1. Yes" ![image](https://hackmd.io/_uploads/HyUO1F3Ea.png) the output should be similar as below ``` [POST-INSTALL] Trigger deamon-reload: [POST-INSTALL] Call 'falco-driver-loader --compile module': * Running falco-driver-loader for: falco version=0.36.2, driver version=6.0.1+driver, arch=x86_64, kernel release=6.2.0-1014-aws, kernel version=14~22.04.1 * Running falco-driver-loader with: driver=module, compile=yes, download=no ================ Cleaning phase ================ * 1. Check if kernel module 'falco' is still loaded: - OK! There is no 'falco' module loaded. * 2. Check all versions of kernel module 'falco' in dkms: - OK! There are no 'falco' module versions in dkms. [SUCCESS] Cleaning phase correctly terminated. ================ Cleaning phase ================ * Looking for a falco module locally (kernel 6.2.0-1014-aws) * Filename 'falco_ubuntu-aws_6.2.0-1014-aws_14~22.04.1.ko' is composed of: - driver name: falco - target identifier: ubuntu-aws - kernel release: 6.2.0-1014-aws - kernel version: 14~22.04.1 install: /usr/lib/gcc/x86_64-linux-gnu/11/ * Trying to dkms install falco module with GCC /usr/bin/gcc DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" Creating symlink /var/lib/dkms/falco/6.0.1+driver/source -> /usr/src/falco-6.0.1+driver Kernel preparation unnecessary for this kernel. Skipping... Building module: cleaning build area... '/tmp/falco-dkms-make'........... Signing module: - /var/lib/dkms/falco/6.0.1+driver/6.2.0-1014-aws/x86_64/module/falco.ko cleaning build area... falco.ko: Running module version sanity check. - Original module - No original module exists within this kernel - Installation - Installing to /lib/modules/6.2.0-1014-aws/updates/dkms/ depmod............ * falco module installed in dkms * falco module found: /var/lib/dkms/falco/6.0.1+driver/6.2.0-1014-aws/x86_64/module/falco.ko * Trying to insmod * Success: falco module found and loaded in dkms [POST-INSTALL] Enable 'falco-kmod.service': Created symlink /etc/systemd/system/falco.service → /lib/systemd/system/falco-kmod.service. Created symlink /etc/systemd/system/multi-user.target.wants/falco-kmod.service → /lib/systemd/system/falco-kmod.service. [POST-INSTALL] Start 'falco-kmod.service': Scanning processes... Scanning linux images... Running kernel seems to be up-to-date. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host. ``` **Use Falco to detect process** On the Control node, create a test pod ``` vi falco-test-pod.yml apiVersion: v1 kind: Pod metadata: name: falco-test-pod spec: nodeName: k8s-worker1 containers: - name: falco-test image: busybox:1.33.1 command: ['sh', '-c', 'while true; do echo "Running..."; sleep 5; done'] kubectl create -f falco-test-pod.yml ``` On k8s-worker1, create a Falco rules `vi falco-rules.yml` Create a rule that will look for spawned processed in the test container: ``` - rule: spawned_process_in_test_container desc: A process was spawned in the test container. condition: container.name = "falco-test" and evt.type = execve output: "%evt.time,%user.uid,%proc.name,%container.id,%container.name" priority: WARNING ``` Run falco for 45 seconds using the rules file: `sudo falco -r falco-rules.yml -M 45` On Control node, ``` cloud_user@k8s-worker1:~/falcorules$ sudo falco -r falco-rules.yml -M 45 Thu Nov 23 07:12:06 2023: Falco version: 0.36.2 (x86_64) Thu Nov 23 07:12:06 2023: Falco initialized with configuration file: /etc/falco/falco.yaml Thu Nov 23 07:12:06 2023: Loading rules from file falco-rules.yml Thu Nov 23 07:12:06 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs) Thu Nov 23 07:12:06 2023: Starting health webserver with threadiness 2, listening on port 8765 Thu Nov 23 07:12:06 2023: Loaded event sources: syscall Thu Nov 23 07:12:06 2023: Enabled event sources: syscall Thu Nov 23 07:12:06 2023: Opening 'syscall' source with Kernel module 07:12:08.692861724: Warning 07:12:08.692861724,0,sh,819b4aa57f9c,falco-test 07:12:08.693091658: Warning 07:12:08.693091658,0,sleep,819b4aa57f9c,falco-test 07:12:13.693929722: Warning 07:12:13.693929722,0,sh,819b4aa57f9c,falco-test 07:12:13.694133305: Warning 07:12:13.694133305,0,sleep,819b4aa57f9c,falco-test 07:12:13.997961095: Warning 07:12:13.997961095,0,sh,819b4aa57f9c,falco-test 07:12:13.998198689: Warning 07:12:13.998198689,0,cat,819b4aa57f9c,falco-test 07:12:18.694791102: Warning 07:12:18.694791102,0,sh,819b4aa57f9c,falco-test 07:12:18.695022744: Warning 07:12:18.695022744,0,sleep,819b4aa57f9c,falco-test 07:12:22.996922742: Warning 07:12:22.996922742,0,sh,819b4aa57f9c,falco-test 07:12:22.997354569: Warning 07:12:22.997354569,0,ls,819b4aa57f9c,falco-test 07:12:23.695916799: Warning 07:12:23.695916799,0,sh,819b4aa57f9c,falco-test 07:12:23.696113073: Warning 07:12:23.696113073,0,sleep,819b4aa57f9c,falco-test 07:12:28.697130319: Warning 07:12:28.697130319,0,sh,819b4aa57f9c,falco-test 07:12:28.697327272: Warning 07:12:28.697327272,0,sleep,819b4aa57f9c,falco-test 07:12:33.014873869: Warning 07:12:33.014873869,0,sh,819b4aa57f9c,falco-test 07:12:33.015098783: Warning 07:12:33.015098783,0,cat,819b4aa57f9c,falco-test 07:12:33.697970841: Warning 07:12:33.697970841,0,sh,819b4aa57f9c,falco-test 07:12:33.698203595: Warning 07:12:33.698203595,0,sleep,819b4aa57f9c,falco-test 07:12:38.699202460: Warning 07:12:38.699202460,0,sh,819b4aa57f9c,falco-test 07:12:38.699431733: Warning 07:12:38.699431733,0,sleep,819b4aa57f9c,falco-test 07:12:43.700184698: Warning 07:12:43.700184698,0,sh,819b4aa57f9c,falco-test 07:12:43.700391711: Warning 07:12:43.700391711,0,sleep,819b4aa57f9c,falco-test 07:12:48.701218073: Warning 07:12:48.701218073,0,sh,819b4aa57f9c,falco-test 07:12:48.701413926: Warning 07:12:48.701413926,0,sleep,819b4aa57f9c,falco-test Syscall event drop monitoring: - event drop detected: 0 occurrences - num times actions taken: 0 Events detected: 24 Rule counts by severity: WARNING: 24 Triggered rules by rule name: spawned_process_in_test_container: 24 ``` On Control node, you can get into the pod and type some commands, it is expected to be detect by falco ``` cloud_user@k8s-control:~/falco$ kubectl exec -it falco-test-pod sh kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. / # cat /etc/shadow root:*::::::: daemon:*::::::: bin:*::::::: sys:*::::::: sync:*::::::: mail:*::::::: www-data:*::::::: operator:*::::::: nobody:*::::::: / # ``` ``` ... 07:12:13.998198689: Warning 07:12:13.998198689,0,cat,819b4aa57f9c,falco-test ... ``` Rules example - Check process spawned but exclude non-infra container ``` - rule: spawned_process_in_test_container desc: A process was spawned in the test container. condition: > evt.dir = < and evt.type = execve and container.id != host and container.name != calico-node and container.name != kube-proxy output: "%evt.time,%user.uid,%proc.name,%container.id,%container.name" priority: WARNING ```