Kubernetes cluster Incident detection
===

### Setup

In order to send the kubernetes logs to the wazuh manager we made use of the webhook accessible on wazuh's official documentation. This sends kubernetes logs over an encrypted communication.

link: https://wazuh.com/blog/auditing-kubernetes-with-wazuh/

### On Kubernetes Server

In order to log activities on a kubernetes cluster, we need to enable kubernetes audit policy.
This policy tracks each requests sent to the kubernetes API server since it is kubernetes central component

> /etc/kubernetes/audit-policy.yaml

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
    # Dont log requests to the following API endpoints
    - level: None
      nonResourceURLs:
          - '/healthz*'
          - '/logs'
          - '/metrics'
          - '/swagger*'
          - '/version'

    # Limit requests containing tokens to Metadata level so the token is not included in the log
    - level: Metadata
      omitStages:
          - RequestReceived
      resources:
          - group: authentication.k8s.io
            resources:
                - tokenreviews

    # Extended audit of auth delegation
    - level: RequestResponse
      omitStages:
          - RequestReceived
      resources:
          - group: authorization.k8s.io
            resources:
                - subjectaccessreviews

    # Log changes to pods at RequestResponse level
    - level: RequestResponse
      omitStages:
          - RequestReceived
      resources:
          # core API group; add third-party API services and your API services if needed
          - group: ''
            resources: ['pods']
            verbs: ['create', 'patch', 'update', 'delete']

    # Log everything else at Metadata level
    - level: Metadata
      omitStages:
          - RequestReceived

> /etc/kubernetes/audit-webhook.yaml

apiVersion: v1
kind: Config
preferences: {}
clusters:
  - name: wazuh-webhook
    cluster:
      insecure-skip-tls-verify: true
      server: https://10.1.1.31:8080 

# kubeconfig files require a context. Provide one for the API server.
current-context: webhook
contexts:
- context:
    cluster: wazuh-webhook
    user: kube-apiserver
  name: webhook

> /etc/kubernetes/manifests/kube-apiserver.yaml

`systemctl restart kubelet`

## Attack & Incident Response

### 1. Privilege Escalation

In order to demonstrate wazuh incident detection and response capabilities, we made use of stratus red team which has a set of attacks for kubernetes environments.


k8s.credential-access.dump-secrets
k8s.credential-access.steal-serviceaccount-token
k8s.persistence.create-admin-clusterrole
k8s.persistence.create-client-certificate
k8s.persistence.create-token
k8s.privilege-escalation.hostpath-volume
k8s.privilege-escalation.nodes-proxy
k8s.privilege-escalation.privileged-pod


This command creates a privileged busybox pod


For us to determine the necessary fields to be parsed by our decoders and rules, we made use of the kubernetes API available at [link](https://github.com/kubernetes/kubernetes/blob/master/api/openapi-spec/swagger.json)


#### 1.1 Custom decoder to parse information

Our custom decoders are stored in the file `/var/ossec/etc/decoders/local_decoder.xml `

<decoder name="pods">
  <parent>json</parent>
  <regex type="pcre2">"kind": "\S+", "apiVersion": "\S+", "metadata": {"name": "(\S+)", "namespace": "(\S+)"</regex>
  <order>pod,ns</order>
</decoder>

#### 1.2 Custom rule to detect privileged pods
Our custom rules are stored in the file `/var/ossec/etc/rules/local_rules.xml `

<group name="kubernetes_audit,">
  <rule id="110000" level="0">
    <decoded_as>json</decoded_as>
    <description>Kubernetes audit log.</description>
  </rule>

  <rule id="110011" level="9">
    <if_sid>110000</if_sid>
    <regex type="pcre2">"verb": "create".+"securityContext": {"privileged": true}</regex>
    <description>Kubernetes privileged pod present</description>
  </rule>
</group>

#### 1.3 Detection on Wazuh dashboard


Fig: Logs


Fig: Privileged pod

#### 1.4 Mapping attack to MITRE ATT&CK Matrix

Regarding MITRE ATT&CK matrix, the attack above can be mapped to an **Escape to Host** Technique

Definition from MITRE ATT&CK: "Adversaries may break out of a container to gain access to the underlying host. This can allow an adversary access to other containerized resources from the host level or to the host itself."


To match this activity to the corresponding Mitre Att&ck technique, the wazuh rule should be modified as follows

  <rule id="110011" level="9">
    <if_sid>110000</if_sid>
    <regex type="pcre2">"verb": "create".+"securityContext": {"privileged": true}</regex>
    <description>Kubernetes privileged pod created in {namespace: $(ns)  pod: $(pod)}</description>
    <mitre>
      <id>T1611</id>
    </mitre>
  </rule>

After performing another attack we have the results shown below



#### 1.5 Active Response



`curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"`

`sudo install -o root -g wazuh -m 0755 /var/ossec/active-response/bin/kubectl`


We add our new command in the configuration file `vim /var/ossec/etc/ossec.conf`

  <command>
    <name>kubectl</name>
    <executable>kubectl</executable>
    <extra_args>delete pod $(pod) -n $(ns)</extra_args>
    <timeout_allowed>yes</timeout_allowed>
  </command>

The corresponding active response matching the privilege pod rule

  <active-response>
    <disabled>no</disabled>
    <command>kubectl</command>
    <location>local</location>
    <rules_id>110011</rules_id>
    <timeout>60</timeout>
  </active-response>


### 2. Detection of pod creation and deletion

 <rule id="110009" level="3">
    <if_sid>110000</if_sid>
    <regex type="pcre2">"verb": "delete".+"resource": "pods"</regex>
    <description>Kubernetes request to delete pod: $(pod)</description>
  </rule>

  <rule id="110010" level="3">
    <if_sid>110000</if_sid>
    <regex type="pcre2">"verb": "create".+"resource": "pods".+"securityContext":(?! {"privileged": true})</regex>
    <description>Kubernetes request to create pod: $(pod)</description>
  </rule>