# Experience as a LFX mentee at Kyverno

This summer I had the chance to work on an exciting feature at Kyverno as part of the LFX Mentorship.
This blog post is a record of my experiences and serves as a documention of my contributions.

## Acknowledgment

First, let me express my gratitude to my mentor **Shuting Zhao**, Senior software engineer at Nirmata, for her patience, kindness and guidance throughout the mentorship program.

I also want to thank **Jim Bugwadia**, Co-founder and CEO at Nirmata, for his inputs and guidance, and **Chip Zoller**, Technical Product Manager at Nirmata, for assisting me during my demo at a [Kubernetes #sig-auth meeting]( https://www.youtube.com/watch?v=etGq-IKxZRk&list=PL69nYSiGNLP0VMOZ-V7-5AchXTHAQFzJw&index=2).

Thanks to the **Linux Foundation** and the **Cloud Native Computing Foundation** for offering this amazing mentorship program and giving us the opportunity to work on open source projects.

## Introduction

### About me

I'm a final year student at Epitech Technology majoring in computer science. I'm passionate about cloud computing and cloud native projects.

### LFX Mentorship

The LFX mentorship is a **remote learning opportunity** for the open-source contributors who will be working for **12 weeks** under the guidance of mentors who are maintainers and developers of the particular project and they help the mentees to contribute to the community and project.
This mentorship program is organized thrice a year i.e. Spring, Summer, and Fall months.
The CNCF [maintains a repository](https://github.com/cncf/mentoring/tree/main/lfx-mentorship) with all the information you need including the participating projects and the required skills.

### How did I learn about the LFX mentorship program?

I was grateful to be a Dan Kohn scholarship recipient to attend the KubeCon + CloudNativeCon Europe 2022.

Some of my friends used Kyverno in their projects, so we stopped by the Kyverno booth where I had the chance to speak with project maintainers. I had already heard of the LFX mentorship program but one prior LFX mentee at Kyverno, Anushka Mittal, shared with me her experience and recommended me to give it a shot.

### Application

I checked the available Kyverno projects and the recommended skills for the Summer 2022 cycle in the [lfx-mentorship repository](https://github.com/cncf/mentoring/tree/main/lfx-mentorship).

One project particularly caught my eyes: [Integrate Kubernetes Pod Security with Kyverno](https://github.com/kyverno/kyverno/issues/3830) because the project might have a significant impact on the entire cloud native community and I already had some basic knowledge in Kubernetes and Go.

I had to submit 2 documents to apply in the [LFX Platform](https://mentorship.lfx.linuxfoundation.org/):

- **Resume**
- **Cover Letter**


In the cover letter, I made sure to highlight that I've used Kyverno, went through the codebase, and my comprehension of the issue that this project seeks to solve. Additionally, I quickly mentioned a few projects where I applied the recommended skills: Go application development and Kubernetes.

After two weeks, I received the following email from the Linux Foundation stating that I was accepted as a LFX mentee at Kyverno:


## Project: Extend Pod Security Admission

### Introduction

**Kubernetes v1.25 was released on August 23rd 2022** and one major change is the removal of `PodSecurityPolicy` and the graduation of `Pod Security Admission` to stable.

The [official release note](https://kubernetes.io/blog/2022/08/23/kubernetes-v1-25-release/) states:

*PodSecurityPolicy was initially deprecated in v1.21, and with the release of v1.25, it has been removed.*

*The updates required to improve its usability would have introduced breaking changes, so it became necessary to remove it in favor of a more friendly replacement.*

*That replacement is Pod Security Admission, which graduates to Stable with this release. If you are currently relying on PodSecurityPolicy, please follow the instructions for migration to Pod Security Admission.*

### Problem

Once Pod Security Admission is enabled for namespaces, a configured level of Privileged, Baseline, or Restricted applies to all pods and workloads within the namespace. The level is configured as a label on the namespace. There is no option to select specific pods or control, for granular policies.

Users can choose to configure controls for select pods, but there is no validation or enforcement beyond meeting the requirements for the specified level. This means that if a namespace is baseline, any pod in that namespace can run with root user privileges.

## Solution

### Kyverno

Kyverno is a Kubernetes-native policy management engine with which you can define logical policies across your Kubernetes cluster around validating, mutating and generative behaviour.

PSA is an in-tree solution for pod security compliance, and its best for users to leverage the built-in solution for pod security enforcement. Kyverno can be integrated with PSA to extend its ability by providing fine grained checks and other functions.

With this solution, Kyverno can reuse the in-tree pod security implementation and not require users to manage multiple policies, one for each control defined in the pod security standards.

### Extend PSA with a new validation rule

Kyverno can extend PSS profiles with a new validation rule: `validate.podSecurity`.

The rule will contain a `level` attribute that is set to a [Pod Security Standard profile](https://kubernetes.io/docs/concepts/security/pod-security-standards/) and a list of exclusions that the user wants to allow. When set to `restricted`, Kyverno will apply both Baseline and Restricted pod security profiles and report violations.

| `level` in Kyverno policy | Applied Pod Security Standard profiles |
| ------------- | ------------- |
| `restricted` | `restricted` + `baseline` |
| `baseline` | `baseline` |
| `privileged` | `privileged` |

If the PSA is enabled with "enforce=restricted" across the cluster, there's not much left to do. However, this is typically not the case. When PSA is in "enforce=baseline" or "enforce=privileged" mode, users can use the Kyverno rule to manage pod security in a granular fashion.

| Pod Security Admission | Kyverno policy |
| ------------- | ------------- |
| `enforce=restricted` | X |
| `enforce=baseline` | `enforce=restricted` and can exclude some restricted level checks |
| `enforce=privileged` | `enforce=baseline` and can exclude some baseline level checks |

The podSecurity validate rule applies configured pod security profile level to the selected namespaces. You can define exemptions from enforcement when creating pods. Exemptions can be configured via the podSecurity.exclude attribute.

We use container images as a selector for pod security controls that are declared at the container-level

### Example 1: Exclude a control

For example, if a user wants to enforce restricted PSS level to the selected namespaces "test" and "staging" but to skip checking the control "Capabilities" for pods running "ghcr.io/example/nginx:1.2.3", the following policy does the job.

validationFailureAction: enforce
rules:
  - name: enforce-restricted-exclude-all-capabilities-nginx
    match:
      any:
        - resources:
            kinds:
              - Pod
            namespaces:
              - test
              - staging
    validate:
      # this new type of rule only deals with PSS profiles
      # as we need to check if the value in the resource
      # is actually allowed by PSS
      podSecurity:
        level: restricted
        # version must be a valid Kubernetes minor version,
        # or `latest`.
        version: v1.24
        exclude:
          # controlName is the Control defined in PSS
          # See: https://kubernetes.io/docs/concepts/security/pod-security-standards/
          # +required
          - controlName: Capabilities
            # Select containers you want to exclude by specifying the image
            images:
              - ghcr.io/example/nginx:1.2.3

### Example 2: Exclude a control's specific restricted fields

If the user wants to specifically skip checking the custom "add" capabilities, restrictedField can be used to define the allowed list. For example, the following rule applies restricted PSS level to namespaces "test" and "staging" while excluding pods running image "ghcr.io/example/nginx:1.2.3" to be able to add kill, SETGID and SETUID capabilities.

validationFailureAction: enforce
rules:
  - name: enforce-restricted-exclude-capabilities-nginx
    match:
      any:
        - resources:
            kinds:
              - Pod
            namespaces:
              - test
              - staging
    exclude:
      any:
        - userInfo:
            username: some-user
    validate:
      # this new type of rule only deals with PSS profiles
      # as we need to check if the value in the resource
      # is actually allowed by PSS
      podSecurity:
        level: restricted
        # version must be a valid Kubernetes minor version,
        # or `latest`.
        version: latest
        exclude:
          # controlName is the Control defined in PSS
          # See: https://kubernetes.io/docs/concepts/security/pod-security-standards/
          # +required
          - controlName: Capabilities
            # Select containers you want to exclude by specifying the image
            images:
              - ghcr.io/example/nginx:1.2.3
            # +optional
            restrictedField: spec.containers[*].securityContext.capabilities.add
            # +optional
            values:
              - "KILL"
              - "SETGID"
              - "SETUID"

### Example 3: Fine grained control over Pod Security Admission

Let's create 2 namespaces to demonstrate the use case of the new validation rule:

- `baseline`: enforce baseline Pod Security Standard level

apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: baseline
    pod-security.kubernetes.io/enforce-version: v1.24
    pod-security.kubernetes.io/enforce: baseline
  name: baseline

- `privileged-with-kyverno`: enforce privileged Pod Security Standard level

apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: privileged-with-kyverno
    pod-security.kubernetes.io/enforce-version: v1.24
    pod-security.kubernetes.io/enforce: privileged
  name: privileged-with-kyverno

Let's try to run the following pod in these namespaces.

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: nginx
  name: nginx
spec:
  containers:
  - image: nginx:1.23.1-alpine
    name: nginx
    resources: {}
    securityContext:
      privileged: true
  initContainers:
  - image: node:18-alpine3.15
    name: init-nodejs
    resources: {}
    securityContext:
      privileged: true
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

For `baseline` namespace, we get the following error message that forbids the creation of the pod because `securityContext.privileged` was set to true for the 2 containers.

Error from server (Forbidden): error when creating "pod-privilged-container.yml": pods "nginx" is forbidden: violates PodSecurity "baseline:v1.23": privileged (containers "init-nodejs", "nginx" must not set securityContext.privileged=true)

Before trying to run the same pod in `privileged-with-kyverno`, let's apply a Kyverno rule that will exclude these checks.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: enforce-baseline-exclude-privileged-containers
spec:
  validationFailureAction: enforce
  rules:
  - name: enforce-baseline-exclude-privileged-containers
    match:
      any:
      - resources:
          kinds:
          - Pod
          namespaces:
          - privileged-with-kyverno
    validate:
      podSecurity:
         level: baseline
         version: v1.24
         exclude:
          - controlName: "Privileged Containers"
            restrictedField: spec.containers[*].securityContext.privileged
            images:
              - nginx:1.23.1-alpine
            values:
              - "true"
          - controlName: "Privileged Containers"
            restrictedField: spec.initContainers[*].securityContext.privileged
            images:
              - node:18-alpine3.15
            values:
              - "true"

If we try to run the pod, it's allowed thanks to the Kyverno rule.
Let's test that our Kyverno policy enforces the baseline PSS level to containers that are not matching the `exclude.images`. I changed the version of the nginx docker image.

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: nginx
  name: nginx
spec:
  containers:
  - image: nginx:1.22.1-alpine # <-- changed the nginx version
  # - image: nginx:1.23.1-alpine
    name: nginx
    resources: {}
    securityContext:
      privileged: true
  initContainers:
  - image: node:18-alpine3.15
    name: init-nodejs
    resources: {}
    securityContext:
      privileged: true
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

If we try to run this pod, Kyverno returns the following error and its creation is forbidden:

policy Pod/privileged-with-kyverno/nginx for resource violation:

enforce-baseline-exclude-privileged-containers:
  enforce-baseline-exclude-privileged-containers: |
    Validation rule 'enforce-baseline-exclude-privileged-containers' failed. It violates PodSecurity "baseline:v1.24": ({Allowed:false ForbiddenReason:privileged ForbiddenDetail:container "nginx" must not set securityContext.privileged=true})

## Organization

I had a weekly follow up meeting with my mentor to discuss the work done and plan the tasks to do for the next meeting. I highly recommend you to have a file with a progress tracker and where you keep meeting notes and share it with your mentor. It serves as a record of discussions for future reference.

The main mode of communication with my mentor and Kyverno maintainers was Slack.

The 1st month was dedicated to understanding [Pod Security Admission](https://kubernetes.io/docs/concepts/security/pod-security-admission/), going through the [source repository](https://github.com/kubernetes/pod-security-admission) and coming up with a first prototype for the new `validate.PodSecurity` Kyverno rule.

After that period, I spent roughly 5 weeks to implement all the Pod [Security Standard control](https://kubernetes.io/docs/concepts/security/pod-security-standards/) and the related unit tests.

Then I created a [Pull Request](https://github.com/kyverno/kyverno/pull/4364) for the feature. I resolved code reviews from Kyverno maintainers and fixed the last issues.

## Contributions

Here is the list of contributions I made throughout the program:

- [Pull Request: Extend Pod Security Admission](https://github.com/kyverno/kyverno/pull/4364)
- Demos at Kyverno Contributors meeting:
- [First demo on July 20th, 2022](https://zoom.us/rec/play/_X3W5Q3KodluVwCzY9R4ph8PA5wfGv0w9sKSxmDq8i40_EPI2UsXHG7L8XOmge32YmC1f07N6jzWQXs.MTWQJwIDftBJxusc?startTime=1658332990000&_x_zm_rtaid=2ww8QApUR4ScU6OITaRnSQ.1661763630312.0cdf2822c99b238835fbf6bd45d6ac43&_x_zm_rhtaid=793)
- Second demo on August 24th, 2022
- [Demo at a Kubernetes sig-auth meeting](https://www.youtube.com/watch?v=etGq-IKxZRk&list=PL69nYSiGNLP0VMOZ-V7-5AchXTHAQFzJw&index=2)

Open issues:

- kyverno/website documentation: [[Enhancement] Extend Pod Security Admission](https://github.com/kyverno/website/issues/604)
- kyverno/kyverno CLI: [[Feature] CLI: add support for validate.podSecurity rule](https://github.com/kyverno/kyverno/issues/4363)
- kubernetes/kubernetes: [Formalized CheckResult](https://github.com/kubernetes/kubernetes/issues/111958)

## Graduation, what's next ?

It was an insightful and challenging experience to work on this project. Big thanks again to all Kyverno maintainers for their help and warm support.

I consider this mentorship program as the starting point of my open source journey and will definitely continue to contribute Kyverno and other projects.

Thanks for reading and connect with me on:

- [Medium](https://medium.com/@altariah007)
- [GitHub](https://github.com/ToLToL)