---
# System prepended metadata

title: Cluster API Security Self-Assessment

---

---
breaks: false
---
<!-- markdownlint-configure-file { "MD013": { "line_length": 120 } } -->
<!-- markdownlint-disable-file MD034 -->

# Cluster API Security Self-Assessment

## Metadata


* **Last Modified**: 2022-02-23
* **Youtube Recordings**: [Cluster API Security Self-assessment Playlist](https://www.youtube.com/watch?v=PhIhakKnJGE&list=PLrQN4xC6U5D8jQyZ9uw7nRX5Xo0bsgctI)
* **Authors** (alphabetical order): 
    * [Ankita Swamy](https://github.com/Ankitasw)
    * [Naadir Jeewa](https://github.com/randomvariable)
    * [Pushkar Joglekar](https://github.com/pushkarj)
    * [Robert Ficcaglia](https://github.com/rficcaglia)
* **Github issue**: https://github.com/kubernetes/sig-security/issues/8

## Overview

This is a working document to describe the security assessment of the [Cluster API](#cluster-api) sub-project of
[Kubernetes](#Kubernetes-Security-Review). This is the pilot exercise to enable
[Kubernetes SIG Security](#Kubernetes-SIG-Security) to extend the great processes and methodologies
from CNCF Security TAG (Technical Advisory Group) on doing [security reviews](#TAG-Security-Reviews)

### Impact

This is a pilot exercise on how Kubernetes SIG Security can perform a security assessment of sub-projects where there may or may not be a separate working group or a SIG focssed on the sub-project and security. For example in this pilot, we have SIG Cluster Lifecycle focused on cluster-api project and sig-security focused on overall security of the Kubernetes project including its sub-projects.

### Scope

#### Process level

Complete an end to end assessment, solicit and incorporate feedback on the process, tools and methodologies to define
how this process can be revised after completion of this initial pilot as a template for future iterations of CAPI
itself, and other future sub-project assessments. As a derivative product of this effort it is hoped that the
sub-project community learns new things about secure coding, secure devops, security reviews / assessments, and
reporting and resolving security issues so as to become more self-sufficient and proactive in the day-to-day maintenance
and development of new features (i.e. instead of relying on discrete and infrequent point-in-time efforts post hoc.).

#### Technical

Cluster API sub-project in the context of AWS Provider is in scope for this assessment. We will assess threats and
review for security by going through two flows: Creation of single control plane node in a workload cluster and creation
and joining of single worker node in the workload cluster

### Not in Scope

This is not a formal code audit, pentest, vulnerability scan, or other point-in-time determination of the security of
Cluster API. It is meant to be an input into such activities in the future - either “organically” by community members, or as
part of a LF/CNCF paid external audit effort (refer to Kubernetes External Audit RFP Roadmap effort), or other community
or commercial efforts.

## Communication Channels

### Slack channels in Kubernetes Workspace

Visit https://slack.k8s.io to request an invite

* [#sig-security-assess-capi](#sig-security-assess-capi) - Working channel for this review
* [#cluster-api](#cluster-api) - Main Cluster API channel
* Provider channels:
  * [#cluster-api-aws](#cluster-api-aws) - Cluster API Provider AWS
  * [#cluster-api-azure](#cluster-api-azure) - Cluster API Provider Azure
  * [#cluster-api-openstack](#cluster-api-openstack) - Cluster API Provider OpenStack
  * [#cluster-api-baremetal](#cluster-api-baremetal) - General bare metal, incl. Metal3, Tinkerbell and BYOH
  * [#cluster-api-vsphere](#cluster-api-vsphere) - Cluster API Provider vSphere

### Mailing lists

* [kubernetes-sig-cluster-lifecycle@googlegroups.com](#kubernetes-sig-cluster-lifecycle@googlegroups.com)
* [kubernetes-sig-security@googlegroups.com](#kubernetes-sig-security@googlegroups.com)

### Github tracking

Tracking issues for this security audit have been opened as follows:

* Kubernetes SIG Security: https://github.com/kubernetes/community/issues/5814
* Cluster API: https://github.com/kubernetes-sigs/cluster-api/issues/4446
* CNCF Security TAG: https://github.com/cncf/tag-security/issues/603

### Primary Community Contact

Same as authors above

## Project Overview

Kubernetes Cluster API’s aim is to provide declarative APIs and tooling to simplify the provisioning, upgrading and
operating of multiple Kubernetes clusters. Cluster API uses Kubernetes-style APIs and patterns to automate cluster
lifecycle management for platform operators. The supporting infrastructure, like virtual machines, networks, load
balancers, and VPCs, as well as the Kubernetes cluster configuration are all defined in the same way that application
developers operate deploying and managing their workloads. This enables consistent and repeatable cluster deployments
across a wide variety of infrastructure environments.

### Project Goals

* To manage the lifecycle (create, scale, upgrade, destroy) of Kubernetes-conformant clusters using a declarative API.
* To work in different environments, both on-premises and in the cloud.
* To define common operations, provide a default implementation, and provide the ability to swap out implementations for
  alternative ones.
* To reuse and integrate existing ecosystem components rather than duplicating their functionality (e.g.
  node-problem-detector, cluster autoscaler, SIG-Multi-cluster).
* To provide a transition path for Kubernetes lifecycle products to adopt Cluster API incrementally. Specifically,
  existing cluster lifecycle management tools should be able to adopt Cluster API in a staged manner, over the course of
  multiple releases, or even adopting a subset of Cluster API.

### Project Non-goals

* To add these APIs to Kubernetes core (kubernetes/kubernetes).
* This API should live in a namespace outside the core and follow the best practices defined by api-reviewers, but is
  not subject to core-api constraints.
* To manage the lifecycle of infrastructure unrelated to the running of Kubernetes-conformant clusters.
* To force all Kubernetes lifecycle products (kops, kubespray, GKE, AKS, EKS, IKS etc.) to support or use these APIs.
* To manage non-Cluster API provisioned Kubernetes-conformant clusters.
* To manage a single cluster spanning multiple infrastructure providers.
* To configure a machine at any time other than create or upgrade.
* To duplicate functionality that exists or is coming to other tooling, e.g., updating kubelet configuration (c.f.
  dynamic kubelet configuration), or updating apiserver, controller-manager, scheduler configuration (c.f.
  component-config effort) after the cluster is deployed.

### Intended Uses of the Project

Cluster API may be used as a standalone tool, using the clusterctl CLI to install and manage Cluster API components as
well as template out clusters. Cluster API may also be used in whole or in part by Kubernetes distributions.

### Personas

#### Cloud Admin

The cloud admin operates and manages the infrastructure on top of which Kubernetes clusters are deployed. They may or
may not have specific experience with Kubernetes, but understand the underlying platform. They are responsible for
tenancy management, networking, security of the cloud environment, as well as capacity planning, policy for usage of the
cloud and backup and disaster recovery.

#### Platform Operator

The platform operator is focused on building a developer platform on top of Kubernetes. They standardise operations and
create policies for the running of clusters on different infrastructures. They onboard development teams onto Kubernetes
and build standardised pipelines for software delivery.

#### Workload Operator

The workload operator is a member of a software development team within a specific line of business. They may have
varying levels of experience with Kubernetes and are interested in automation of software delivery for their LOB. They
are also interested in monitoring, backup and restore and the consumption of cloud services.


#### Application Developer

The workload operator is a member of a software development team within a specific line of business. They may have
little experience with Kubernetes, and mostly care about getting their application running. They will typically work
only with the Kubernetes clusters they have been given access to.


### Primary Components


### Types of Clusters 

In Cluster API, we normally refer to two types of clusters: _management clusters_ and _workload clusters_.

#### Workload Clusters

Clusters managed _by_ Cluster API are called _workload clusters_, and are primarily what workload operators and
application developers will interact with.

#### Management Clusters

A management cluster is a cluster that actually runs the Cluster API components, contains the resources representing the
cluster and is reaching out to cloud provider services to perform infrastructure operations. We would expect the Cloud
Admin, Platform Operator and to some extent Workload Operators to interact with the Management Cluster, but not
typically Application Developers. The diagram below illustrates the relationship between management and workload
clusters.

:::warning
Move image below from imgur to Github for final PR!
:::

![Schematic of relation between management cluster and workload cluster](https://i.imgur.com/Jf0fuls.png)

##### Self-managed Management Cluster

A self-managed management cluster is a management cluster whose lifecycle is being managed by Cluster API running on the
management cluster itself.


#### Bootstrap Cluster

A bootstrap cluster is typically a Kubernetes “cluster” running on a local workstation - in the Cluster API book, we
typically recommend the usage of kind, but any developer Kubernetes distribution is sufficient. This is used to spin up
an initial cluster with Cluster API which maybe converted _into_ a management cluster. This is covered later. (TODO).
It is expected that platform operators use a bootstrap cluster to provision the initial management cluster.


##### Cluster API Core Controller

Cluster API Core controller is a pluggable provider in Cluster API which provides high level management of basic
primitives in Cluster API. It provides a blueprint of the cluster and its components like machine deployment,
machineset, and machines which acts as reference for cloud infrastructure providers to create infrastructure related
components.


##### Cluster API Bootstrap Kubeadm Controller

Cluster API Bootstrap Kubeadm Controller is a provider which helps to convert a machine into a kubernetes node and thus
bootstrapping the whole cluster and its machines. It uses kubeadm for bootstrapping kubernetes.

It takes care of initialising control plane nodes and bootstrapping other nodes as well.

Once all the nodes are bootstrapped, the Bootstrap kubeadm Controller then attaches the control plane and worker nodes
to the cluster with their specific roles within the cluster.


##### Cluster API Kubeadm Control Plane Controller

Cluster API Kubeadm Control plane controller provides ability to manage a control plane, including bootstrapping,
scaling and deletion of control plane without disturbing the whole cluster quorum.

The general expectation from the control plane controller is to make sure that the control plane node is created with
the services like etcd, kubernetes API server, kubernetes controller manager, scheduler, cloud controller manager,
kube-proxy and CoreDNS. It also provides the current status of the control plane to the operators.

Kubeadm control plane controller also manages kubeconfig secrets, which can be utilized by the operators to access the
workload clusters.

##### Cluster API Provider AWS Controller

Cluster API Provider AWS Controller is the infrastructure provider which manages resources that provides the
infrastructure required to run the cluster.

The general expectation from AWS controllers is to provision the network and compute resources while adhering to the AWS
best practices. Once the infrastructure is provisioned, then the EC2 instances are provisioned and form the AWS
Kubernetes cluster.

##### Kubeadm

Kubeadm is used by the bootstrap controller for bootstrapping Kubernetes clusters.

##### Cloud-Init

The Cluster API bootstrap kubeadm controller generates the cloud-init scripts which contains userdata to convert a
machine into a kubernetes node using kubeadm. This userdata is fetched from AWS secrets manager in the form of bash
script. This cloud-init script is then stored as secrets in kubeadm config and used by the Cluster API Provider AWS
controller for bootstrapping and creating EC2 instances.

The userdata also contains the CA certificate which is used by the control plane.

### **Boundaries and control plane interactions**

Cluster API components interact across the following boundaries:

* Node boundary
* VPC boundary
* Internet boundary

There are three types of clusters in a typical cluster API environment. For more details please refer
to [Types of Clusters](#types-of-clusters).

We will limit scope of this assessment, to two point in time states with a beginning and end state:

1. Existing Management cluster creates a control plane node of a workload cluster
2. Existing Management cluster and Workload cluster control plane add a new worker node

### **Data Stores and Data flows**

#### Data flow diagrams

The following data flow diagram is applicable to all of the different data flows in this document, but the exact
contents of each flow changes depending on the scenario.

:::warning
Github will not render embedded mermaid. We will need to generate the SVG for the PR!
:::

```mermaid
flowchart TB
    subgraph Bootstrap Node / Management Cluster
    kcp[Kubeadm Control Plane Controller]--https-->mgmtk8s
    kbc[Kubeadm Bootstrap Controller]--https-->mgmtk8s
    capi[Cluster API Controller]--https-->mgmtk8s
    capa[Cluster API AWS Controller]--https-->mgmtk8s
    mgmtk8s[Management Kubernetes API Server]--https-->mgmtetcd[etcd]
    end
    capa--https-->secrets
    capa--https-->EC2
    capa--https-->ELB
    kcp--https-->k8sapi
    capi--https-->k8sapi
    kbc--https-->k8sapi
    subgraph AWS Regional Services
    secrets[AWS Secrets Manager]
    EC2[Amazon EC2]
    ELB[Elastic Load Balancing]
    end
    subgraph VPC[Provisioned VPC]
    ELB--TCP Passthrough-->k8sapi
    IMDS[Instance Metadata Service]
    subgraph Workload EC2 Instance
    Kubelet
    Kubeadm
    cloud-init
    awscli[AWS CLI]
    cloud-init--executes-->awscli
    cloud-init--executes-->Kubeadm
    cloud-init--starts-->Kubelet
    end
    k8sapi--websocket-->Kubelet
    awscli--https-->secrets
    Kubeadm--https-->k8sapi
    Kubelet--http-->IMDS
    awscli--http-->IMDS
    Kubelet--https-->k8sapi
    subgraph Workload control plane
    k8sapi[Workload Kubernetes API server]
    end
    end

    classDef Amazon fill:#FF9900;
    classDef ThirdParty fill:#FFB6C1;
    classDef AmazonBoundary fill:#fff2e6;
    class EC2,secrets,EC2,ELB,IMDS,awscli Amazon
    class cloud-init ThirdParty
    class VPC AmazonBoundary
```

**Figure 1**: General data flow for Cluster API and Cluster API Provider AWS

#### Common Flows

##### EC2 Instance start

This is the common start sequence for a EC2 instance (VM) on AWS, and is included in the other flows.

1. AWS controller receives notification of created secret containing generated cloud-init config, connects with AWS
   Secrets Manager API, chunks and stores the cloud-init data at `/aws.cluster.x-k8s.io/<Random UUID v4>`
2. AWS controller runs a new EC2 instance (VM) at the Amazon EC2 API for the new node. The inserted instance userdata
   contains a bash script containing the location of the cloud-init data in AWS Secrets Manager
   (`/aws.cluster.x-k8s.io/<Random UUID v4>`)
3. On start-up, cloud-init in the new worker node is started by systemd.
4. cloud-init reads the userdata from AWS IMDS (Instance MetaData Service) that exists between VM and hypervisor over
   link-local where it is hosted without any authentication or encryption. However, IMDS is not accessible outside of
   the physical machine’s local network.
5. cloud-init reads and executes the bash script, which in turn executes the AWS CLI to first download the 'real'
   cloud-init configuration. The CLI is assumed to be installed on the image. Credentials for working with Amazon APIs
   are also retrieved from the instance metadata, and is set and rotated by the AWS EC2 control plane automatically.
6. Once fetched this data is deleted by AWS CLI from Secrets Manager API.
7. If the deletion fails, then as long as the node has successfully registered with the API server, then CAPA will
   also attempt a deletion of the secret. This operation is idempotent

###### Important considerations for other cloud providers

Cluster API Provider AWS is unique in this instance in using AWS Secrets Manager to store private key material
independently of the hypervisor<-guest->communication protocol (aka Instance Meta Data Service (IMDS)). This protects
against unauthorised retrieval of CA key material and bootstrap tokens by placing them in a seperate datastore to the
hypervisor, and using special credentials only visible to the VM itself to authorise retrieval. In most cloud providers,
including AWS, Azure and vSphere, read-only access to the cloud provider's VM service also allows users to read the data
stored in the IMDS, e.g. giving "read-only" access to the EC2 web console can allow someone to read the IMDS. Where a
mechanism like Secrets Manager isn't used, the plain-text data can be read unencrypted. When Secrets Manager is used,
the user will be able to see the URL of the location containing the private key material, but unless they have been
expressly granted authorisation to the URL, they are not able to read it. This feature is unique to the cloud-init
implementation in Cluster API Provider AWS. Support for reading from S3 has been added for Ignition, as well as AWS SSM
Parameter Store for special regions not having the Secrets Manager API. At the present time, similar mechanisms are not
present in Cluster API Provider Azure or Cluster API Provider vSphere.

#### Flow 1: Instantiation of a workload cluster control plane

1. Cloud admin OR workload operator creates CRDs on the API server of the management cluster. Standard authentication
   methods for Kubernetes API access are used (auth tokens, mutual TLS, authentication plugins).
2. Cluster API Core controller is assumed to be running on the same node (co-located) as API server IFF this is a
   bootstrap scenario, but this is not a hard requirement for a long-lived management cluster..
3. Bootstrap controller, Control plane controller and AWS controller interact directly with API server. All personas
   indirectly interact with these controllers via CRUD operations on CRDs and core K8s resources on the API server.
4. The kubeadm control plane controller creates CA key material, stored in etcd as Kubernetes secrets.
    * These include all standard CAs for Kubernetes:
        * etcd CA
        * kubernetes CA
        * kubernetes front proxy CA
    * In addition, an administrative kubeconfig is also generated for the new cluster and stored as a secret.
    * Kubernetes secrets are **not** encrypted by default. Encrypting these secrets is out of scope of this activity for
    * Cluster API
5. The three controllers then create several different CRDs within the management/bootstrap cluster’s API server.
6. The Cluster API Provider AWS (CAPA) controller starts reaching out to EC2 API which crosses internet boundary, with
   authorisation headers using [AWS4-HMAC-SHA256] to create a network for the workload cluster.
7. CAPA then connects with ELB API to create a load balancer for API server of workload cluster, and waits for the API
   server to be DNS resolvable.
8. See "EC2 Instance Start" flow
9. cloud-init writes out a kubeadm configuration, as well as all of the  key material to /etc/kubernetes/pki and
    executes kubeadm pointing at the configuration,
10. kubeadm starts the kubelet process via systemd.
11. KubeAdm the default bootstrap provider, then ingests the KubeAdm configuration and then restarts Kubelet until
    kubelet is successfully started with kubelet configuration. Note: kubeadm flow to get the kubelet up and running is
    out of scope of this assessment, but is broadly.
    * Use the key material to generate all required certificates.
    * Generate static pod manifests for the control plane components.
      * OCI images for kubernetes control plane components (scheduler, controller-manager, API server, etcd) are
        typically pre-baked in the machine for which kubeAdm creates static pods.
    * Wait for kubelet to initialize etcd and the control plane.
    * Generate a bootstrap service account token for the kubelet to self-register
      as a node.
12. Control plane, CAPI Core, AWS and Bootstrap controller then talk to API server via ELB created by ELB API to manage
    the workload cluster.

At the end of this flow, we have a workload cluster running with one control plane node and zero worker nodes.

#### Flow 2: Adding a machine to the workload cluster control plane

1. KubeadmControlPlane Controller (KCP) reads the CA certificate key material for the cluster from the management
   cluster (as secrets).
2. KCP templates out a set of Machine, AWSMachine and KubeadmConfig resources for the new node, including references to
   the copies (secretRefs) of the CA certificate key material in the KubeadmConfig resource.
3. Bootstrap controller is notified of new KubeadmConfig resource, and renders a cloud-init configuration as a secret.
4. See "EC2 Instance Start flow"
5. KCP continuously connects to the workload cluster's control plane API server,
   scans for all control plane nodes,
   and opens a proxied connection via the API server, and the node's kubelet to
   etcd using a generated client certificate for etcd.
6. cloud-init writes out a kubeadm configuration and all CA certificate key material to relevant locations in
   /etc/kubernetes/pki
7. cloud-init executes kubeadm, and kubeadm starts kubelet via systemd.
8. kubeadm reads a kubeadm configmap from the kube-system namespace on the existing control plane member (via the ELB)
   to get the list of etcd peers.
9. kubeadm stamps out static pod manifests for the control plane instance
10. kubeadm monitors the etcd join process.
11. kcp monitors the etcd join process via its proxying.
12. kubeadm generates a bootstrap token for kubelet to register to the control plane.
13. kubeadm completes the join process as per Flow 3.

#### Flow 3: Adding a new worker node to a workload cluster

1. Cluster API controller templates out a set of Machine, AWSMachine and KubeadmConfig resources for the new node.
2. Bootstrap controller is notified of new KubeadmConfig resource, and creates a service account on the workload cluster.
3. This service account token with generated cloud init data is stored in management cluster's Kubernetes API server.
4. See "EC2 Instance Start" flow
5. cloud-init writes out a kubeadm configuration containing the service account token and executes kubeadm pointing at
    the configuration.
6. kubeadm starts the kubelet process via systemd.
7. Kubelet then creates a certificate signing request and sends it to the workload kubernetes API server, using the
    service account token as the authorisation header. The serving certificate of the control plane is checked
    beforehand against that provided in the configuration.
8. Kubeadm *on the worker node* checks for the existence of the CSR on the workload API server and  *authorises* the
    CSR request. The only check, performed on the client side is that the machine name is not already registered.
9. Workload control plane's running instance of the kubernetes-controller-manager signs the certificate.
10. Kubelet retrieves the signed certificate, and then registers itself as a node in the workload control plane.
11. Bootstrap controller keeps track of the existence of the worker node in the workload cluster. If the node does not
    appear within the expiry period (15 mins) of the token, the bootstrap controller will keep renewing the token
    without any limits to how many times it can be renewed.
12. Cloud controller manager of the workload cluster will keep track of the existence of the new worker node and then
    validate the node name against the EC2 API to confirm that node is now part of a cluster and does not need to be
    deleted.

### 3rd Party Requirements (source, libraries, services, APIs)

SBOMs are not currently produced by the Cluster API project, but dependencies are viewable in Github at:

* [https://github.com/kubernetes-sigs/cluster-api/network/dependencies](https://github.com/kubernetes-sigs/cluster-api/network/dependencies)
* [https://github.com/kubernetes-sigs/cluster-api-provider-aws/network/dependencies](https://github.com/kubernetes-sigs/cluster-api-provider-aws/network/dependencies)
* [https://github.com/kubernetes-sigs/cluster-api-provider-azure/network/dependencies](https://github.com/kubernetes-sigs/cluster-api-provider-azure/network/dependencies)
* [https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/network/dependencies](https://github.com/kubernetes-sigs/cluster-api-provider-vsphere/network/dependencies)

For all of the subprojects, the main dependency is
 [Kubernetes Controller Runtime](https://github.com/kubernetes-sigs/controller-runtime), and for each of the providers,
 it would be there primary infrastructure API, e.g. AWS SDK for Go, Microsoft Azure SDK for Go and
 [govmomi](https://github.com/vmware/govmomi).

Cluster API requires a bootstrap cluster in order to produce a self-managed management cluster, and we normally
recommend “[kind](https://kind.sigs.k8s.io/)” (Kubernetes in Docker), which depends on Docker or Podman for operation.

#### Identified Gaps

A process should be implemented to generate an SBOM for the primary artifacts of the Cluster API subprojects, including
OCI images and command line binaries.

### Secure development practices

_Refer to the CII Best Practices, specifically the security section(s) as a start. As a bonus, consider NIST standards
and industry guidance._

* _Development Pipeline Impact. It is assumed that a sub-project may already be subject to the parent project’s
  decisions so link to, or briefly describe the testing and review processes that the software undergoes as it is
  developed and built. Of interest might be specific information as to whether contributors are required to sign
  commits, if project artifacts/images are immutable and signed, any automated checks for vulnerabilities, etc._
* _Communication Channels. Reference where you document how to reach your main project team and sub-project leads, for example:_
  * _Internal. How do team members communicate with each other?_
  * _Inbound. How do users or prospective users communicate with the team?_
  * _Outbound. How do you communicate with your users? (e.g. flibble-announce@ mailing list)_
* _Ecosystem. How does your software fit into the main project and other cloud native ecosystems?_

#### Development Workflow

Cluster API’s development workflow is documented within the repository’s
[CONTRIBUTING.md](https://github.com/kubernetes-sigs/cluster-api/blob/main/CONTRIBUTING.md)

Key processes include:

* All source code is publicly available in Github
* All pull requests trigger jobs that perform:
  * Linting, including gosec static code analysis
  * Unit tests
  * E2E tests using nested Docker as a simulated infrastructure provider
  * Checks for breaking API changes
  * Tests for broken URLs in generated documentation
  * PRs cannot be merged without passing tests without an explicit /override from a maintainer
* Each PR requires approval from maintainers listed in the OWNERS file
* For all Cluster API subprojects, contributors are required to have signed the CNCF CLA, but no cryptographic
  signatures are required for individual commits.
* Pushes to the main branch are forbidden by convention
* Release process is almost fully automated via Github actions
* Checksum, signature and SPDX files are not currently generated for any artifacts
* Container images are produced automatically by Google CloudBuild, and Cluster API uses the upstream Kubernetes image
  promotion process to promote individual OCI SHAs from staging to release.
* Deployment manifests reference image tags rather than SHA digests
* ACLs for ownership of the repository and approvals are configured in code within the OWNERS file at the top of the
  repository as well as configured via the github.com/kubernetes/org repository.
* For Cluster API Provider AWS since v0.6.7 and Cluster API Provider Azure since v0.4.13 maintainers do GPG sign all
  tags, and maintainers are expected to maintain their GPG keys with public keys verifiable in Github. Cluster API Core
  repository has tags signed by Github Actions since v1.0.3

#### Identified gaps in development workflow

##### [DEV-01: Outdated OWNERS file](https://github.com/kubernetes-sigs/cluster-api/issues/6151)

Listed maintainers of the Cluster API project are out of date and may continue to have administrative access to the
Github repository as well as perform image promotion and access the staging Google Cloud account. A process to
regularly review ACLs should be instituted.

#### Core Infrastructure Initiative

Cluster API asserts being 96% in progress with regards to the [Core Infrastructure Initiative (CII) badging process](https://bestpractices.coreinfrastructure.org/en/projects/5288).

##### Identified gaps vis-a-vis Core Infrastructure Initiative

##### [DEV-02: Vulnerability report process is poorly defined](https://github.com/kubernetes-sigs/cluster-api/issues/5398)

The project has a SECURITY_CONTACTS file which nominally follows the Kubernetes vulnerability reporting process, but
the file has not been maintained, and has not been invoked for subprojects as far as the authors are aware. In addition
the contacts only lists Github usernames - it isn’t clear how Kubernetes security is to get in touch with project
maintainers in the case of private disclosure.


## Threat Modeling

### Attack Trees

:::warning
* Define adversary goals and techniques
* Consider [Failure Mode Analysis](https://en.wikipedia.org/wiki/Failure_mode_and_effects_analysis)
* Consider Fuzzing, Fault Injection
* Graph Paths from Entry Points to Critical Targets {#graph-paths-from-entry-points-to-critical-targets}
* Inventory the nodes in the system
* Define the relationships between nodes
* Define meaningful attributes to nodes and relationships and clusters of nodes and critical paths
* Use graph search algorithms to discover and catalog novel paths 
:::

### STRIDE

#### Spoofing

##### STRIDE-SPOOF-1 - Spoofing of Cloud Admin

The identity of the Cloud Admin with access to Management Cluster can be spoofed by other users via
stolen cloud and kubernetes credentials since cloud provider credentials are stored or are able to retrived from
the workload cluster.

For on-prem infrastructures, the Cloud Admin will have provisioned the cluster with wide ranging
permissions, including the ability to create and delete VMs, add and remove disks etc... These by default will be
written to the CPI credentials file on the disk of the workload control plane instances. A user of the cluster
can start a pod with a host mount of /etc, and exfiltrate the cloud provider credentials.

For hyperscaler environments like AWS, there is the ability to scope short lived credentials to the level of
EC2 instances. However, the permissions are still broad for CPI and CSI to function, and a management cluster
hosted in AWS will have even more permissions, such as the ability to create and destroy EC2 instances. A user of
the cluster can start a pod, and scrape the IMDS endpoint living on link-local (`http://169.254.169.254`) and
retrieve these credentials. They are short-lived but a confused deputy attack is still possible.

Cloud Admin has access to multiple Cloud services like EC2, ELB, Secrets Manager. Access to the management cluster
basically provides this admin access to all the managed workload clusters and its workloads as well.

###### STRIDE-SPOOF-1 Recommended mitigations

Securely manage Cloud and Kubernetes Credentials with following methods applied:

* Credentials are short-lived
  * Status: Implemented (AWS)
* Credentials are managed centrally from an authoritative identity store
  * Status: Implemented (AWS & Azure)
* Credentials can be revoked, if needed
  * Status: Implemented (AWS & Azure)
* Second pair of eyes are applied when executing privileged actions such as creating/deleting/updating a cluster
  * Status: [End user guidance to be written][cluster-api-end-user-guide]
* Ensure credentials used by Cluster API are least privileges and setting access control on Cluster API
  controller namespaces to prevent unauthorised access by anyone other than cloud admin.
  - Status: Implemented for K8s RBAC, developer guidance to be written for providers
* Enable auditing on the management cluster as well as on the cloud provider
  * Status: [End user guidance to be written][cluster-api-end-user-guide]

##### STRIDE-SPOOF-2 Spoofing of cloud provider services

Cloud provider services like Secrets Manager store sensitive data like cloudinit configuration files
(shortlived). Other services like EC2 and ELB API have the ability to sniff traffic destined to the API server of
the workload cluster. Successful spoofing of a cloud service could result in information disclosure and tampering
of in flight requests relevant to cluster and machine management.

###### STRIDE-SPOOF-2 Recommended Mitigations

* Verify server side identity to confirm that AWS service is indeed what it advertises to be.
  * Status: Implemented (vSphere) / Out-of-scope (AWS/Azure)
  * AWS maintainers note: We rely on the [AWS Shared Responsibility model]. There is realistically little we can do
    in the event of a major breach. Validation of server endpoints is left to the standard SDK.
    
##### STRIDE-SPOOF-3 Spoofing of image-builder maintainer
Description: Machine image for cluster is pre-baked and built by cluster API maintainers. This image could be
tampered with to include a cryptominer if the accounts of maintainers that have privileges and ownership over the
code and machine image built for Nodes in a cluster.

###### STRIDE-SPOOF-3 Recommended Mitigations

* Implement 2FA for all maintainer accounts on Github. Apply the second pair of eyes
  principle when performing privileged actions such as image building or updates to the contents of the machine
  images.
  * Status: Implemented
  * AWS maintainers note: Public community consumed images are currently maintained by VMware staff. Access to this
    environment is 2FA protected and managed by an SRE team. In order to give more confidence to the community, it
    is planned to transfer image building to CNCF owned accounts.

##### STRIDE-SPOOF-4 Spoofing of a control plane node in a workload cluster

When a worker node is created, a user could somehow interfere with the boot, e.g. breaking
networking, assigning incorrect security groups. This would prevent the encrypted cloud-init data from being
deleted. If the user could gain access to the node, they could use the IMDS provided credentials to read the
bootstrap token from Secrets Manager and attempt to register AS a control plane node, since validation is only
done client- side by kubeadm. This can allow the attacker to gain access to secrets they should not be able to.

###### STRIDE-SPOOF-4 Recommended Mitigations

* Replace kubeadm node join mechanism with a host attestation mechanism.
  * Status: Planned

:::info
> [name=Pushkar Joglekar] Naadir to rewrite this as per this comment:
> https://docs.google.com/document/d/1Fj_cLUN9kLruHbEgmYiEgoqZjf2rRuVOmQDGOKByaf4/edit#bookmark=id.myigotfujhmz
:::

##### STRIDE-SPOOF-5 Spoofing of a worker node in a workload cluster

Any attacker controlled machine could claim to be a worker node that needs to join a workload cluster
and request user data from Secrets Manager API.

###### STRIDE-SPOOF-5 Recommended Mitigations

* Same as 5b.
  * Status: implemented / planned / to be implemented / End User Guidance

:::info
> [name=Pushkar Joglekar] Naadir to rewrite this as per this comment:
> https://docs.google.com/document/d/1Fj_cLUN9kLruHbEgmYiEgoqZjf2rRuVOmQDGOKByaf4/edit#bookmark=id.7id7y2naua8j
:::

#### Tampering

##### STRIDE-TAMPER-1 Tampering of Cluster API component binaries and configuration files

Cluster API components (Controllers and CAPI Core) can be tampered during build time, during install

###### STRIDE-TAMPER-1 Recommended Mitigations

* At build time: Create SBoM of all the Cluster API components and verify checksum as a post build action
  * Status: [Issue filed](https://github.com/kubernetes-sigs/cluster-api/issues/6153)
* During install time: Perform mutual authentication when invoking clusterctl command. Verify checksum of the
   downloaded components. Download with TLS enabled to prevent in-transit tampering.
  * Status: implemented / planned / to be implemented / End User Guidance
* During runtime: Alert on modification or restarts of cluster API components to detect for potential tampering.
  * Status: [End user guidance to be written][cluster-api-end-user-guide]

TODO for Naadir : Add image selection attack vector for cluster creation

##### STRIDE-TAMPER-2 Tampering of Machine Images

Machine images used as Node OS images, can be tampered during build time, during install time and
during runtime

###### STRIDE-TAMPER-2 Recommended Mitigations

* At build time: Create SBoM of the Machine Image and verify checksum as a post build action
  * Status: [Issue filed](https://github.com/kubernetes-sigs/image-builder/issues/803)
* During install time: Perform mutual authentication when invoking clusterctl command. Verify checksum of the
  downloaded machine image, before marking them as available. Download with TLS enabled to prevent in-transit
  tampering.
  * Status: Partially implemented
    * Maintainers note: clusterctl by default uses Github APIs for releases
* During runtime: Disable or restrict SSH access to prevent updates to machine image at runtime. Create
  alerting for sensitive file systems and when the machine is restarted.
  * Status: [End user guidance to be written][cluster-api-end-user-guide]
      * Maintainers note: No password or SSH auth key is stored in image builder by default, and a user may
        optionally set a password or add authorised keys during first-boot using cloud-init configurations.


#### Repudiation

##### STRIDE-REPUDIATE-01 Repudiation of actions performed by Cloud Admin

Infra Admin has highly privileged access to cloud and cluster resources. Misuse of permissions via
actions performed by Infra Admin should be detectable and logged for auditing purposes.

###### STRIDE-REPUDIATE-01 Recommended Mitigations

* Implement API server auditing for all clusters and Cloud API auditing to log all actions
  performed by Infra Admins. Additionally, implement centralized audit collection and alerting for any suspicious
  activity.
  * Status: [End user guidance to be written][cluster-api-end-user-guide]

##### STRIDE-REPUDIATE-02 Repudiation of actions performed using cloud credentials

Cloud credentials that are used by the Cluster API components, have highly privileged access to cloud
resources. Misuse of these credentials by accidental leakage or by cluster admins, should be detectable and logged
for audit purposes

###### STRIDE-REPUDIATE-02 Recommended mitigations

* Recommended Mitigation: Implement Cloud API auditing to log all actions performed using cloud credentials granted
  to the management and workload cluster’s control plane. Additionally, implement centralized audit collection and
  alerting for any suspicious activity.
  * Status: [End user guidance to be written][cluster-api-end-user-guide]

#### Information Disclosure

##### STRIDE-INFODISCLOSE-1 Privilege escalation from workload cluster co-located with management cluster

Cluster API Provider AWS by default uses the same set of permissions for workload and management
control planes, and if a workload cluster is colocated in the same AWS account as a management cluster, a user can
use the credentials in the workload cluster to snapshot the disk of the management cluster, create a new EC2
volume, mount it to a node in the workload cluster, and exfiltrate the etcd datastore.

###### STRIDE-INFODISCLOSE-1 Recommended mitigations

* CAPA should by default use separate IAM roles for workload clusters to management clusters to reduce the amount
  of privileges workload clusters have.
  * Status: Not yet considered
* Cluster API should consider supporting running CPI, CSI and other privileged control plane workloads on the
  management cluster to further minimise the privileges required by workload clusters.
  * Status: [Issue filed](https://github.com/kubernetes-sigs/cluster-api/issues/5491)
* Strongly advise users to seperate workload clusters into different AWS accounts or Azure subscriptions.
  * Status: [End user guidance to be written][cluster-api-end-user-guide]

##### STRIDE-INFODISCLOSE-2 Exposure of kubeadm token credentials
Exposure of kubeadm token located on the node or in Cloud resources like Secrets Manager can lead to
full or partial control over one or more clusters, and can be reused to register as the control plane node itself,
after which attacker can get access to control plane secrets.

###### STRIDE-INFODISCLOSE-2 Recommended Mitigations

* kubeadm token is short lived or deleted
  * Status: implemented / planned / to be implemented / End User Guidance
* Disable or restrict SSH access to nodes in a cluster
  * Status: implemented / planned / to be implemented / End User Guidance
* Audit KubeConfig files access, edits that are located on the node
  * Status: implemented / planned / to be implemented / End User Guidance
* Implement cluster level Pod Security to prevent host mount access to pod
  * Status: implemented / planned / to be implemented / End User Guidance

##### STRIDE-INFODISCLOSE-3 - Exposure of Management and Workload Cluster CA private keys and certs

Exposure of Management Cluster CA private keys and certs on the node or in the cloud resources like
Secrets Manager can lead to full or partial control over one or more clusters.

###### STRIDE-INFODISCLOSE-3 Recommended Mitigations

* Keep CA key/cert pairs short lived or deleted after use from Secrets Manager
* Shorten CA cert lifetime
* Disable or restrict SSH access to nodes in a cluster
* Audit access or edits to CA private keys and cert files located on the node
* Implement cluster level Pod Security to prevent host mount access to pod
  * Status: implemented / planned / to be implemented / End User Guidance

##### STRIDE-INFODISCLOSE-4 - Exposure of Cloud credentials that allow privileged actions

Cloud credentials consumed by cluster API components can be leaked or exfiltrated to spoof Cluster
API components or launch a denial of service attack

###### STRIDE-INFODISCLOSE-4 Recommended Mitigations

* Use short-lived credentials that are auto-renewed using node level attestation
  * Status: implemented / planned / to be implemented / End User Guidance
* Disable or restrict SSH access to nodes in a cluster
  * Status: implemented / planned / to be implemented / End User Guidance
* Implement cluster level Pod Security to prevent host mount access to pod
  * Status: implemented / planned / to be implemented / End User Guidance

#### Denial of Service

##### STRIDE-DOS-1 - Misuse of cloud credentials to create unlimited number of cloud resources

Cloud credentials can be misused by someone who has access to it by accidentally or maliciously
creating very large number of cloud resources to exhaust cloud capacity or for financial damage caused due to
surprise bills

###### STRIDE-DOS-1 Recommended Mitigations

* Create safe defaults (soft and hard) for the maximum number of cloud resources that can be created.
* Apply rate limits on how many “create cloud resources” requests can be successfully processed using the same
  cloud credentials.
* Implement threshold based alerts when a particular soft limit is exceeded
  * Status: implemented / planned / to be implemented / End User Guidance

##### STRIDE-DOS-2 - Misuse of Infra Admin credentials to create unlimited number of clusters

Infra Admin credentials can be misused by someone who has access to it by accidentally or maliciously
creating very large number of clusters to exhaust capacity or slow down the operations team or for financial
damage caused due to surprise bills

###### STRIDE-DOS-2 Recommended Mitigations

* Create safe defaults (soft and hard) for the maximum number of clusters that can be created using a
  management cluster
  * Status: implemented / planned / to be implemented / End User Guidance
* Apply rate limits on clusterctl and underlying APIs on concurrent and successive requests to create new
  clusters can be processed using the same Infra admin
  * Status: implemented / planned / to be implemented / End User Guidance
* Implement threshold based alerts when a particular soft limit is exceeded for number of clusters
  * Status: implemented / planned / to be implemented / End User Guidance
* Apply second pair of eyes principle when performing privilege actions such as creating a cluster by an Infra admin
  * Status: implemented / planned / to be implemented / End User Guidance

##### STRIDE-DOS-3 - Misuse of Infra Admin credentials to create unlimited number of clusters

Infra Admin credentials can be misused by someone who has access to it by accidentally or maliciously
creating very large number of clusters to exhaust capacity or slow down the operations team or for financial
damage caused due to surprise bills

###### STRIDE-DOS-3 Recommended Mitigations

* Create safe defaults (soft and hard) for the maximum number of clusters that can be created using a
  management cluster
  * Status: implemented / planned / to be implemented / End User Guidance
* Apply rate limits on clusterctl and underlying APIs on concurrent and successive requests to create new
  clusters can be processed using the same Infra admin
  * Status: implemented / planned / to be implemented / End User Guidance
* Implement threshold based alerts when a particular soft limit is exceeded for number of clusters
  * Status: implemented / planned / to be implemented / End User Guidance
* Apply second pair of eyes principle when performing privilege actions such as creating a cluster by an Infra admin
  * Status: implemented / planned / to be implemented / End User Guidance

##### STRIDE-DOS-4 - Misuse of Infra Admin credentials to create unlimited number of CRDs to exhaust etcd storage

Infra Admin credentials can be misused by someone who has access to it by accidentally or maliciously
creating a very large number of CRD objects in the management cluster to exhaust underlying storage of etcd. This
could occur as a side effect of the previous threat of creating an unlimited number of clusters but can also
happen through an unlimited number of nodes and other cluster resources being added to one or more clusters.

###### STRIDE-DOS-4 Recommended Mitigations

* Implement safe defaults (soft and hard) on how many CRD objects a Cluster API component can create,
  especially in a management cluster
  * Status: implemented / planned / to be implemented / End User Guidance
* Safe defaults on number of cluster and nodes will also have side effect of preventing this threat
  * Status: implemented / planned / to be implemented / End User Guidance

##### STRIDE-DOS-5 - Deleting ELB instance for API server of workload cluster to disconnect management cluster from workload cluster(s)

Elastic Lb instance is the single point of failure in terms of connectivity between management
cluster and workload clusters. If that load balancer instance is deleted, all connectivity between management
cluster and workload cluster control planes is lost

###### STRIDE-DOS-5 Recommended Mitigations

* Implement heartbeats between management and workload cluster to quickly detect connectivity issues
  * Status: implemented / planned / to be implemented / End User Guidance
* Alert on deletion of Elastic Lb so that it can be quickly restored
  * Status: implemented / planned / to be implemented / End User Guidance

##### STRIDE-DOS-6 Exhausting Cloud resources

Misuse of cloud resources to accidentally or purposefully create, delete, update machine images or
ELB instances could lead to resource exhaustion or surprise bills or traffic congestion on cloud API pathways

###### STRIDE-DOS-6 Recommended Mitigations

* Implement rate limits for creation, deletion and update of cloud resources
  * Status: implemented / planned / to be implemented / End User Guidance
* Apply second pair of eyes whenever possible for such activity
  * Status: implemented / planned / to be implemented / End User Guidance
* Apply alerting on all cloud resource creation, update, deletion activities
  * Status: implemented / planned / to be implemented / End User Guidance

#### Elevation of Privilege

##### STRIDE-EOP-1  Using workload cluster credentials to get access to mgmt. Cluster

Workload cluster credentials could be used to elevate privileges such that the attacker can get full
or partial control over management cluster

###### STRIDE-EOP-1 Recommended Mitigations

* Separate workload cluster and management cluster at network level through VPC boundary
  * Status: implemented / planned / to be implemented / End User Guidance
* Implement Cluster level Pod Security to prevent host access for someone with Create Pod access
  * Status: implemented / planned / to be implemented / End User Guidance
* Disable or limit node level SSH access
  * Status: implemented / planned / to be implemented / End User Guidance
* Do not build a chain of trust for cluster CAs such that gaining access to workload cluster CA and private
  keys does _not_ provide access to management cluster
  * Status: implemented / planned / to be implemented / End User Guidance
* Limit who can create pod on control plane nodes
  * Status: implemented / planned / to be implemented / End User Guidance

##### STRIDE-EOP-2 Using create pod permission to get cluster admin access to workload cluster

Someone with create pod permissions can elevate their access by locating the credentials that can
give cluster-admin access to the attacker

###### STRIDE-EOP-2 Recommended Mitigation

* Limit who can create pod on control plane nodes
  * Status: implemented / planned / to be implemented / End User Guidance
* Implement Cluster level Pod Security to prevent host access for someone with Create Pod access
  * Status: implemented / planned / to be implemented / End User Guidance
* Disable or limit node level SSH access
  * Status: implemented / planned / to be implemented / End User Guidance

##### STRIDE-EOP-3 Use shared cloud credentials to get access to other unrelated cloud resources

Cloud credentials consumed by cluster API components could be used to create cloud resources that are
unrelated to the lifecycle of the cluster and misused by attacker for their own gain

###### STRIDE-EOP-3 Recommended Mitigations

* Create dedicated cloud credentials that have only the necessary permissions to manage lifecycle of a cluster
  * Status: implemented / planned / to be implemented / End User Guidance
* Any cloud resource not linked to a cluster after a fixed configurable period of time created by these cloud
  credentials, should be auto-deleted or marked for garbage collection
  * Status: implemented / planned / to be implemented / End User Guidance

### Security issue resolution

* Responsible Disclosures Process.  It is assumed that a sub-project may already be subject to the parent project’s
  decisions so link to, or briefly describe the project's responsible disclosures process should suspected security
  issues, incidents, or vulnerabilities be discovered. The outline should discuss communication methods/strategies both
  in the sub-project and relative to the parent project.
    * Vulnerability Response Process. Who is responsible for responding to a report. What is the reporting process? How
      would you respond?
* Incident Response. A description of the defined procedures for triage, confirmation, notification of vulnerability or
  security incident, and patching/update availability. For example if the sub-project community or build infrastructure
  is itself a target of an attack, or perhaps suspected of assisting in an attack. Possibly dry run or perform thought
  experiments on what worst case scenarios could occur and how you would manage these, relative to the parent project,
  LF/CNCF and as it pertains to humans involved, authorities.


## References

### Docs

* [Introduction to Cluster API](https://cluster-api.sigs.k8s.io/introduction.html)
* [Introduction to Image Builder](https://image-builder.sigs.k8s.io/introduction.html)
* [Introduction to Kubebuilder](https://book.kubebuilder.io/)

### Talks

* [Cluster API Deep Dive - Naadir Jeewa, VMware & Cecile Robert-Michon, Microsoft](https://www.youtube.com/watch?v=9SfuQQeeK6Q)

### Links

##### [AWS4-HMAC-SHA256](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html)
##### [AWS Shared Responsibility model](https://aws.amazon.com/compliance/shared-responsibility-model/)
##### [Cluster API](https://github.com/kubernetes-sigs/cluster-api/)
##### [Kubernetes Cluster API](https://github.com/kubernetes-sigs/cluster-api/)
##### [Kubernetes Security Review](https://github.com/cncf/tag-security/issues/github.com/kubernetes/kubernetes)
##### [TAG Security Reviews](https://github.com/cncf/tag-security/tree/main/assessments/guide)
##### [Kubernetes SIG Security](https://github.com/kubernetes/community/tree/master/sig-security)
##### [#sig-security-assess-capi](https://kubernetes.slack.com/archives/C022K4F2W4W)
##### [#cluster-api](https://kubernetes.slack.com/archives/C8TSNPY4T)
##### [#cluster-api-aws](https://kubernetes.slack.com/archives/CD6U2V71N)
##### [#cluster-api-vsphere](https://kubernetes.slack.com/archives/CKFGK3SSD)
##### [#cluster-api-openstack](https://kubernetes.slack.com/archives/CFKJB65G9)
##### [#cluster-api-baremetal](https://kubernetes.slack.com/archives/CHD49TLE7)
##### [kubernetes-sig-cluster-lifecycle@googlegroups.com](mailto:kubernetes-sig-cluster-lifecycle@googlegroups.com)
##### [kubernetes-sig-security@googlegroups.com](mailto:kubernetes-sig-security@googlegroups.com)
##### [cluster-api-end-user-guide](https://github.com/kubernetes-sigs/cluster-api/issues/6152)