owned this note
owned this note
Published
Linked with GitHub
# Checking expiry of Secrets in the clusters
## Introduction
`airshipctl secret check-certificate-expiration` checks for all the expirable secret and certificate entities in the cluster and reports back to the operator
Basically, there are three different types of entities identified
1. TLS Secrets in the cluster
2. Workload Cluster kubeconfig secret
3. Workload Cluster node certificates
## Explanation
### TLS Secrets in the cluster
The tls secrets are a type of Kubernetes secrets with 'Type' set as 'kubernetes.io/tls'. This is basically the TLS CA and the resultant certificates related secrets which will be applied/used in the Management and the Workload CAPI clusters
### Workload Cluster kubeconfig secret
This secret gets created while deploying the Workload cluster and is named as `cluster-name`-kubeconfig
The base64 decoded content of the secret value is basically a kubeconfig which is used to connect to the Workload Cluster. The kubeconfig contains the expirable contents of cluster and the user certificate information
### Workload Cluster node certificates
When a Workload Cluster is deployed, control plane nodes and worker nodes gets created. These are nothing but a normal Kubernetes cluster nodes and hence has the certificate content in /etc/kubernetes/pki directory for each node. These certificate's expiry information needs to be identified
## Implementation
`airshipctl secret check-certificate-expiration` command has been introduced to manage all the above 3 conditions.
> Note: The command must be flagged with kubeconfig data to connect to the corresponding cluster
```
ubuntu@airship-f1:$ ./airshipctl cluster -h
This command provides capabilities for interacting with a Kubernetes cluster,
such as getting status and deploying initial infrastructure.
Usage:
airshipctl cluster [command]
Available Commands:
check-certificate-expiration Check for expiring TLS certificates, secrets and kubeconfigs in the kubernetes cluster
init Deploy cluster-api provider components
move Move Cluster API objects, provider specific objects and all dependencies to the target cluster
rotate-sa-token Rotate tokens of Service Accounts
status Retrieve statuses of deployed cluster components
Flags:
-h, --help help for cluster
Global Flags:
--airshipconf string Path to file for airshipctl configuration. (default "$HOME/.airship/config")
--debug enable verbose output
Use "airshipctl cluster [command] --help" for more information about a command.
ubuntu@airship-f1:~/code-sops/airshipctl$ ./airshipctl cluster check-certificate-expiration -h
Checks for the expiring entities in the both the Management and the Workload
Clusters or in a self-managed cluster and reports with the intention on
alerting the operator to take necessary actions. Checks for TLS Secrets,
kubeconf secrets (which gets created while creating the workload cluster) and
also the node certificates present inside /etc/kubernetes/pki directory for
each nodes
Usage:
airshipctl cluster check-certificate-expiration [flags]
Examples:
# To display all the expirying entities in the cluster
airshipctl cluster check-certificate-expiration
# To display the entities whose expiry is within threshold of 30 days
airshipctl cluster check-certificate-expiration -t 30
# To output the contents to json (default operation)
airshipctl cluster check-certificate-expiration -o json
or
airshipctl cluster check-certificate-expiration
# To output the contents to yaml
airshipctl cluster check-certificate-expiration -o yaml
# To output the contents whose expiry is within 30 days to yaml
airshipctl cluster check-certificate-expiration -t 30 -o yaml
Flags:
-h, --help help for check-certificate-expiration
--kubeconfig string Path to kubeconfig associated with cluster being managed
-o, --output string convert output to yaml or json (default "json")
-t, --threshold int The max expiration threshold in days before a secret is expiring. Displays all the secrets by default (default -1)
Global Flags:
--airshipconf string Path to file for airshipctl configuration. (default "$HOME/.airship/config")
--debug enable verbose output
ubuntu@airship-f1:~/code-sops/airshipctl$
```
## Working
### TLS Secrets in the cluster
The command just runs through all the TLS secrets in the cluster and checks if any of the crt is within the duration and if so, displays it
### Workload Cluster kubeconfig secret
The command fetches all the secrets suffixed with `-kubeconfig` and checks its association with a Workload cluster by looking for a kubeadmcontrolplane reference, fetches the 'value' field, base64 decodes it and checks for the certificate expiry
### Workload Cluster node certificates
Communicating and fetching some contents in the Workload nodes is tricky. Hence, an Ansible Operator called HOSTCONFIG operator has been introduced as part of a different work item and the same is leveraged here. The Ansible operator runs like a cron job inside one of the control plane nodes (reconcile period can be set in days), fetches the expiry information using `kubeadm alpha certs check-expiration` command and annotates each node with its certificate-expiry information.`airshipctl secret check-certificate-expiration` just reads the annotation, parses it and displays if the limit is within the duration.
### `airshipctl secret check-certificate-expiration`
``` yaml
./airshipctl cluster check-certificate-expiration --kubeconfig ~/.airship/kubeconfig -o yaml
---
kubeconfs:
- cluster:
- certificateName: CertificateAuthorityData
expiryDate: 2030-09-29 14:17:57 +0000 UTC
name: dtc
secretName: dtc-kubeconfig
secretNamespace: default
user:
- certificateName: ClientCertificateData
expiryDate: 2021-10-01 14:17:58 +0000 UTC
name: dtc-admin
tlsSecrets:
- certificate:
- expiryDate: 2020-12-30 14:16:18 +0000 UTC
name: tls.crt
- expiryDate: 2020-12-30 14:16:18 +0000 UTC
name: ca.crt
name: capd-webhook-service-cert
namespace: capd-system
- certificate:
- expiryDate: 2020-12-30 14:16:17 +0000 UTC
name: tls.crt
- expiryDate: 2020-12-30 14:16:17 +0000 UTC
name: ca.crt
name: capi-kubeadm-bootstrap-webhook-service-cert
namespace: capi-webhook-system
- certificate:
- expiryDate: 2020-12-30 14:16:17 +0000 UTC
name: tls.crt
- expiryDate: 2020-12-30 14:16:17 +0000 UTC
name: ca.crt
name: capi-kubeadm-control-plane-webhook-service-cert
namespace: capi-webhook-system
- certificate:
- expiryDate: 2020-12-30 14:16:16 +0000 UTC
name: tls.crt
- expiryDate: 2020-12-30 14:16:16 +0000 UTC
name: ca.crt
name: capi-webhook-service-cert
namespace: capi-webhook-system
- certificate:
- expiryDate: 2025-09-30 14:16:05 +0000 UTC
name: tls.crt
- expiryDate: 2025-09-30 14:16:05 +0000 UTC
name: ca.crt
name: cert-manager-webhook-ca
namespace: cert-manager
- certificate:
- expiryDate: 2021-10-01 14:16:10 +0000 UTC
name: tls.crt
- expiryDate: 2025-09-30 14:16:05 +0000 UTC
name: ca.crt
name: cert-manager-webhook-tls
namespace: cert-manager
- certificate:
- expiryDate: 2020-10-12 09:29:21 +0000 UTC
name: tls.crt
name: tester
namespace: default
...
```
```
./airshipctl cluster check-certificate-expiration --kubeconfig ~/.airship/kubeconfig
{
"tlsSecrets": [
{
"name": "capd-webhook-service-cert",
"namespace": "capd-system",
"certificate": [
{
"name": "tls.crt",
"expiryDate": "2020-12-30 14:16:18 +0000 UTC"
},
{
"name": "ca.crt",
"expiryDate": "2020-12-30 14:16:18 +0000 UTC"
}
]
},
{
"name": "capi-kubeadm-bootstrap-webhook-service-cert",
"namespace": "capi-webhook-system",
"certificate": [
{
"name": "tls.crt",
"expiryDate": "2020-12-30 14:16:17 +0000 UTC"
},
{
"name": "ca.crt",
"expiryDate": "2020-12-30 14:16:17 +0000 UTC"
}
]
},
{
"name": "capi-kubeadm-control-plane-webhook-service-cert",
"namespace": "capi-webhook-system",
"certificate": [
{
"name": "tls.crt",
"expiryDate": "2020-12-30 14:16:17 +0000 UTC"
},
{
"name": "ca.crt",
"expiryDate": "2020-12-30 14:16:17 +0000 UTC"
}
]
},
{
"name": "capi-webhook-service-cert",
"namespace": "capi-webhook-system",
"certificate": [
{
"name": "tls.crt",
"expiryDate": "2020-12-30 14:16:16 +0000 UTC"
},
{
"name": "ca.crt",
"expiryDate": "2020-12-30 14:16:16 +0000 UTC"
}
]
},
{
"name": "cert-manager-webhook-ca",
"namespace": "cert-manager",
"certificate": [
{
"name": "tls.crt",
"expiryDate": "2025-09-30 14:16:05 +0000 UTC"
},
{
"name": "ca.crt",
"expiryDate": "2025-09-30 14:16:05 +0000 UTC"
}
]
},
{
"name": "cert-manager-webhook-tls",
"namespace": "cert-manager",
"certificate": [
{
"name": "tls.crt",
"expiryDate": "2021-10-01 14:16:10 +0000 UTC"
},
{
"name": "ca.crt",
"expiryDate": "2025-09-30 14:16:05 +0000 UTC"
}
]
},
{
"name": "tester",
"namespace": "default",
"certificate": [
{
"name": "tls.crt",
"expiryDate": "2020-10-12 09:29:21 +0000 UTC"
}
]
}
],
"kubeconfs": [
{
"secretName": "dtc-kubeconfig",
"secretNamespace": "default",
"cluster": [
{
"name": "dtc",
"certificateName": "CertificateAuthorityData",
"expiryDate": "2030-09-29 14:17:57 +0000 UTC"
}
],
"user": [
{
"name": "dtc-admin",
"certificateName": "ClientCertificateData",
"expiryDate": "2021-10-01 14:17:58 +0000 UTC"
}
]
}
]
```