changed 4 years ago  
Linked with GitHub

Checking expiry of Secrets in the clusters

Introduction

airshipctl secret check-certificate-expiration checks for all the expirable secret and certificate entities in the cluster and reports back to the operator

Basically, there are three different types of entities identified

  1. TLS Secrets in the cluster
  2. Workload Cluster kubeconfig secret
  3. Workload Cluster node certificates

Explanation

TLS Secrets in the cluster

The tls secrets are a type of Kubernetes secrets with 'Type' set as 'kubernetes.io/tls'. This is basically the TLS CA and the resultant certificates related secrets which will be applied/used in the Management and the Workload CAPI clusters

Workload Cluster kubeconfig secret

This secret gets created while deploying the Workload cluster and is named as cluster-name-kubeconfig

The base64 decoded content of the secret value is basically a kubeconfig which is used to connect to the Workload Cluster. The kubeconfig contains the expirable contents of cluster and the user certificate information

Workload Cluster node certificates

When a Workload Cluster is deployed, control plane nodes and worker nodes gets created. These are nothing but a normal Kubernetes cluster nodes and hence has the certificate content in /etc/kubernetes/pki directory for each node. These certificate's expiry information needs to be identified

Implementation

airshipctl secret check-certificate-expiration command has been introduced to manage all the above 3 conditions.

Note: The command must be flagged with kubeconfig data to connect to the corresponding cluster

ubuntu@airship-f1:$ ./airshipctl cluster -h
This command provides capabilities for interacting with a Kubernetes cluster,
such as getting status and deploying initial infrastructure.

Usage:
  airshipctl cluster [command]

Available Commands:
  check-certificate-expiration Check for expiring TLS certificates, secrets and kubeconfigs in the kubernetes cluster
  init                         Deploy cluster-api provider components
  move                         Move Cluster API objects, provider specific objects and all dependencies to the target cluster
  rotate-sa-token              Rotate tokens of Service Accounts
  status                       Retrieve statuses of deployed cluster components

Flags:
  -h, --help   help for cluster

Global Flags:
      --airshipconf string   Path to file for airshipctl configuration. (default "$HOME/.airship/config")
      --debug                enable verbose output

Use "airshipctl cluster [command] --help" for more information about a command.

ubuntu@airship-f1:~/code-sops/airshipctl$ ./airshipctl cluster check-certificate-expiration -h
Checks for the expiring entities in the both the Management and the Workload
Clusters or in a self-managed cluster and reports with the intention on
alerting the operator to take necessary actions. Checks for TLS Secrets,
kubeconf secrets (which gets created while creating the workload cluster) and
also the node certificates present inside /etc/kubernetes/pki directory for
each nodes

Usage:
  airshipctl cluster check-certificate-expiration [flags]

Examples:

# To display all the expirying entities in the cluster
airshipctl cluster check-certificate-expiration

# To display the entities whose expiry is within threshold of 30 days
airshipctl cluster check-certificate-expiration -t 30

# To output the contents to json (default operation)
airshipctl cluster check-certificate-expiration -o json
or
airshipctl cluster check-certificate-expiration

# To output the contents to yaml
airshipctl cluster check-certificate-expiration -o yaml

# To output the contents whose expiry is within 30 days to yaml
airshipctl cluster check-certificate-expiration -t 30 -o yaml


Flags:
  -h, --help                help for check-certificate-expiration
      --kubeconfig string   Path to kubeconfig associated with cluster being managed
  -o, --output string       convert output to yaml or json (default "json")
  -t, --threshold int       The max expiration threshold in days before a secret is expiring. Displays all the secrets by default (default -1)

Global Flags:
      --airshipconf string   Path to file for airshipctl configuration. (default "$HOME/.airship/config")
      --debug                enable verbose output
ubuntu@airship-f1:~/code-sops/airshipctl$

Working

TLS Secrets in the cluster

The command just runs through all the TLS secrets in the cluster and checks if any of the crt is within the duration and if so, displays it

Workload Cluster kubeconfig secret

The command fetches all the secrets suffixed with -kubeconfig and checks its association with a Workload cluster by looking for a kubeadmcontrolplane reference, fetches the 'value' field, base64 decodes it and checks for the certificate expiry

Workload Cluster node certificates

Communicating and fetching some contents in the Workload nodes is tricky. Hence, an Ansible Operator called HOSTCONFIG operator has been introduced as part of a different work item and the same is leveraged here. The Ansible operator runs like a cron job inside one of the control plane nodes (reconcile period can be set in days), fetches the expiry information using kubeadm alpha certs check-expiration command and annotates each node with its certificate-expiry information.airshipctl secret check-certificate-expiration just reads the annotation, parses it and displays if the limit is within the duration.

airshipctl secret check-certificate-expiration

./airshipctl cluster check-certificate-expiration --kubeconfig ~/.airship/kubeconfig -o yaml
---
kubeconfs:
- cluster:
  - certificateName: CertificateAuthorityData
    expiryDate: 2030-09-29 14:17:57 +0000 UTC
    name: dtc
  secretName: dtc-kubeconfig
  secretNamespace: default
  user:
  - certificateName: ClientCertificateData
    expiryDate: 2021-10-01 14:17:58 +0000 UTC
    name: dtc-admin
tlsSecrets:
- certificate:
  - expiryDate: 2020-12-30 14:16:18 +0000 UTC
    name: tls.crt
  - expiryDate: 2020-12-30 14:16:18 +0000 UTC
    name: ca.crt
  name: capd-webhook-service-cert
  namespace: capd-system
- certificate:
  - expiryDate: 2020-12-30 14:16:17 +0000 UTC
    name: tls.crt
  - expiryDate: 2020-12-30 14:16:17 +0000 UTC
    name: ca.crt
  name: capi-kubeadm-bootstrap-webhook-service-cert
  namespace: capi-webhook-system
- certificate:
  - expiryDate: 2020-12-30 14:16:17 +0000 UTC
    name: tls.crt
  - expiryDate: 2020-12-30 14:16:17 +0000 UTC
    name: ca.crt
  name: capi-kubeadm-control-plane-webhook-service-cert
  namespace: capi-webhook-system
- certificate:
  - expiryDate: 2020-12-30 14:16:16 +0000 UTC
    name: tls.crt
  - expiryDate: 2020-12-30 14:16:16 +0000 UTC
    name: ca.crt
  name: capi-webhook-service-cert
  namespace: capi-webhook-system
- certificate:
  - expiryDate: 2025-09-30 14:16:05 +0000 UTC
    name: tls.crt
  - expiryDate: 2025-09-30 14:16:05 +0000 UTC
    name: ca.crt
  name: cert-manager-webhook-ca
  namespace: cert-manager
- certificate:
  - expiryDate: 2021-10-01 14:16:10 +0000 UTC
    name: tls.crt
  - expiryDate: 2025-09-30 14:16:05 +0000 UTC
    name: ca.crt
  name: cert-manager-webhook-tls
  namespace: cert-manager
- certificate:
  - expiryDate: 2020-10-12 09:29:21 +0000 UTC
    name: tls.crt
  name: tester
  namespace: default
...

./airshipctl cluster check-certificate-expiration --kubeconfig ~/.airship/kubeconfig
{
    "tlsSecrets": [
        {
            "name": "capd-webhook-service-cert",
            "namespace": "capd-system",
            "certificate": [
                {
                    "name": "tls.crt",
                    "expiryDate": "2020-12-30 14:16:18 +0000 UTC"
                },
                {
                    "name": "ca.crt",
                    "expiryDate": "2020-12-30 14:16:18 +0000 UTC"
                }
            ]
        },
        {
            "name": "capi-kubeadm-bootstrap-webhook-service-cert",
            "namespace": "capi-webhook-system",
            "certificate": [
                {
                    "name": "tls.crt",
                    "expiryDate": "2020-12-30 14:16:17 +0000 UTC"
                },
                {
                    "name": "ca.crt",
                    "expiryDate": "2020-12-30 14:16:17 +0000 UTC"
                }
            ]
        },
        {
            "name": "capi-kubeadm-control-plane-webhook-service-cert",
            "namespace": "capi-webhook-system",
            "certificate": [
                {
                    "name": "tls.crt",
                    "expiryDate": "2020-12-30 14:16:17 +0000 UTC"
                },
                {
                    "name": "ca.crt",
                    "expiryDate": "2020-12-30 14:16:17 +0000 UTC"
                }
            ]
        },
        {
            "name": "capi-webhook-service-cert",
            "namespace": "capi-webhook-system",
            "certificate": [
                {
                    "name": "tls.crt",
                    "expiryDate": "2020-12-30 14:16:16 +0000 UTC"
                },
                {
                    "name": "ca.crt",
                    "expiryDate": "2020-12-30 14:16:16 +0000 UTC"
                }
            ]
        },
        {
            "name": "cert-manager-webhook-ca",
            "namespace": "cert-manager",
            "certificate": [
                {
                    "name": "tls.crt",
                    "expiryDate": "2025-09-30 14:16:05 +0000 UTC"
                },
                {
                    "name": "ca.crt",
                    "expiryDate": "2025-09-30 14:16:05 +0000 UTC"
                }
            ]
        },
        {
            "name": "cert-manager-webhook-tls",
            "namespace": "cert-manager",
            "certificate": [
                {
                    "name": "tls.crt",
                    "expiryDate": "2021-10-01 14:16:10 +0000 UTC"
                },
                {
                    "name": "ca.crt",
                    "expiryDate": "2025-09-30 14:16:05 +0000 UTC"
                }
            ]
        },
        {
            "name": "tester",
            "namespace": "default",
            "certificate": [
                {
                    "name": "tls.crt",
                    "expiryDate": "2020-10-12 09:29:21 +0000 UTC"
                }
            ]
        }
    ],
    "kubeconfs": [
        {
            "secretName": "dtc-kubeconfig",
            "secretNamespace": "default",
            "cluster": [
                {
                    "name": "dtc",
                    "certificateName": "CertificateAuthorityData",
                    "expiryDate": "2030-09-29 14:17:57 +0000 UTC"
                }
            ],
            "user": [
                {
                    "name": "dtc-admin",
                    "certificateName": "ClientCertificateData",
                    "expiryDate": "2021-10-01 14:17:58 +0000 UTC"
                }
            ]
        }
    ]
Select a repo