---
# System prepended metadata

title: CertificateSecretWillExpireInLessThanTwoWeeks

---

---
title: "CertificateSecretWillExpireInLessThanTwoWeeks"
description: >
  Ops recipe for expiring cert-manager secrets.
---

## Background

You're here because you got paged for an expiring TLS certificate; some background
is useful to understand how this can happen:

- there is a bug in `cert-manager` which means that it isn't automatically renewing
the certificate.
   - this is the cause of [this PM](https://github.com/giantswarm/giantswarm/issues/13108).
- the resource which created the cert has been removed and the secret wasn't
cleaned up afterwards.
   - older versions of `cert-manager-app` would not automatically delete secrets
when the parent resource (an Ingress or Certificate) was deleted.

## Establish the root cause

Figuring out which of the above scenarios is correct is important as it will mean
that the steps required to resolve the issue are different.

### Check the expiring certificate

You can get the expiry date of the TLS cert thusly:

```bash
export SECRET=<certificate-secret-to-check>
$ kubectl get secret ${SECRET} -o jsonpath="{.data['tls\.crt']}" \
    | base64 -d | openssl x509 -noout -enddate
```

Example output:
```bash
notAfter=Apr 29 10:18:50 2021 GMT
```

If the certificate expiry date is more than two weeks, it's possible that the alert
is actually being caused by a `ca.crt` cert which is also stored in the same secret
(this is unusual, but possible). Check this too (if the secret contains one):

```bash
export SECRET=<certificate-secret-to-check>
$ kubectl get secret ${SECRET} -o jsonpath="{.data['ca\.crt']}" \
    | base64 -d | openssl x509 -noout -enddate
```

Example output:
```bash
notAfter=Apr 29 10:18:50 2021 GMT
```

### Check ingresses

Ingresses which use a TLS certificate will reference the secret name in their
configuration. A quick and dirty way to check if the expiring secret is in use by
an ingress is to do the following (assuming the expiring cert is stored in the
secret `happa-certs`):

```bash
export SECRET=happa-certs
$ kubectl get ingress -A -o yaml | grep secretName | grep ${SECRET}
```

Example output:
```bash
secretName: happa-certs
```

If this command returns output like the above (a `secretName` key) then it shows
that it is currently being consumed by an ingress.

### Check certificates

Certificate secrets are usually named the same as the parent Certificate resource.
First check if this returns any Certificates:

```bash
export SECRET=<certificate-secret-to-check>
$ kubectl get certificate -A | grep ${SECRET}
```

Example output:
```bash
giantswarm   happa-certs       True    happa-certs               136d
```

If this command returns output like the above then it shows that it is currently
being consumed by a Certificate.

If this returns nothing, you should also check if there is a Certificate under
a different name which references the secret (this is unlikely, but possible):

```bash
export SECRET=<certificate-secret-to-check>
$ kubectl get certificate -A -o yaml | grep secretName | grep ${SECRET}
```

Example output:
```bash
secretName: happa-certs
```

## Resolution

### Secret is not being consumed

If all of the above commands return nothing, it is likely that the Secret is no
longer being reconciled. This happens when an app is deployed and then removed,
or if the certificate was renamed without the secret being deleted; basically no
cleanup was carried out.

In this case, it _should_ be safe to just delete the secret, but this should be
checked thoroughly before being carried out - do not just blindly delete it. That
being said, if it _is_ actually being reconciled by `cert-manager`, then it will
be recreated.

### Secret is being consumed

If there is a parent resource which references the secret, then it is being used
and care must be taken to resolve the situation without affecting cluster users.

#### Deleting the secret

It is possible to just remove the secret and allow `cert-manager` to recreate it,
but this will cause some period of outage for the resources which rely on it.
Additionally, if there is an issue with `cert-manager` which stops it recreating
the certificate then the disruption will be greater.

**Use this method very carefully**.

Before deleting the secret, follow the logs of the `cert-manager-controller` pod
which usually lives in `kube-system`. This will allow you to see any issues which
may arise.

#### Force `cert-manager` to re-issue the cert

Updating the `Certificate` resource will usually kick `cert-manager` into reconciling
the resource again. To do this, patch the Certificate to force a renewal:

```bash
export SECRET=<certificate-secret-to-check>
$ kubectl patch certificate app-operator-2-7-0-ingress --type='json' \
    -p='[{"op": "add", "path": "/spec/renewBefore", "value": "1440h"}]'
```

Example output:
```bash
certificate.cert-manager.io/happa-certs patched
```

Following the `cert-manager-controller` logs and checking the `Certificate` status
field should show the cert being re-issued:

```yaml
  - lastTransitionTime: "2021-01-29T11:18:24Z"
    message: Renewing certificate as renewal was scheduled at 2021-01-14 05:20:44
      +0000 UTC
    reason: Renewing
    status: "True"
    type: Issuing
```

If this is successful, then remove the field you just added:

```bash
$ kubectl patch certificate app-operator-2-7-0-ingress --type='json' \
    -p='[{"op": "remove", "path": "/spec/renewBefore"}]'
```

Example output:
```bash
certificate.cert-manager.io/happa-certs patched
```