# Why the Kubernetes Auth Isn't Appropriate when Operating a Central Vault Instance, and Why the Vault JWT Auth Makes More Sense > TBD: explain how GKE, EKS, AKS, and OpenShift users can find what JWKS endpoint they can use in two steps: > 1. Make sure their JWKS endpoint is exposed to unauthenticated requests. > 2. Find the JWKS endpoint that can be reached outside of the cluster. > > The problem right now is that `k get --raw` gives you the JWKS endpoint as seen from within the cluster rather than the one from outside the cluster: > > ```console > $ k get --raw /.well-known/openid-configuration | jq > { > "issuer": "https://container.googleapis.com/v1/projects/jetstack-mael-valais/locations/europe-west2-b/clusters/test", > "jwks_uri": "https://10.154.0.43:443/openid/v1/jwks", > "response_types_supported": [ > "id_token" > ], > "subject_types_supported": [ > "public" > ], > "id_token_signing_alg_values_supported": [ > "RS256" > ] > } > ``` > Here, https://10.154.0.43:443/openid/v1/jwks isn't accessible from the external Vault instance since Vault doesn't live in the cluster. The IP 10.154.0.43 is a cluster IP. > > To get the outside-of-cluster JWKS endpoint, one has to access the OIDC discovery endpoint instead of `k get --raw`, ie: > > ```bash > curl "https://container.googleapis.com/v1/projects/jetstack-mael-valais/locations/europe-west2-b/clusters/test/.well-known/openid-configuration" > ``` > which shows: > ```json > { > "issuer": "https://container.googleapis.com/v1/projects/jetstack-mael-valais/locations/europe-west2-b/clusters/test", > "jwks_uri": "https://container.googleapis.com/v1/projects/jetstack-mael-valais/locations/europe-west2-b/clusters/test/jwks", > "response_types_supported": [ > "id_token" > ], > "subject_types_supported": [ > "public" > ], > "id_token_signing_alg_values_supported": [ > "RS256" > ], > "claims_supported": [ > "iss", > "sub", > "kubernetes.io" > ], > "grant_types": [ > "urn:kubernetes:grant_type:programmatic_authorization" > ] > } > ``` > > The JWKS endpoint https://container.googleapis.com/v1/projects/jetstack-mael-valais/locations/europe-west2-b/clusters/test/jwks is correct. In this page, I'll first explain why using the Kubernetes auth isn't a good idea when your Vault instance is outside of the Kubernetes cluster in which cert-manager is installed. I will then offer a step-by-step guide on how to use the JWT Auth using the existing Kubernetes Auth API in cert-manager. ## Why You Should Use the JWT Auth Instead of the Kubernetes Auth cert-manager encourages the use of the Kubernetes Auth with Vault. For a long time, this method made sense since the Kubernetes service account tokens (which take the form of JWTs) never expired. Now that Kubernetes has moved to time-bound tokens, checking the revokation status is much less relevant. The main drawback to the Kubernetes Auth method is that Vault needs to be able to connect to the Kubernetes API server. The second drawback is that it requires Vault to know how to authenticate to the Kubernetes API server. As explained in [#6150][], cert-manager requires you to configure the Vault Kubernetes Auth with an old-style static token; the "secretless" way doesn't work. Although it would be possible to implement the secretless mechanism, it doesn't make sense anymore to continue using the Kubernetes Auth. [#6150]: https://github.com/cert-manager/cert-manager/issues/6150#issuecomment-1733934244 Thus, we recommend using the JWT Auth instead of the Kubernetes Auth. The next section will detail how to use the JWT Auth. ## How to Use the JWT Auth using the existing Kubernetes Auth In the remaining of this page, I will explain how to configure cert-manager to use an external Vault instance with the JWT Auth method. The trick is that cert-manager issuer will be using the Kubernetes Auth but we will set the mount path to a JWT Auth method in Vault. It works because both auth methods have the same API. ```yaml apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: vault-issuer spec: vault: path: pki/sign/vault-issuer server: http://vault:8200 auth: kubernetes: role: kind mountPath: /v1/auth/jwt # <--- ✨✨✨ serviceAccountRef: name: vault-issuer ``` Let's start with creating a Kind cluster and installing cert-manager. ```bash kind create cluster helm upgrade --install cert-manager jetstack/cert-manager --namespace cert-manager --set installCRDs=true --create-namespace --wait ``` Our Vault instance is "external" to the cluster. I decided to run it in a container to prove that it doesn't run inside the cluster: ```bash docker run -d --name vault --network=kind --cap-add=IPC_LOCK -p 8200:8200 -e VAULT_DEV_ROOT_TOKEN_ID=root -e VAULT_ADDR=http://vault:8200 hashicorp/vault server -dev -log-level=debug ``` By default, Kind clusters don't have their OIDC discovery endpoint available to unauthenticated users, but actual production Kubernetes clusters will be available by default. So let's fix that in our Kind cluster: ```bash kubectl create clusterrolebinding oidc-reviewer --clusterrole=system:service-account-issuer-discovery --group=system:unauthenticated ``` Let's make sure Vault can trust the Kubernetes API server server certificate: ```bash docker exec -i vault tee /opt/cacrt < \ <(kubectl config view --minify --flatten -ojson | jq -r '.clusters[].cluster."certificate-authority-data"' | base64 -d) ``` ```bash docker exec -i vault vault login --method=token token=root docker exec -i vault vault auth enable jwt docker exec -i vault vault write auth/jwt/config \ jwks_url="https://172.18.0.2:6443/openid/v1/jwks" \ jwks_ca_pem=@/opt/cacrt ``` ```bash docker exec -i vault vault policy write vault-issuer - <<EOF path "pki*" { capabilities = ["read", "list"] } path "pki/roles/vault-issuer" { capabilities = ["create", "update"] } path "pki/sign/vault-issuer" { capabilities = ["create", "update"] } path "pki/issue/vault-issuer" { capabilities = ["create"] } EOF docker exec -i vault vault write auth/jwt/role/kind \ policies="vault-issuer" \ role_type="jwt" \ bound_subject="system:serviceaccount:default:vault-issuer" \ bound_audiences="vault://default/vault-issuer" \ user_claim="sub" \ ttl="1h" ``` Now, let's check that it works: ```console $ docker exec -i vault vault write auth/jwt/login jwt="$(kubectl create token --audience=vault -n cert-manager cert-manager)" role=kind Key Value --- ----- token hvs.CAESIDvqKcqu50rSmYw3XYlGmLcZvhwyKj8sIFT0mj3VERDjGh4KHGh2cy5BUm9qaGNZRnFqMTU1QkRNbHlSV1NENWQ token_accessor LcjJ931UrtG0WFW9st9Ko6Yy token_duration 1h token_renewable true token_policies ["default" "vault-issuer"] identity_policies [] policies ["default" "vault-issuer"] token_meta_role kind ``` Finally, let's configure a Vault issuer: ```bash docker exec -i vault vault secrets enable pki docker exec -i vault vault secrets tune -max-lease-ttl=175200h pki docker exec -i vault vault write pki/root/generate/internal common_name=example.com key_type=ec key_bits=256 ttl=175200h docker exec -i vault vault write pki/config/urls \ issuing_certificates="http://vault:8200/v1/pki/ca" \ url_distribution_points="http://vault:8200/v1/pki/crl" docker exec -i vault vault write pki/roles/vault-issuer \ allowed_domains=cluster.local \ allow_subdomains=true max_ttl=48h \ key_type=ec key_bits=256 ``` ```bash kubectl apply -f- <<EOF apiVersion: v1 kind: ServiceAccount metadata: name: vault-issuer --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: vault-issuer rules: - apiGroups: [''] resources: ['serviceaccounts/token'] resourceNames: ['vault-issuer'] verbs: ['create'] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: vault-issuer subjects: - kind: ServiceAccount name: cert-manager namespace: cert-manager roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: vault-issuer --- apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: vault-issuer spec: vault: path: pki/sign/vault-issuer server: http://vault.default.svc.cluster.local:8200 auth: kubernetes: role: kind mountPath: /v1/auth/jwt serviceAccountRef: name: vault-issuer --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: example-com spec: secretName: example-com-tls issuerRef: name: vault-issuer commonName: example.cluster.local dnsNames: - example.cluster.local privateKey: algorithm: ECDSA EOF ``` And it should work: ``` $ kubectl get cert NAME READY SECRET AGE example-com True example-com-tls 66s ``` ## Debugging `error validating token: invalid subject (sub) claim` I haven't found any way to make these error messages more explicit... - `-log-level=debug` doesn't display anything when someone hits `/auth/jwt/login`. - [`verbose_oidc_logging=true`](https://developer.hashicorp.com/vault/api-docs/auth/jwt#verbose_oidc_logging) doesn't show anything either.
Last changed by  
 
Maël Valais
130

Read more

Fixing etcd NOSPACE (kind cluster)

docker exec -it kind-control-plane sh -c 'crictl ps -name etcd -o json | jq "."'

4/26/2024
Reproducing cert-manager’s serviceAccountRef with Out-of-Cluster Vault Without Setting token_reviewer_jwt

Tested on macOS with Docker running on a VM created by limactl. It shouldalso work on macOS with Docker Desktop, but also on Linux. The reason we runVault in a container instead of locally is because it needs to be on the sameVM as the kind cluster so that cert-manager can reach Vault, and Vault canreach kube-apiserver.

2/7/2024
Fixing “403 Forbidden” GitHub Actions, Docker or Helm Push to GitHub Container Registry (GHCR)

The first time, the job will work.

12/15/2023
Fixing “500 Certificate in an error state. Fix any errors, and then click Retry.” Blocking Renewal In cert-manager

The problem: When enrolling a new certificate, for example by running vcert enroll or when using cert-manager, people get “stuck” with an error of the like:

10/6/2023
Read more from Maël Valais
Published on HackMD