# Reproducing cert-manager's `serviceAccountRef` with Out-of-Cluster Vault Without Setting `token_reviewer_jwt` I created this tutorial to reproduce the scenario (3) in the cert-manager issue [#6150](https://github.com/cert-manager/cert-manager/issues/6150#issuecomment-1733934244). > Tested on macOS with Docker running on a VM created by `limactl`. It should > also work on macOS with Docker Desktop, but also on Linux. The reason we run > Vault in a container instead of locally is because it needs to be on the same > VM as the kind cluster so that cert-manager can reach Vault, and Vault can > reach kube-apiserver. ```bash # yq version 4 or above is required. Here, I use Vault 1.14.2. kind create cluster make -j e2e-setup-certmanager docker run -d --name vault --network=kind --cap-add=IPC_LOCK -p 8200:8200 -e VAULT_DEV_ROOT_TOKEN_ID=root hashicorp/vault server -dev -log-level=debug export VAULT_ADDR=http://localhost:8200 vault login --method=token token=root vault secrets enable pki vault secrets tune -max-lease-ttl=175200h pki vault write pki/root/generate/internal common_name=example.com key_type=ec key_bits=256 ttl=175200h vault write pki/config/urls issuing_certificates="http://vault:8200/v1/pki/ca" vault write pki/roles/vault-issuer allowed_domains=cluster.local allow_subdomains=true max_ttl=48h key_type=ec key_bits=256 vault auth enable kubernetes vault write auth/kubernetes/config \ kubernetes_host=$(kind get kubeconfig --internal | yq '.clusters[0].cluster.server') \ kubernetes_ca_cert=@<(kind get kubeconfig --internal | yq '.clusters[0].cluster.certificate-authority-data' | base64 -d) \ issuer=https://kubernetes.default.svc.cluster.local vault policy write vault-issuer - <<EOF path "pki*" { capabilities = ["read", "list"] } path "pki/roles/vault-issuer" { capabilities = ["create", "update"] } path "pki/sign/vault-issuer" { capabilities = ["create", "update"] } path "pki/issue/vault-issuer" { capabilities = ["create"] } EOF vault write auth/kubernetes/role/vault-issuer \ bound_service_account_names=vault-issuer \ bound_service_account_namespaces=default \ audience=vault://default/vault-issuer \ policies=vault-issuer \ ttl=1m ``` Then, create the issuer and a certificate: ```bash kubectl apply -f- <<EOF apiVersion: v1 kind: ServiceAccount metadata: name: vault-issuer namespace: default --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: vault-issuer namespace: default rules: - apiGroups: [''] resources: ['serviceaccounts/token'] resourceNames: ['vault-issuer'] verbs: ['create'] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: vault-issuer namespace: default subjects: - kind: ServiceAccount name: cert-manager namespace: cert-manager roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: vault-issuer --- apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: vault-issuer namespace: default spec: vault: path: pki/sign/vault-issuer server: http://vault:8200 auth: kubernetes: role: vault-issuer mountPath: /v1/auth/kubernetes serviceAccountRef: name: vault-issuer # ✨ --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: example-com spec: secretName: example-com-tls issuerRef: name: vault-issuer commonName: example.cluster.local dnsNames: - example.cluster.local privateKey: algorithm: ECDSA EOF ``` There is no way to know that the problem comes from an audience mismatch. To "see" the error, I had to dig into the kube-apiserver logs: ``` kubectl logs -n kube-system pods/kube-apiserver-kind-control-plane | grep auth | tail -2 ``` should show > "Unable to authenticate the request" err="[invalid bearer token, token audiences [\"vault://default/vault-issuer\"] is invalid for the target audiences [\"https://kubernetes.default.svc.cluster.local\"]]" > The Vault logs and the CertificateRequest status aren't helpful: ```bash docker logs vault --follow ``` doesn't say anything about the `aud` mismatch: > 2023-09-26T09:53:39.085Z [DEBUG] auth.kubernetes.auth_kubernetes_86f03291: login unauthorized: err="lookup failed: service account unauthorized; this could mean it has been deleted or recreated with a new token" ```bash kubectl get issuer vault-issuer -oyaml | yq '.status.conditions[].message' ``` It doens't say anything about the `aud` mismatch either: > Failed to initialize Vault client: while requesting a Vault token using the Kubernetes auth: error calling Vault server: Error making API request. > > URL: POST http://vault:8200/v1/auth/kubernetes/login. Code: 403. Errors: > > * permission denied ## Testing PR 6718 PR: https://github.com/cert-manager/cert-manager/pull/6718 Do the same as above; but this time, set the audiences: ```yaml apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: vault-issuer namespace: default spec: vault: path: pki/sign/vault-issuer server: http://vault.default.svc.cluster.local:8200 auth: kubernetes: role: vault-issuer mountPath: /v1/auth/kubernetes serviceAccountRef: name: vault-issuer # ✨ audiences: [https://kubernetes.default.svc.cluster.local] ``` To find the audience to be used for your cluster, run the following: ```bash kubectl get --raw /.well-known/openid-configuration | jq .issuer -r ``` We also need to give the permission to perform token reviews to the service account `vault-issuer`. At the moment, it has no role attached to it. ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: cert-manager:vault-issuer roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegator subjects: - kind: ServiceAccount name: vault-issuer namespace: default ``` ## Extra: See the Token Review call that Vault does to Kubernetes I wanted to see exactly what Vault was sending to Kubernetes when performing its token reviews. I also wanted to see all the HTTP calls coming/going from the external Vault and from the Kubernetes API. That's why I use mitmproxy, ktunnel, kubectl-incluster, and telepresence here. Here is what I went with: ```bash # yq version 4 or above is required. Here, I use Vault 1.14.2. kind create cluster telepresence helm install telepresence connect make -j e2e-setup-certmanager vault server -dev -dev-root-token-id=root --log-level=debug -dev-listen-address=:8201 mitmproxy --mode regular@9090 --ssl-insecure -s ~/code/kubectl-incluster/watch-stream.py --set client_certs=$(k incluster --print-client-cert >/tmp/me.pem && echo /tmp/me.pem) --mode reverse:http://localhost:8201@8200 sudo tee -a /etc/hosts <<EOF 127.0.0.1 cm-to-k8s 127.0.0.1 cm-to-vault 127.0.0.1 vault-to-k8s EOF # Open new shell. go run github.com/omrikiei/ktunnel@latest expose cm-to-vault 8200:cm-to-k8s:8200 # Open new shell. ``` Now, let's configure our external Vault: ```bash export VAULT_ADDR=http://localhost:8200 vault login --method=token token=root vault secrets enable pki vault secrets tune -max-lease-ttl=175200h pki vault write pki/root/generate/internal common_name=example.com key_type=ec key_bits=256 ttl=175200h vault write pki/config/urls issuing_certificates="http://vault:8200/v1/pki/ca" vault write pki/roles/vault-issuer allowed_domains=cluster.local allow_subdomains=true max_ttl=48h key_type=ec key_bits=256 vault auth enable kubernetes vault write auth/kubernetes/config \ kubernetes_host=$(kubectl config view --minify --flatten -ojson | jq -r '.clusters[].cluster.server' | sed 's|127.0.0.1|vault-to-k8s|') \ kubernetes_ca_cert=@$HOME/.mitmproxy/mitmproxy-ca-cert.pem \ issuer=https://kubernetes.default.svc.cluster.local vault policy write vault-issuer - <<EOF path "pki*" { capabilities = ["read", "list"] } path "pki/roles/vault-issuer" { capabilities = ["create", "update"] } path "pki/sign/vault-issuer" { capabilities = ["create", "update"] } path "pki/issue/vault-issuer" { capabilities = ["create"] } EOF vault write auth/kubernetes/role/vault-issuer \ bound_service_account_names=vault-issuer \ bound_service_account_namespaces=default \ audience=vault://default/vault-issuer \ policies=vault-issuer \ ttl=1m ``` The token review looks like this: ```http POST /apis/authentication.k8s.io/v1/tokenreviews HTTP/1.1 Host: vault-to-k8s:58484 User-Agent: Go-http-client/1.1 Content-Length: 1050 Accept: application/json Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImJVcHRRcVVrZDVSZ1lnTU1QQmkxSDAzcHh6ZmQ2VURMdE9tX0dKdXl5eGsifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJ2YXVsdDovL2RlZmF1bHQvdmF1bHQtaXNzdWVyIl0sImV4cCI6MTcwNzMxNjE3NiwiaWF0IjoxNzA3MzE1NTc2LCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbCIsImt1YmVybmV0ZXMuaW8iOnsibmFtZXNwYWNlIjoiZGVmYXVsdCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJ2YXVsdC1pc3N1ZXIiLCJ1aWQiOiIyYzEyM2MwOC1lNjhlLTRkMjctOTI4OS05MDBhYjQwZWQ5M2QifX0sIm5iZiI6MTcwNzMxNTU3Niwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6dmF1bHQtaXNzdWVyIn0.ovfxAIbg85zPp3blhfUA50YyQed4TNeVp4iRGdBkmXVcXFIyr0bn7PouGjLbnBGP1jnjpWNvYS97nQbE3n4MRuOgpa21pQS31ws8rJAsUZlGpjAFQ4lTmMht-wu8CS8pOFmyZGrQi0sAHOjD9Bs4_IxfmN8fqCjVUWeOgYNI4DUTfiAQ7QkfzCDtf9HeES6tp0S6W1xk1MOnkOraaRoRrDEFaLNBZ2EQqKMqY4BWcwaB5eVnZdSlAGMI1uTYiJw29vvtoORSlNDMKEXYOF1Ujn7tX6cEG8ySYHaNGVQUH6cuHb-pxnw6D4z55T_KIgIMAcoYGJRE4XHtwAgTp0Imuw Content-Type: application/json Accept-Encoding: gzip {"metadata":{"creationTimestamp":null},"spec":{"token":"eyJhbGciOiJSUzI1NiIsImtpZCI6ImJVcHRRcVVrZDVSZ1lnTU1QQmkxSDAzcHh6ZmQ2VURMdE9tX0dKdXl5eGsifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJ2YXVsdDovL2RlZmF1bHQvdmF1bHQtaXNzdWVyIl0sImV4cCI6MTcwNzMxNjE3NiwiaWF0IjoxNzA3MzE1NTc2LCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbCIsImt1YmVybmV0ZXMuaW8iOnsibmFtZXNwYWNlIjoiZGVmYXVsdCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJ2YXVsdC1pc3N1ZXIiLCJ1aWQiOiIyYzEyM2MwOC1lNjhlLTRkMjctOTI4OS05MDBhYjQwZWQ5M2QifX0sIm5iZiI6MTcwNzMxNTU3Niwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6dmF1bHQtaXNzdWVyIn0.ovfxAIbg85zPp3blhfUA50YyQed4TNeVp4iRGdBkmXVcXFIyr0bn7PouGjLbnBGP1jnjpWNvYS97nQbE3n4MRuOgpa21pQS31ws8rJAsUZlGpjAFQ4lTmMht-wu8CS8pOFmyZGrQi0sAHOjD9Bs4_IxfmN8fqCjVUWeOgYNI4DUTfiAQ7QkfzCDtf9HeES6tp0S6W1xk1MOnkOraaRoRrDEFaLNBZ2EQqKMqY4BWcwaB5eVnZdSlAGMI1uTYiJw29vvtoORSlNDMKEXYOF1Ujn7tX6cEG8ySYHaNGVQUH6cuHb-pxnw6D4z55T_KIgIMAcoYGJRE4XHtwAgTp0Imuw","audiences":["vault://default/vault-issuer"]},"status":{"user":{}}} HTTP/1.1 201 Created Audit-Id: ac6b210f-cb92-46cc-9ca7-6de68a795af2 Cache-Control: no-cache, private Content-Type: application/json X-Kubernetes-Pf-Flowschema-Uid: 6e628e3b-9a22-493a-9eac-72582c8af712 X-Kubernetes-Pf-Prioritylevel-Uid: c8c2f10e-c679-43f0-ab63-56abe4567b6b Date: Wed, 07 Feb 2024 14:19:36 GMT Content-Length: 1586 {"kind":"TokenReview","apiVersion":"authentication.k8s.io/v1","metadata":{"creationTimestamp":null,"managedFields":[{"manager":"Go-http-client","operation":"Update","apiVersion":"authentication.k8s.io/v1","time":"2024-02-07T14:19:36Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:audiences":{},"f:token":{}}}}]},"spec":{"token":"eyJhbGciOiJSUzI1NiIsImtpZCI6ImJVcHRRcVVrZDVSZ1lnTU1QQmkxSDAzcHh6ZmQ2VURMdE9tX0dKdXl5eGsifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJ2YXVsdDovL2RlZmF1bHQvdmF1bHQtaXNzdWVyIl0sImV4cCI6MTcwNzMxNjE3NiwiaWF0IjoxNzA3MzE1NTc2LCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbCIsImt1YmVybmV0ZXMuaW8iOnsibmFtZXNwYWNlIjoiZGVmYXVsdCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJ2YXVsdC1pc3N1ZXIiLCJ1aWQiOiIyYzEyM2MwOC1lNjhlLTRkMjctOTI4OS05MDBhYjQwZWQ5M2QifX0sIm5iZiI6MTcwNzMxNTU3Niwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6dmF1bHQtaXNzdWVyIn0.ovfxAIbg85zPp3blhfUA50YyQed4TNeVp4iRGdBkmXVcXFIyr0bn7PouGjLbnBGP1jnjpWNvYS97nQbE3n4MRuOgpa21pQS31ws8rJAsUZlGpjAFQ4lTmMht-wu8CS8pOFmyZGrQi0sAHOjD9Bs4_IxfmN8fqCjVUWeOgYNI4DUTfiAQ7QkfzCDtf9HeES6tp0S6W1xk1MOnkOraaRoRrDEFaLNBZ2EQqKMqY4BWcwaB5eVnZdSlAGMI1uTYiJw29vvtoORSlNDMKEXYOF1Ujn7tX6cEG8ySYHaNGVQUH6cuHb-pxnw6D4z55T_KIgIMAcoYGJRE4XHtwAgTp0Imuw","audiences":["vault://default/vault-issuer"]},"status":{"authenticated":true,"user":{"username":"system:serviceaccount:default:vault-issuer","uid":"2c123c08-e68e-4d27-9289-900ab40ed93d","groups":["system:serviceaccounts","system:serviceaccounts:default","system:authenticated"]},"audiences":["vault://default/vault-issuer"]}} ```