# Reproducing cert-manager's `serviceAccountRef` with Out-of-Cluster Vault Without Setting `token_reviewer_jwt`

I created this tutorial to reproduce the scenario (3) in the cert-manager issue [#6150](https://github.com/cert-manager/cert-manager/issues/6150#issuecomment-1733934244).

> Tested on macOS with Docker running on a VM created by `limactl`. It should
> also work on macOS with Docker Desktop, but also on Linux. The reason we run
> Vault in a container instead of locally is because it needs to be on the same
> VM as the kind cluster so that cert-manager can reach Vault, and Vault can
> reach kube-apiserver.

# yq version 4 or above is required. Here, I use Vault 1.14.2.
kind create cluster
make -j e2e-setup-certmanager
docker run -d --name vault --network=kind --cap-add=IPC_LOCK -p 8200:8200 -e VAULT_DEV_ROOT_TOKEN_ID=root hashicorp/vault server -dev -log-level=debug
export VAULT_ADDR=http://localhost:8200
vault login --method=token token=root
vault secrets enable pki
vault secrets tune -max-lease-ttl=175200h pki
vault write pki/root/generate/internal common_name=example.com key_type=ec key_bits=256 ttl=175200h
vault write pki/config/urls issuing_certificates="http://vault:8200/v1/pki/ca"
vault write pki/roles/vault-issuer allowed_domains=cluster.local allow_subdomains=true max_ttl=48h key_type=ec key_bits=256
vault auth enable kubernetes
vault write auth/kubernetes/config \
   kubernetes_host=$(kind get kubeconfig --internal | yq '.clusters[0].cluster.server') \
   kubernetes_ca_cert=@<(kind get kubeconfig --internal | yq '.clusters[0].cluster.certificate-authority-data' | base64 -d) \
   issuer=https://kubernetes.default.svc.cluster.local

vault policy write vault-issuer - <<EOF
path "pki*"                 { capabilities = ["read", "list"] }
path "pki/roles/vault-issuer"   { capabilities = ["create", "update"] }
path "pki/sign/vault-issuer"    { capabilities = ["create", "update"] }
path "pki/issue/vault-issuer"   { capabilities = ["create"] }
EOF
vault write auth/kubernetes/role/vault-issuer \
   bound_service_account_names=vault-issuer \
   bound_service_account_namespaces=default \
   audience=vault://default/vault-issuer \
   policies=vault-issuer \
   ttl=1m

Then, create the issuer and a certificate:

kubectl apply -f- <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-issuer
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: vault-issuer
  namespace: default
rules:
  - apiGroups: ['']
    resources: ['serviceaccounts/token']
    resourceNames: ['vault-issuer']
    verbs: ['create']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: vault-issuer
  namespace: default
subjects:
  - kind: ServiceAccount
    name: cert-manager
    namespace: cert-manager
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: vault-issuer
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-issuer
  namespace: default
spec:
  vault:
    path: pki/sign/vault-issuer
    server: http://vault:8200
    auth:
      kubernetes:
        role: vault-issuer
        mountPath: /v1/auth/kubernetes
        serviceAccountRef:
          name: vault-issuer # ✨
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-com
spec:
  secretName: example-com-tls
  issuerRef:
    name: vault-issuer
  commonName: example.cluster.local
  dnsNames:
  - example.cluster.local
  privateKey:
    algorithm: ECDSA
EOF

There is no way to know that the problem comes from an audience mismatch. To "see" the error, I had to dig into the kube-apiserver logs:

kubectl logs -n kube-system pods/kube-apiserver-kind-control-plane | grep auth | tail -2

should show

> "Unable to authenticate the request" err="[invalid bearer token, token audiences [\"vault://default/vault-issuer\"] is invalid for the target audiences [\"https://kubernetes.default.svc.cluster.local\"]]"

> The Vault logs and the CertificateRequest status aren't helpful:

docker logs vault --follow

doesn't say anything about the `aud` mismatch:

> 2023-09-26T09:53:39.085Z [DEBUG] auth.kubernetes.auth_kubernetes_86f03291: login unauthorized: err="lookup failed: service account unauthorized; this could mean it has been deleted or recreated with a new token"

kubectl get issuer vault-issuer -oyaml | yq '.status.conditions[].message'

It doens't say anything about the `aud` mismatch either:

> Failed to initialize Vault client: while requesting a Vault token using the Kubernetes auth: error calling Vault server: Error making API request.
>
> URL: POST http://vault:8200/v1/auth/kubernetes/login. Code: 403. Errors:
>
> * permission denied

## Testing PR 6718

PR: https://github.com/cert-manager/cert-manager/pull/6718

Do the same as above; but this time, set the audiences:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-issuer
  namespace: default
spec:
  vault:
    path: pki/sign/vault-issuer
    server: http://vault.default.svc.cluster.local:8200
    auth:
      kubernetes:
        role: vault-issuer
        mountPath: /v1/auth/kubernetes
        serviceAccountRef:
          name: vault-issuer # ✨
        audiences: [https://kubernetes.default.svc.cluster.local]

To find the audience to be used for your cluster, run the following:

kubectl get --raw /.well-known/openid-configuration | jq .issuer -r

We also need to give the permission to perform token reviews to the service account `vault-issuer`. At the moment, it has no role attached to it.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cert-manager:vault-issuer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-issuer
  namespace: default

## Extra: See the Token Review call that Vault does to Kubernetes

I wanted to see exactly what Vault was sending to Kubernetes when performing its token reviews.

I also wanted to see all the HTTP calls coming/going from the external Vault and from the Kubernetes API. That's why I use mitmproxy, ktunnel, kubectl-incluster, and telepresence here.

Here is what I went with:

# yq version 4 or above is required. Here, I use Vault 1.14.2.
kind create cluster
telepresence helm install
telepresence connect
make -j e2e-setup-certmanager
vault server -dev -dev-root-token-id=root --log-level=debug -dev-listen-address=:8201
mitmproxy --mode regular@9090 --ssl-insecure -s ~/code/kubectl-incluster/watch-stream.py --set client_certs=$(k incluster --print-client-cert >/tmp/me.pem && echo /tmp/me.pem) --mode reverse:http://localhost:8201@8200
sudo tee -a /etc/hosts <<EOF
127.0.0.1 cm-to-k8s
127.0.0.1 cm-to-vault
127.0.0.1 vault-to-k8s
EOF
# Open new shell.
go run github.com/omrikiei/ktunnel@latest expose cm-to-vault 8200:cm-to-k8s:8200
# Open new shell.

Now, let's configure our external Vault:

export VAULT_ADDR=http://localhost:8200
vault login --method=token token=root
vault secrets enable pki
vault secrets tune -max-lease-ttl=175200h pki
vault write pki/root/generate/internal common_name=example.com key_type=ec key_bits=256 ttl=175200h
vault write pki/config/urls issuing_certificates="http://vault:8200/v1/pki/ca"
vault write pki/roles/vault-issuer allowed_domains=cluster.local allow_subdomains=true max_ttl=48h key_type=ec key_bits=256
vault auth enable kubernetes
vault write auth/kubernetes/config \
   kubernetes_host=$(kubectl config view --minify --flatten -ojson | jq -r '.clusters[].cluster.server' | sed 's|127.0.0.1|vault-to-k8s|') \
   kubernetes_ca_cert=@$HOME/.mitmproxy/mitmproxy-ca-cert.pem \
   issuer=https://kubernetes.default.svc.cluster.local

vault policy write vault-issuer - <<EOF
path "pki*"                 { capabilities = ["read", "list"] }
path "pki/roles/vault-issuer"   { capabilities = ["create", "update"] }
path "pki/sign/vault-issuer"    { capabilities = ["create", "update"] }
path "pki/issue/vault-issuer"   { capabilities = ["create"] }
EOF
vault write auth/kubernetes/role/vault-issuer \
   bound_service_account_names=vault-issuer \
   bound_service_account_namespaces=default \
   audience=vault://default/vault-issuer \
   policies=vault-issuer \
   ttl=1m

The token review looks like this:

POST /apis/authentication.k8s.io/v1/tokenreviews HTTP/1.1
Host: vault-to-k8s:58484
User-Agent: Go-http-client/1.1
Content-Length: 1050
Accept: application/json
Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6ImJVcHRRcVVrZDVSZ1lnTU1QQmkxSDAzcHh6ZmQ2VURMdE9tX0dKdXl5eGsifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJ2YXVsdDovL2RlZmF1bHQvdmF1bHQtaXNzdWVyIl0sImV4cCI6MTcwNzMxNjE3NiwiaWF0IjoxNzA3MzE1NTc2LCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbCIsImt1YmVybmV0ZXMuaW8iOnsibmFtZXNwYWNlIjoiZGVmYXVsdCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJ2YXVsdC1pc3N1ZXIiLCJ1aWQiOiIyYzEyM2MwOC1lNjhlLTRkMjctOTI4OS05MDBhYjQwZWQ5M2QifX0sIm5iZiI6MTcwNzMxNTU3Niwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6dmF1bHQtaXNzdWVyIn0.ovfxAIbg85zPp3blhfUA50YyQed4TNeVp4iRGdBkmXVcXFIyr0bn7PouGjLbnBGP1jnjpWNvYS97nQbE3n4MRuOgpa21pQS31ws8rJAsUZlGpjAFQ4lTmMht-wu8CS8pOFmyZGrQi0sAHOjD9Bs4_IxfmN8fqCjVUWeOgYNI4DUTfiAQ7QkfzCDtf9HeES6tp0S6W1xk1MOnkOraaRoRrDEFaLNBZ2EQqKMqY4BWcwaB5eVnZdSlAGMI1uTYiJw29vvtoORSlNDMKEXYOF1Ujn7tX6cEG8ySYHaNGVQUH6cuHb-pxnw6D4z55T_KIgIMAcoYGJRE4XHtwAgTp0Imuw
Content-Type: application/json
Accept-Encoding: gzip

{"metadata":{"creationTimestamp":null},"spec":{"token":"eyJhbGciOiJSUzI1NiIsImtpZCI6ImJVcHRRcVVrZDVSZ1lnTU1QQmkxSDAzcHh6ZmQ2VURMdE9tX0dKdXl5eGsifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJ2YXVsdDovL2RlZmF1bHQvdmF1bHQtaXNzdWVyIl0sImV4cCI6MTcwNzMxNjE3NiwiaWF0IjoxNzA3MzE1NTc2LCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbCIsImt1YmVybmV0ZXMuaW8iOnsibmFtZXNwYWNlIjoiZGVmYXVsdCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJ2YXVsdC1pc3N1ZXIiLCJ1aWQiOiIyYzEyM2MwOC1lNjhlLTRkMjctOTI4OS05MDBhYjQwZWQ5M2QifX0sIm5iZiI6MTcwNzMxNTU3Niwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6dmF1bHQtaXNzdWVyIn0.ovfxAIbg85zPp3blhfUA50YyQed4TNeVp4iRGdBkmXVcXFIyr0bn7PouGjLbnBGP1jnjpWNvYS97nQbE3n4MRuOgpa21pQS31ws8rJAsUZlGpjAFQ4lTmMht-wu8CS8pOFmyZGrQi0sAHOjD9Bs4_IxfmN8fqCjVUWeOgYNI4DUTfiAQ7QkfzCDtf9HeES6tp0S6W1xk1MOnkOraaRoRrDEFaLNBZ2EQqKMqY4BWcwaB5eVnZdSlAGMI1uTYiJw29vvtoORSlNDMKEXYOF1Ujn7tX6cEG8ySYHaNGVQUH6cuHb-pxnw6D4z55T_KIgIMAcoYGJRE4XHtwAgTp0Imuw","audiences":["vault://default/vault-issuer"]},"status":{"user":{}}}

HTTP/1.1 201 Created
Audit-Id: ac6b210f-cb92-46cc-9ca7-6de68a795af2
Cache-Control: no-cache, private
Content-Type: application/json
X-Kubernetes-Pf-Flowschema-Uid: 6e628e3b-9a22-493a-9eac-72582c8af712
X-Kubernetes-Pf-Prioritylevel-Uid: c8c2f10e-c679-43f0-ab63-56abe4567b6b
Date: Wed, 07 Feb 2024 14:19:36 GMT
Content-Length: 1586

{"kind":"TokenReview","apiVersion":"authentication.k8s.io/v1","metadata":{"creationTimestamp":null,"managedFields":[{"manager":"Go-http-client","operation":"Update","apiVersion":"authentication.k8s.io/v1","time":"2024-02-07T14:19:36Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{"f:audiences":{},"f:token":{}}}}]},"spec":{"token":"eyJhbGciOiJSUzI1NiIsImtpZCI6ImJVcHRRcVVrZDVSZ1lnTU1QQmkxSDAzcHh6ZmQ2VURMdE9tX0dKdXl5eGsifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJ2YXVsdDovL2RlZmF1bHQvdmF1bHQtaXNzdWVyIl0sImV4cCI6MTcwNzMxNjE3NiwiaWF0IjoxNzA3MzE1NTc2LCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbCIsImt1YmVybmV0ZXMuaW8iOnsibmFtZXNwYWNlIjoiZGVmYXVsdCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJ2YXVsdC1pc3N1ZXIiLCJ1aWQiOiIyYzEyM2MwOC1lNjhlLTRkMjctOTI4OS05MDBhYjQwZWQ5M2QifX0sIm5iZiI6MTcwNzMxNTU3Niwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6dmF1bHQtaXNzdWVyIn0.ovfxAIbg85zPp3blhfUA50YyQed4TNeVp4iRGdBkmXVcXFIyr0bn7PouGjLbnBGP1jnjpWNvYS97nQbE3n4MRuOgpa21pQS31ws8rJAsUZlGpjAFQ4lTmMht-wu8CS8pOFmyZGrQi0sAHOjD9Bs4_IxfmN8fqCjVUWeOgYNI4DUTfiAQ7QkfzCDtf9HeES6tp0S6W1xk1MOnkOraaRoRrDEFaLNBZ2EQqKMqY4BWcwaB5eVnZdSlAGMI1uTYiJw29vvtoORSlNDMKEXYOF1Ujn7tX6cEG8ySYHaNGVQUH6cuHb-pxnw6D4z55T_KIgIMAcoYGJRE4XHtwAgTp0Imuw","audiences":["vault://default/vault-issuer"]},"status":{"authenticated":true,"user":{"username":"system:serviceaccount:default:vault-issuer","uid":"2c123c08-e68e-4d27-9289-900ab40ed93d","groups":["system:serviceaccounts","system:serviceaccounts:default","system:authenticated"]},"audiences":["vault://default/vault-issuer"]}}