Does Vault check for the audience when using the Kubernetes auth method?

⚠️ WARNING: ⚠️ this document is mostly wrong. I mixed up the paramater audiences= with the correct audience=. Since Vault doesn't validate parameters (at least not in the Kubernetes plugin), I didn't realize that audiences= was ignored entirely. Thanks to Tom Proctor for finding this out (comment).

tl;dr: Vault 1.12 and below have a bug that prevents checking the JWT audience. The issue is documented in vault-plugin-auth-kubernetes#175. If you configure a Kubernetes auth role with the parameter audience set to vault, that parameter won't do anything. Any audience will be accepted by the role:

vault write auth/kubernetes/role/cert-manager \
    bound_service_account_names=vault \
    bound_service_account_namespaces=cert-manager \
    audience=vault   # <-- ignored 🔥

Context: as part of The Vault issuer can now be given a serviceAccountRef (PR 5502), I had a doubt: does Vault verify the audience (aud value in the JWT) when using the Kubernetes auth method?

The Vault documentation says:

audience (string: "") - Optional Audience claim to verify in the JWT.

This suggests that Vault checks the audience, probably by passing the audience parameter to the TokenReview request. When the parameter audiences is left empty, we can assume that the TokenReview is performed with audiences: [], meaning that the audience is expected to be https://kubernetes.default.svc.cluster.local for example.

The TokenReview documentation makes it clear that in case of an empty audiences array, the aud is matched with the Kubernetes API server's audience:

audiences, string array: Audiences is a list of the identifiers that the resource server presented with the token identifies as. Audience-aware token authenticators will verify that the token was intended for at least one of the audiences in this list. If no audiences are provided, the audience will default to the audience of the Kubernetes apiserver.

To check that the TokenReview API works this way, let us create a token with some audience:

token=$(kubectl create --raw /api/v1/namespaces/default/serviceaccounts/default/token -f- <<EOF | jq -r '.status.token'
{
  "apiVersion": "authentication.k8s.io/v1",
  "kind": "TokenRequest",
  "spec": {
    "audiences": ["vault"]
  }
}
EOF
)

Then, let's review that token with an empty audiences, which reproduces what Vault does when audiences is left empty:

kubectl create --raw /apis/authentication.k8s.io/v1/tokenreviews -f- <<EOF | jq
 {
  "apiVersion": "authentication.k8s.io/v1",
  "kind": "TokenReview",
  "spec": {
    "token": "$token",
    "audiences": []
  }
}
EOF

As suspected, the TokenReview fails:

invalid bearer token, token audiences ["vault"] is invalid for the target audiences ["https://kubernetes.default.svc.cluster.local"].

Now, let's see how Vault behaves when the parameter audiences= isn't used when configuring the role. First, I install Vault:

kind cluster create
helm upgrade --install vault hashicorp/vault --set server.dev.enabled=true --set server.logLevel=debug --set global.tlsDisable=true --wait
kubectl exec vault-0 -i -- vault auth enable kubernetes
kubectl exec vault-0 -i -- sh -c 'vault write auth/kubernetes/config \
   token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
   kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
   kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
   issuer=https://kubernetes.default.svc.cluster.local'

Let's create a role without using the audiences parameter:

kubectl exec vault-0 -i -- vault write auth/kubernetes/role/empty-audience \
   bound_service_account_names=default \
   bound_service_account_namespaces=default

Let's get a token with some audience:

token=$(kubectl create --raw /api/v1/namespaces/default/serviceaccounts/default/token -f- <<EOF | jq -r '.status.token'
{
  "apiVersion": "authentication.k8s.io/v1",
  "kind": "TokenRequest",
  "spec": {"audiences": ["vault"]}
}
EOF
)

From the above results, we should get an error since "no audience in token review call = the audience must be the Kubernetes API server's audience".

But the TokenReview works:

$ kubectl exec vault-0 -i -- vault write auth/kubernetes/login role=empty-audience jwt="$token"
Key                                       Value
---                                       -----
token                                     hvs.CAESICj...kW6jr8rl-G

Now, let's see if a mismatching audience is caught:

kubectl exec vault-0 -i -- vault write auth/kubernetes/role/with-audience \
   bound_service_account_names=default \
   bound_service_account_namespaces=default \
   audiences=mismatch

$ kubectl exec vault-0 -i -- vault write auth/kubernetes/login role=with-audience jwt="$token"
Key                                       Value
---                                       -----
token                                     hvs.CAESIHdhpK5bTrOil

It works too… Vault seems to be ignoring when the audience doesn't match.

Now, let's see what is happening behind the scene using mitmproxy. In one shell session, run mitmproxy:

mitmproxy -p 9090

Let us create a service account that Vault will use to do the TokenReview requests:

kubectl create sa vault
kubectl create clusterrolebinding default:vault --clusterrole=cluster-admin --serviceaccount=default:vault

Let us run Vault in another shell session:

HTTPS_PROXY=:9090 vault server -dev -dev-root-token-id=root --log-level=debug

Now, configure Vault:

export VAULT_ADDR='http://127.0.0.1:8200'
vault auth enable kubernetes
vault write auth/kubernetes/config \
   token_reviewer_jwt="$(kubectl create token vault)" \
   kubernetes_host="$(kubectl config view --minify --flatten -ojson | jq '.clusters[0].cluster.server' -r | sed 's|127.0.0.1|me|')" \
   kubernetes_ca_cert="$(kubectl config view --minify --flatten -ojson | jq '.clusters[0].cluster."certificate-authority-data"' -r | base64 -d)" \
   issuer=https://kubernetes.default.svc.cluster.local

Finally, let us trigger a token review:

vault write auth/kubernetes/role/with-audience \
   bound_service_account_names=default \
   bound_service_account_namespaces=default \
   audiences=mismatch
vault write auth/kubernetes/login role=with-audience \
  jwt="$(kubectl create token -n default default --audience=foobar)"

POST /apis/authentication.k8s.io/v1/tokenreviews HTTP/1.1
Authorization: Bearer abc
Content-Type: application/json

{
  "metadata": { "creationTimestamp": null },
  "spec": {
    "token": "eyJhbGciOiJSUzI1NiIsImtpZCI6Im9SYUp6YTM0LV82dUtBa0ZzYVI5TGIxOHNiRGdSYUhLUG5OSXpPN1gwY1EifQ.eyJhdWQiOlsiZm9vYmFyIl0sImV4cCI6MTY3MDk2Nzk3NiwiaWF0IjoxNjcwOTY0Mzc2LCJpc3MiOiJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbCIsImt1YmVybmV0ZXMuaW8iOnsibmFtZXNwYWNlIjoiZGVmYXVsdCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJkZWZhdWx0IiwidWlkIjoiMWI1YTllMjEtMTE0Yy00YWY1LTg0NTMtYmE4ZWZjODNmY2JmIn19LCJuYmYiOjE2NzA5NjQzNzYsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.RMx_1qireu3Ujn3GoSFktqPMBAUIFEXaCJLl6E9fO0CA8qCXYJoDLsDGg09NZ8Lpf01fETqI6arX6xiT5ElaAv4d0kdla1hx1MZcRGGFsjC1nUldyDaGr43NqZfaKNfva8a9edhdomc7IjGa8efiORcy1jCS6RDeGA3Bxkf81smc-IudjHCUvcco7iNXuIUj7Kugjv6ZVj5kKTJd3VUqyR846XzqHAuXV4L6cFKqbct03O3h4eSjA7yPpUmpAYrDXPjAsVqkbPnrczAylrG7MI7S7iPD5AXv8NZAmKLgBFw10xRErUU1ihvMVcSxPQrlMSoQPqIO7w8FcQlp7LE9Xg",
    "audiences": ["foobar"]
  },
  "status": { "user": {} }
}

The response is positive:

HTTP/1.1 201 Created

As you can see, Vault does not pass the mismatch audience to the TokenReview request. Instead, it copies whatever aud is in the JWT into the TokenReview request.

It seems there is a bug. The code in path_login.go goes something like this:

sa := &serviceAccount{}
_ = mapstructure.Decode(jwtClaims, sa)
r, _ := tr.Review(ctx, client, jwtStr, sa.Audience)

Vault is checking that the audience in the JWT matches the audience in the JWT, instead of checking that the audience in the JWT matches the role's audience.