---
tags: api-design
---

# API Design Proposal: Impersonation Proxy

This document is an informal design document for API changes related to the new "impersonation proxy" functionality of the Pinniped concierge component.

## Goals

- Allow everything to work out of the box in as many cases as possible
- Eventually support full configuration of all proxy-related behavior
- Deprecate and later eliminate the singleton CredentialIssuer resource.

## Non-Goals

- Add all possible configuration options in the first changeset.

## Changes

1. **Start detecting managed clusters and autocreate a Service and TLS CA for impersonation proxy (v0.6.0)**

In v0.6.0, add some checks to detect when the concierge is running in a managed cluster environment (EKS/AKS/GKE to start).
In these environments:
- Generate and persist a certificate authority for serving the impersonation proxy endpoint.
- Create a Service of `type: LoadBalancer` and load its ExternalName. This should not require any other specific annotations.
- Issue a serving certificate matching the ExternalName and use it to serve the impersonation proxy.
- Update the authenticator CRD status fields (described below).

1. **Add concierge status information to authenticator CRDs (v0.6.0)**

This is a new set of status fields on the WebhookTokenAuthenticator and JWTAuthenticator CRDs.
The new status information describes how the concierge is exposing that particular authenticator.
Currently, in the case where you have multiple authenticators, this information would be mirrored across all of them.

    apiVersion: authentication.concierge.pinniped.dev/v1alpha1
    kind: WebhookAuthenticator
    metadata:
      name: my-webhook
      namespace: pinniped-concierge
    spec:
      endpoint: https://mywebhook.example.com/auth/token
      tls:
        certificateAuthorityData: LS0[...]
    status:
      conciergeStrategies:
      - type: TokenCredentialRequestAPI
        # basic status to tell if this is functioning
        status: Success
        reason: FetchedKey
        message: Key was fetched successfully
        lastUpdateTime: "2021-02-02T19:08:25Z"

        # connection details for connecting to the TokenCredentialRequest API
        # "tagged union" style fields, this field would be expected only on type: TokenCredentialRequestAPI
        tokenCredentialRequestConfig:
          endpoint: https://1.2.3.4:4321/ # Kubernetes API endpoint
          certificateAuthorityData: "LS0tL[...]" # Kubernetes API CA

      - type: ImpersonationProxy
        # basic status to tell if this is functioning
        status: Success
        reason: Listening
        message: Listening on TCP 6444
        lastUpdateTime: "2021-02-02T19:08:25Z"
        
        # connection details for connecting to the impersonation proxy
        # "tagged union" style fields, this field would be expected only on type: ImpersonationProxy
        impersonationProxyConfig:
          endpoint: https://proxy.example.com:6444 # External name of impersonation proxy
          certificateAuthorityData: "LS0tL[...]" # CA bundle for impersonation proxy

Note that the `TokenCredentialRequestAPI` status entry can be trivially mirrored into the existing CredentialIssuer singleton API for backwards compatibility.

1. **Deprecate the CredentialIssuer API (v0.6.0)**

Now that status information is available on the authenticator objects, we no longer need to maintain the CredentialIssuer API. We can begin the deprecation process in v0.6.0 and remove it after a suitable grace period (perhaps v0.8.0?).

1. **Add a new concierge configuration section for proxy settings (after v0.6.0)**

This is a new section in the `concierge.Config` struct which is loaded via ConfigMap:

    # existing configuration
    # [...]

    # new section (exact details to be decided in a future design)
    impersonationProxy: 
      #mode: enable # always run the proxy
      #mode: disable # never run the proxy
      mode: auto # (default) run the proxy when we detect a managed cluster
      
      # specify the external name that will route to the impersonation proxy port
      # (future) if empty, automatically provision a LoadBalancer service and use its ExternalName
      endpoint: https://proxy.example.com/
      
      tls:
        # specify the CA that clients should validate with when connecting to the endpoint
        # (default) if not specified, generate a CA and persist it in a Secret with some rotation period
        certificateAuthorityData: "[...]"

        # specify a secret with type "kubernetes.io/tls" which will be used to serve the endpoint
        # If empty, issue a certificate using the generated CA above.
        # Specifying certificateAuthorityData but not secretName is not valid.
        secretName: my-tls-certificate-and-key

Note that the default values here are valid and enable a fully automatic configuration that should work on a large proportion of clusters.

The v0.6.0 release would not have any of these fields, but its defaults would correspond roughly to a future version configured with:

    impersonationProxy: 
      mode: auto

A future configuration could also allow the "certificate agent" functionality to be disabled by default on managed clusters or be manually disabled, for example:

    kubeCertAgent:
      mode: enabled # (default, current behavior as of v0.4.1)
      mode: auto # (enable only on detected self-hosted clusters)
      mode: disabled # (always disabled)
      

### Examples

1. A cluster with self-hosted control plane (functioning TokenCredentialRequest API):

    apiVersion: authentication.concierge.pinniped.dev/v1alpha1
    kind: WebhookAuthenticator
    metadata:
      name: my-webhook
      namespace: pinniped-concierge
    spec:
      endpoint: https://mywebhook.example.com/auth/token
      tls:
        certificateAuthorityData: LS0[...]
    status:
      conciergeStrategies:
      - type: TokenCredentialRequestAPI
        status: Success
        reason: FetchedKey
        message: Key was fetched successfully
        lastUpdateTime: "2021-02-02T19:08:25Z"
        tokenCredentialRequestConfig:
          endpoint: https://1.2.3.4:4321/
          certificateAuthorityData: "LS0tL[...]"
      - type: ImpersonationProxy
        status: NotAvailable
        reason: DisabledByDefault
        message: Not running because no managed cloud environment was detected
        lastUpdateTime: "2021-02-02T19:08:25Z"

1. A managed cluster (non-functioning TokenCredentialRequest API):

    apiVersion: authentication.concierge.pinniped.dev/v1alpha1
    kind: WebhookAuthenticator
    metadata:
      name: my-webhook
      namespace: pinniped-concierge
    spec:
      endpoint: https://mywebhook.example.com/auth/token
      tls:
        certificateAuthorityData: LS0[...]
    status:
      conciergeStrategies:
      - type: TokenCredentialRequestAPI
        status: Error
        reason: CouldNotFetchKey
        message: Failed to schedule on control plane node
        lastUpdateTime: "2021-02-02T19:08:25Z"
      - type: ImpersonationProxy
        status: Success
        reason: Listening
        message: Listening on TCP 6444
        lastUpdateTime: "2021-02-02T19:08:25Z"
        impersonationProxyConfig:
          endpoint: https://proxy.example.com:6444
          certificateAuthorityData: "LS0tL[...]"

## Addendum: Updated Design Including Implementation Details

Note that after adding the example below, we later decided to simplify the implementation. What is shown below is left here to illustrate an intermediate idea, not the final design. We're not going to use so many controllers as described in the comments below, and we're not going to add `status.generateCerts` or `status.tlsCertSecretName` as shown below, and the `spec` shown below has been moved to a ConfigMap.

---
kind: CredentialIssuer # Could go elsewhere, but CredentialIssuer seems perfect.
apiVersion: whatever
metadata:
   name: some-name
   namespace: pinniped-concierge

spec:
 impersonationProxy:
   #mode: enable # always run the proxy
   #mode: disable # never run the proxy
   mode: auto # (default) run the proxy when we auto-detect a managed cluster

   # Specify the external name that will route to the impersonation proxy port.
   # (default) If empty, automatically provision a LoadBalancer service and
   # use its ExternalName.
   endpoint: https://proxy.example.com:8443/

   # (default) If not specified, auto-generate CA and TLS certs.
   # If specified, must specify both fields.
   tls:
     # Specify the CA that clients should validate with when connecting to the
     # endpoint.
     certificateAuthoritySecretName: my-ca-crt

     # Specify a secret with type "kubernetes.io/tls" which will be used to
     # serve the endpoint.
     tlsSecretName: my-tls-certificate-and-key

status:
  # This field is filled in by our `ProxyConfig` controller and will either be:
  # a copy of the `endpoint` spec,
  # or a copy of the relevant status of the auto-created LoadBalancer Service,
  # or nil when the proxy is not going to run,
  # or nil when the user specified their own cert.
  #
  # A `CertGenerator` controller will watch this and will generate these certs.
  #
  # A `CertsExpirer` controller will watch the generated secret and delete
  # it when it expires, which will force the `CertGenerator` controller to
  # re-create it.
  #
  # A human could force new certs by manually deleting one or both of these
  # secrets, which will force the `CertGenerator` controller to re-create
  # one or both.
  generateCerts:
    certificateAuthoritySecretName: where-to-put-the-ca-cert-and-key
    tlsSecretName: where-to-put-the-tls-cert-and-key
    subjectAltName:
      hostname: proxy.example.com
      ip: 1.2.3.4
      # TODO: do we care about port numbers for TLS certs?

  # This field is filled in by our `ProxyConfig` controller and will either be:
  # a copy of the `spec.tls.tlsSecretName`,
  # or will be the name of an auto-generated secret which would be the same
  # value as `status.generateCerts.tlsSecretName`.
  #
  # A `CertsObserver` controller will watch this field and will load the
  # TLS certs from the specified secret into the in-memory cache used by
  # the running proxy server.
  tlsCertSecretName: my-tls-certificate-and-key

  # An `AuthenticatorStatusSync` controller watches the `strategies` part
  # of the status and copies it to all of the Authenticators,
  # including when a new Authenticator is created.
  strategies:
    # This whole array entry is filled in by our `ProxyConfig` controller.
    # Other array entries regarding other strategy types can be filled by
    # other controller who manage those strategies.
    # The `ProxyConfig` controller will also start or stop the https
    # port and handlers, so it will know if it was successful in opening
    # the port.
    - type: ImpersonationProxy
      status: Success
      reason: Listening
      message: Listening on TCP 6444
      lastUpdateTime: "2021-02-02T19:08:25Z"
      # This data is for cliet discovery...
      impersonationProxyConfig:
        # The `ProxyConfig` controller is watching the `endpoint` spec field
        # and the LoadBalancer Service (if one was created), so it knows the
        # endpoint url either way.
        endpoint: https://proxy.example.com:6444 # External name of impersonation proxy
        # The `ProxyConfig` controller would need to watch either the Secret
        # from `status.generateCerts.certificateAuthoritySecretName` or from
        # `spec.tls.certificateAuthoritySecretName` to know the CA cert data,
        # and also to know when it changes.
        certificateAuthorityData: "LS0tL[...]" # CA bundle for impersonation proxy