---
tags: api-design
---

# Multiple upstream identity provider support in the Pinniped Supervisor

**Note: This doc is a Work in Progress** 👷‍️🚧👷‍♀️

## Use cases for multiple identity providers (IDPs)

Imagine you are an IT administrator at a large company who is responsible for creating and handing out fleets of Kubernetes clusters. You have decided to use the Pinniped Supervisor to control authentication to those clusters.

Each pool of clusters should allow authentication from a distinct pool of users, so each pool of clusters will share a `FederationDomain` as their source of users. Therefore, you like to have configuration knobs which help you craft a unique pool of users for each `FederationDomain`.

Some examples of when you might like to employ multiple IDPs could be:

- You would like to create a pool of clusters for each development team, and ensure that members of any team are only allowed to authenticate to their own clusters.
- You would like to allow your users to log in either via OIDC using your corporate directory, or via LDAP using your corporate directory for CI use cases, and you would like to restrict the LDAP logins to only include service accounts.
- Your company just acquired another company, and you would like to allow authentication from both corporate IDPs to some or all pools of clusters, by adding the acquired company's IDP to your existing `FederationDomain(s)`.
- You would like to have users of one pool of clusters authenticate using your corporate directory, while another pool of clusters allows authentication using either your corporate directory or another IDP which includes users from a company with which you contract for consulting services.
- *What other uses cases might be interesting to consider for this design effort?*

## How it works today, including current limitations (as of v0.9)

### Summary of how it works today

Today the Supervisor allows multiple `FederationDomains`. All `FederationDomains` use the currently defined `OIDCIdentitiyProvider` and `LDAPIdentityProvider` resources to authenticate users. But, if more than one identity provider is defined, then the Supervisor blocks all end user authentication to all `FederationDomains` because this is not currently a supported configuration.

We would like to add support for multiple identity providers.

### Details of how it works today

Today the Pinniped Supervisor supports several types of upstream identity providers (IDPs). We currently offer `OIDCIdentitiyProvider` and `LDAPIdentityProvider`, and are working on adding `ActiveDirectoryIdentityProvider` in an upcoming release.

Upstream IDPs are used by each `FederationDomain`. A `FederationDomain` represents a pool of users who can all authenticate into a pool of Kubernetes clusters. Each `FederationDomain` becomes an OIDC issuer hosted by the Supervisor, each offering unique discovery and OIDC authorization endpoints.

Users use `pinniped get kubeconfig` to generate a Pinniped-compatible kubeconfig for clusters. When that cluster is configured using a `JWTAuthenticator` to trust tokens issued by a Supervisor's `FederationDomain`, the [command performs discovery API calls](https://github.com/vmware-tanzu/pinniped/blob/91a1fec5cf7f42b41634659efae00013827f54b4/cmd/pinniped/cmd/kubeconfig.go#L248) using the `FederationDomain`'s [IDP discovery endpoint](https://github.com/vmware-tanzu/pinniped/blob/e25eb054502def3b6fb10fcd53028b12196541c3/internal/oidc/idpdiscovery/idp_discovery_handler.go#L31) to find the list of available IDPs for that `FederationDomain`. It then encodes the selection of one particular IDP into the resulting kubeconfig by noting the IDP's name and type.

When an end user uses `kubectl` with the generated kubeconfig, the `pinniped` CLI performs the OIDC authcode flow with the `FederationDomain` issuer endpoints, which allows the user to authenticate using the configured upstream IDP. This establishes an authenticated session between the user and the Supervisor. Then the CLI performs an additional token exchange with the Supervisor to get a cluster-scoped token which will allow authentication to a specific workload cluster's Pinniped Concierge.

The Kubernetes API allows the user to define many of the IDP resources. The Pinniped controllers validate and load them all. However, because we did not have time to plan all details of how multiple simultaneous upstream IDPs should work before the first release, we created a temporary limit in the Supervisor's code to effectively prevent users from doing this in any productive configuration. Whenever an end user starts an authorization flow with any `FederationDomain`, [the endpoint immediately returns an error](https://github.com/vmware-tanzu/pinniped/blob/629bf616554834945589e0269a98e6e7fcf8a71e/internal/oidc/auth/auth_handler.go#L282-L284) if there are currently multiple IDP resources defined.

Although we allow the creation of multiple simultaneous `FederationDomains`, it is of very limited value to create multiple in practice since we do not yet support multiple IDPs. The pool of users represented by each `FederationDomain` is identical.

### Today's existing configurations to consider during upgrades

#### Existing Supervisor configurations

Any existing deployment of the Pinniped Supervisor today has one of the following configurations:

- No `OIDCIdentityProvider` or `LDAPIdentityProvider` defined. No end user can authenticate to any `FederationDomain` that might be configured on this Supervisor. Therefore, this seems unlikely to be a configuration in a production system, unless it is an unused deployment of Pinniped.
- More than one `OIDCIdentityProvider` or `LDAPIdentityProvider` defined. No end user can authenticate to any `FederationDomain` that might be configured on this Supervisor. Therefore, this seems unlikely to be a configuration in a production system.
- Exactly one of either `OIDCIdentityProvider` or `LDAPIdentityProvider` defined (not both), and one or more `FederationDomains` defined. End users can authenticate to any of the `FederationDomains` using the single IDP on this Supervisor. This seems like the most likely configuration.

#### Existing kubeconfig files

Old Supervisor versions do not offer upstream IDP discovery endpoints for the `FederationDomains`. More recent versions do. Depending on which `pinniped` CLI version and which Supervisor version were used at the time when the admin ran `pinniped get kubeconfig`, the end user may or may not have the IDP name and type encoded inside the kubeconfig file as command-line flags for the `pinniped` auth plugin.

Although recent versions of the `pinniped` client [send the IDP name and type CLI flag values as custom query parameters](https://github.com/vmware-tanzu/pinniped/blob/91a1fec5cf7f42b41634659efae00013827f54b4/cmd/pinniped/cmd/login_oidc.go#L158-L161) on the `FederationDomain` authorize request, the server endpoint currently [ignores those params and instead chooses the only currently defined IDP](https://github.com/vmware-tanzu/pinniped/blob/629bf616554834945589e0269a98e6e7fcf8a71e/internal/oidc/auth/auth_handler.go#L286-L289).

## Design considerations for allowing multiple IDPs in a future version

Several open questions must be resolved.

- *How does the user indicate which IDPs shall be used by which `FederationDomains`?*

Other Kubernetes APIs use labels in similar situations. For example, a `Service` uses label selectors to choose pods. One possible solution would be to have the `FederationDomain` choose IDPs via labels. However, this may be awkward for several reasons. One reason is that the names of the IDP resources are meaningful and end up in client's kubeconfigs, whereas using a label selector might imply to the user that the name of the resource is not meaningful in the context of the `FederationDomain`. Labels do not provide a place to add `FederationDomain`-specific configuration to an IDP. For these reasons (and others described below), labels are perhaps not the best choice.

- *Can an individual IDP be used by multiple `FederationDomains` at the same time?*

This seems desirable. IDPs use OIDC clients or LDAP service accounts, which can be inconvenient to acquire. They can take effort to configure well enough to get username and group name mappings to work correctly. Once an IDP configuration is working well, it seems like a user might like to reuse it without needing to copy/paste its configuration.

- *During an upgrade, how will pre-existing `FederationDomains` become associated with the single pre-existing IDP?*

If we choose labels as the association mechanism, then during an upgrade the existing `FederationDomains` will not have any label selectors. Should they stop working? Should a lack of selectors mean to select all available IDPs, which would allow them to keep working?

- *Each IDP offers a unique set of usernames, UIDs, and group names, but can have overlapping values between them. Once a single `FederationDomain` has multiple sources of upstream identity, how can the user prevent identity collisions?*

One possibility would be to allow the user to configure arbitrary upstream-to-downstream identity transformations. See the prototype in [PR #694](https://github.com/vmware-tanzu/pinniped/pull/694) for details about the concept of using the Starlark programming language to provide configuration hooks. Note that identity transformations are also potentially useful outside the context of multiple IDPs, as outlined by the use cases proposed by that PR.

- *At what level would the identity transformation configuration take place? At the `OIDCIdentitiyProvider` and `LDAPIdentityProvider`? At the `FederationDomain`? In the Concierge at the `JWTAuthenticator`? At any and all of these levels?*

Configuring identity transformations at the IDP level seems like a natural fit. The job of the transformation is, "Given the identity of a user from one upstream IDP, transform it to a downstream identity."

Configuring identity transformations at the `FederationDomain` level might be convenient for the use case where several `FederationDomains` are drawing users from a shared IDP, but then applying further restrictions such as rejecting users who lack certain group membership. For that use case, if transformations/policies can only be applied at the IDP level, then the user would need to maintain several copies of nearly identical IDP configurations which differ only by transformation/policy.

Perhaps a `FederationDomain` should have a way to attach different transformation functions to each associated IDP. This might be awkward if `FederationDomains` choose IDPs using label selectors, but could be easy if they instead choose `FederationDomains` by referring to their specific names (or type/version/names).

Perhaps a `FederationDomain` would declare a transformation which applies to all associated IDPs. This is perhaps a little harder to reason about. The job of the transformation would be, "Given the identity from one of *several* possible upstream IDPs, transform it to a downstream identity."

Perhaps a `FederationDomain` could instead list the specific IDPs that it would like to associate along with transformations for each. More discussion of this option is given in later sections.

Configuring identity transformations at the Concierge level might be a nice way for a user who is attaching their cluster to somebody else's `FederationDomain` to exercise some control over authentication. However, this would not be required to support multiple IDPs in the Supervisor. It could be added later if desired.

- *Would a user be required to write custom Starlark code just to make basic use of multiple IDPs?*

This seems undesirable. They would need to learn the programming language which takes some effort even if the programming language is very simple. Writing custom logic means that they probably need to write unit tests for the logic too, or else it could have bugs.

Basic usage of multiple upstream IDPs implies that there needs to be a basic way to avoid potential collisions between usernames, group names, and UIDs from multiple IDPs. Perhaps a simple string prefixing option on the IDP configuration could allow a user to solve this simply, and they could reach for Starlark only if they want something more custom.

Also note that if your company is in full control of the users in an IDP, then perhaps you do not need any transformations if you can ensure that your usernames and group names are non-overlapping. This could be easy for usernames, for example by using confirmed email addresses as the usernames. However, this may be harder for group names, especially if you like to use short, readable group names.

- *After an upgrade, how will pre-existing kubeconfigs behave against the upgraded Supervisor?*

When the authorize endpoint gets an IDP's name and type as custom query params then it should use them to perform a lookup in its authenticator cache (filtered by the IDPs associated with the endpoint's `FederationDomain`). What if the endpoint does not receive the query params, like when a client with an old kubeconfig, or an old client, made the call? If there is only one IDP associated to the `FederationDomain` is it okay to proceed by assuming that IDP when one is not requested, as we do today?

- *Is the selection of which upstream IDP to use determined by the admin at the time of `pinniped get kubeconfig` (this is how it works today), or should it be an option for the end user at `kubectl` auth time?*

A feature to allow the end user to interactively choose the IDP that they would like to use could be deferred until later, if it is desired, since the CLI could directly query the `FederationDomain` discovery endpoints. Or we could explore designing and adding this now if it is a desirable part of the user experience.

*What other open questions should we consider for this design effort?*

## Sketching a potential solution

The following sketch is a [strawdog proposal](https://en.wikipedia.org/wiki/Straw_man_proposal) meant to stimulate discussion.

Changes to `OIDCIdentitiyProvider` and `LDAPIdentityProvider`:

- None? That would be nice, and would create a clear conceptual division of labor:
- An `OIDCIdentitiyProvider`, `LDAPIdentityProvider`, and other future kind of IDP defines how to authenticate users to an upstream provider using a specific authentication protocol, and how to use the specific schema of the upstream provider to pull out usernames and group names. The output of an authentication attempt is success or failure, along with a normalized representation of a user.
- A `FederationDomain` defines how to project those authenticated and normalized users downstream into a pool of clusters in a generic way that is not specific to any upstream authentication protocol or schema, potentially transforming their normalized usernames and group names along the way, or even potentially rejecting their downstream authentication due to some rule specific to this pool of clusters.

- Side note: recall that the "login once per day" user experience is really "login once per day per FederationDomain". Does that effect this approach?

Changes to `FederationDomains`:

- `FederationDomains` choose IDPs by listing them explicitly by name. This gives us a way to attach other configuration knobs at the join point of the `FederationDomain` with each of its IDPs.

- Each IDP in this list could have an optional string prefix field which causes all usernames and group names (and UID strings?) from that IDP to be automatically prefixed by that value when used by this `FederationDomain`, with a colon separator. E.g. a prefi of "my-idp" would generate usernames like `my-idp:ryan`.

- Each IDP in this list could have an optional reference to a `Secret` of type `secrets.pinniped.dev/identity-transformation` which contains one key/value pair, where the value is a Starlark transformation script to be applied whenever this IDP is used by this `FederationDomain`. Or maybe an optional *list* of references to apply a list of transformations in the order specified?

Putting these in a `Secret` isn't strictly necessary but seems useful in case the user wants to include personally identifiable information of users in the source code, and to help them treat their transformation scripts like a little library of available scripts to be mixed and matched with IDPs and `FederationDomains`.

- If both the optional prefix and the optional transformation script are specified, then the prefixing happens first, and the already prefixed strings are passed to the Starlark transformation script(s).

- Another feature could be to allow each IDP in this list to be renamed in the context of this `FederationDomain`, to protect the end user's kubeconfigs from future resource `Name` changes of the `OIDCIdentitiyProvider` and `LDAPIdentityProvider` resources.

Changes to the authorization and callback endpoints:

- When an authorization request includes the IDP name and type, perform a lookup into the authenticator cache which first filters by the `FederationDomain`'s associated IDPs, then reverse-maps the IDP name included on the request back to the Kubernetes resource name, then grabs the authenticator from the cache.
- When an identity is returned by any authenticator, optionally prefix the username and group name strings by the prefix configured for that IDP on the `FederationDomain`, then optionally pass them through the transformation(s) configured for that IDP on the `FederationDomain`, and then finally create a downstream session.

Changes to the IDP discovery endpoint:

- When there is an IDP name mapping configured for an IDP in the `FederationDomain`'s list of IDPs, apply the optional name mapping and include the mapped name in the discovery document.

During upgrade:

- The least code approach would be to do nothing and require the user to manually add the name of their pre-existing IDP to each of their `FederationDomains` during upgrade, ot else risk their users not being about to log in anymore. This doesn't make for a smooth unattended upgrade though.

Alternatively, we could take advantage of the observation that an empty list of IDPs on a `FederationDomain` is not a useful configuration if it is interpreted to mean that the `FederationDomain` has no IDPs and thus no users. Perhaps instead an empty (or null?) list of IDPs could be interpreted to mean that *all* IDPs are associated to this `FederationDomain` with no transformations and no prefixing, which would be backwards compatible. If desired, the controller could consider it to be an error whenever the list is empty and there is more than one IDP defined, to protect users from accidentally adding more IDPs to a `FederationDomain` without considering the ability to configure prefixes/transformations for each.

### Sketch of some YAML

The following sketch assumes that we create a new CRD for transformation scripts.

apiVersion: config.supervisor.pinniped.dev/v1alpha1
kind: FederationDomain
metadata:
  name: example
  namespace: supervisor
spec:
  issuer: https://example.pinniped.dev
  tls:
    secretName: my-certificate
  # identityProviders is the list of available IDPs for this
  # FederationDomain. When empty (or null?), do something
  # backwards compatible with older versions of Pinniped.
  identityProviders:
  # name is the name that will be visible to clients. Within this
  # list, the names must be unique.
  - name: default-idp
    # objectRef is a reference to which IDP resource.
    objectRef:
      kind: OIDCIdentityProvider
      name: my-default-oidc-identity-provider
  - name: corp-directory
    objectRef:
      kind: LDAPIdentityProvider
      name: my-ldap-identity-provider
    # transforms is an optional list of transformation functions.
    # Each must be StarlarkFunction resources in this namespace of
    # type usernameAndGroups.transform.pinniped.dev. They will be
    # called as a chain in the order listed here. If the function
    # allows params, then those params should also be listed here.
    transforms:
    - name: pinniped:prefix
      params:
        prefix: "sso:corp-directory:"
    - name: pinniped:prefix-groups
      params:
        prefix: "sso:corp-directory:"
  - name: org1
    objectRef:
      kind: OIDCIdentityProvider
      name: my-oidc-identity-provider
    transforms:
    - name: custom-require-hd-claim
      params:
        hd: "org1"
  - name: org2
    objectRef:
      kind: OIDCIdentityProvider
      name: my-oidc-identity-provider
    transforms:
    - name: custom-require-hd-claim
      params:
        hd: "org2"
    # Maybe we want to allow inline Starlark code? If so, it might
    # need to repeat some of the same fields from the
    # StarlarkFunction CRD, such as declaring params.
    - inline: | 
        # [... inline Starlark to add a custom transform ...]

apiVersion: config.supervisor.pinniped.dev/v1alpha1
kind: StarlarkFunction
metadata:
  name: pinniped:prefix-groups
  namespace: supervisor
spec:
  # The type of transformation decides where it is appropriate to
  # use, what inputs are passed to the script, and what outputs are
  # expected for the script to return.
  # All transformable inputs are also returned as outputs,
  # which allows the chaining of multiple transform functions as
  # long as they are all of the same type. These types are defined
  # in the Pinniped source code and are not user-defined.
  type: usernameAndGroups.transform.pinniped.dev/v1
  # Declare what additional params are allowed. These are metadata
  # to assist in the business logic of the transform function,
  # but are not returned by the transform function because they are
  # immutable and each call in the chain will define thier own
  # metadata param values. They will be passed to the function
  # as extra params in the same order that they are defined here.
  additionalParams:
    - name: prefix
      type: string
      optional: true # Maybe? It would be easier if they are all required!
  # This is the actual Starlark code.
  transformScript: |
    # Given a username and groups, plus one additional custom param,
    # return a username and groups. This transform function decides to
    # prepend a string to each group name. It does not modify the username.
    def transform(username, groups, prefix):
      prefixedGroups = []
      for g in groups:
          prefixedGroups.append("group_prefix:" + g.lower())
      return username, prefixedGroups
  # unitTestsScript is optional Starlark code which will be executed upon
  # load by the Pinniped controller. If there are any errors either loading
  # the above script, loading the below script, or running all of the
  # test functions in the below script, then the controller will
  # refuse to make this transform available for use in other contexts
  # and will write the errors to the status of this resource.
  # Each function that starts with the word "test" is executed as a
  # test. Other functions are assumed to be helpers and are not
  # automatically executed. Regardless of pass or fail, any output
  # from the below script is captured and written to the status of
  # this resource, to aid in debugging tests.
  unitTestsScript: |
    def testWhenTheGroupsListIsEmpty():
      transformedUser, transformedGroups = transform(
        "my-user", [], "my-prefix"
      )
      print("debug output: the transformed user was", transformedUser)
      assertEquals("my-user", transformedUser)
      assertEquals([], transformedGroups)
    def testWhenThereAreGroups():
      transformedUser, transformedGroups = transform(
        "my-user", ["group1", "group2"], "my-prefix"
      )
      assertEquals("my-user", transformedUser)
      assertEquals(["my-prefix:group1", "my-prefix:group2"], transformedGroups)