# MOVED TO https://github.com/kcp-dev/kcp/pull/2828
Further additions will be follow-up PRs

# APIs in kcp

## Overview

kcp supports several built-in Kubernetes APIs, provides extensibility using CustomResourceDefinitions, and adds a new
way to export custom APIs for sharing with other workspaces.

## Built-in APIs

kcp includes some, but not all, of the APIs you are likely familiar with from Kubernetes:

### (core) v1
- Namespaces
- ConfigMaps
- Secrets
- Events
- LimitRanges
- ResourceQuotas
- ServiceAccounts

### admissionregistration.k8s.io/v1
- MutatingWebhookConfigurations
- ValidatingWebhookConfigurations

### apiextensions.k8s.io/v1
- CustomResourceDefinitions

### authentication.k8s.io/v1
- TokenReviews

### authorization.k8s.io/v1
- LocalSubjectAccessReviews
- SelfSubjectAccessReviews
- SelfSubjectRulesReviews
- SubjectAccessReviews

### certificates.k8s.io/v1
- CertificateSigningRequests

### coordination.k8s.io/v1
- Leases

### events.k8s.io/v1
- Events

### flowcontrol.apiserver.k8s.io/v1beta1 (temporarily removed)
- FlowSchemas
- PriorityLevelConfigurations

### rbac.authorization.k8s.io/v1
- ClusterRoleBindings
- ClusterRoles
- RoleBindings
- Roles

Notably, workload-related APIs (Pods, ReplicaSets, Deployments, Jobs, CronJobs, StatefulSets), cluster-related APIs (Nodes), storage-related APIs (PersistentVolumes, PersistentVolumeClaims) are all missing - kcp does not include these, and it instead relies on workload clusters to provide this functionality.

## CustomResourceDefinitions

kcp, like Kubernetes, allows developers to add new APIs using [CustomResourceDefinitions](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/) (CRDs). Unlike Kubernetes,
kcp allows multiple copies of a CRD (e.g. `widgets.example.com`) to be installed in multiple workspaces at the same time. Each copy is entirely independent and isolated from all other copies. This means the API versions and schemas can be entirely different. kcp makes this possible because each workspace is its own isolated "cluster."

There are currently some limitations to be aware of with CRDs in kcp:

- Conversion webhooks are not supported
- `service`-based validating/mutating webhooks are not supported; you must use `url`-based `clientConfigs` instead.

CRDs are a fantastic way to add new APIs to a workspace, but if you want to share a CRD with other workspaces, you have to install it in each workspace separately. You also need a controller that can reconcile CRs in all the workspaces where your CRD is installed, which typically means 1 distinct controller per workspace. CRDs are not "cheap" in the API server (each one consumes memory), and kcp offers an improved workflow that significantly reduces overhead.

## Exporting APIs

If you're looking to provide APIs that can be consumed by multiple workspaces, this section is for you!

kcp adds new APIs that enable this new model of publishing and consuming APIs. The following diagram shows a workspace that an API provider would use to export their `widgets` API. The provider uses kcp's new `APIResourceSchema` type to define the schema for `widgets`, and the new `APIExport` type to export `widgets` to other workspaces.

2 separate consumer workspaces use kcp's new `APIBinding` type to add the `widgets` API to their workspaces. They can then proceed to CRUD `widgets` in their workspaces, just like any other API type (e.g. `namespaces`, `configmaps`, etc.).

                                     ┌───────────────────────┐
                                     │  Consumer Workspace   │
                                     ├───────────────────────┤
                                     │                       │
                                ┌────┼─ Widgets APIBinding   |
                                │    │                       │
                                │    │  Widget A             │
┌───────────────────────────┐   │    │  Widget B             │
│  API Provider Workspace   │   │    │  Widget C             │
├───────────────────────────┤   │    └───────────────────────┘
│                           │   │
│     Widgets APIExport ◄───┼───┤
│             │             │   │
│             ▼             │   │
│ Widgets APIResourceSchema │   │    ┌───────────────────────┐
└───────────────────────────┘   │    │  Consumer Workspace   │
                                │    ├───────────────────────┤
                                │    │                       │
                                └────┼─ Widgets APIBinding   │
                                     │                       │
                                     │  Widget A             │
                                     │  Widget B             │
                                     │  Widget C             │
                                     └───────────────────────┘

**Diagram: 1 APIExport consumed by 2 different workspaces** ([source](https://asciiflow.com/#/share/eJyrVspLzE1VssorzcnRUcpJrEwtUrJSqo5RqohRsrI0NdGJUaoEsozMzYCsktSKEiAnRkmBGPBoyh5qoZiYPGKtVFBwzs8rLs1NLVIIzy%2FKLi5ITE6FyJBgyIC4G5cMEYZgtVwhPDMlPbWkWMExwNMpMy8lMy%2BdFAOp5C44BXGNgiMWY6gY4igBgNUBTtgdAGQDw0khoCi%2FLDMFNfHgNMp5gPxCxeSJO4YR8YeqEilVuVYU5BeVKDya3kKCDdj5ONROw68WyS1BqcX5pUXJqcHJGam5iehx1vNoSgM10AT6xHATzlKsiZRcN4dKvl5C1xIDS9DgKMmICQyoqU24ZUgyBEcpRpYh6CURWYagl0EkGDKFSsljRoxSrVItAH%2FrdL4%3D))

Above we talked about needing 1 controller instance per workspace, as workspace is roughly synonymous with cluster, and most/all controllers out there currently only support a single cluster. But because multiple workspaces coexist in kcp, we can be much more efficient and have 1 controller handle `widgets` in multiple workspaces!

We achieve this using a specific URL just for your controller for your `APIExport`.

You'll need to do a few things:

1. Define 1 or more `APIResourceSchema` objects, 1 per API resource (just like you'd do with CRDs).
2. Define an `APIExport` object that references all the `APIResourceSchema` instances you want to export.
3. Write controllers that are multi-workspace aware.

Let's look at each of these in more detail.

### Define APIResourceSchemas

An `APIResourceSchema` defines a single custom API type. It is almost identical to a CRD, but creating an `APIResourceSchema` instance does not add a usable API to the server. By intentionally decoupling the schema definition from serving, API owners can be more explicit about API evolution. In the future, we could envision the possibility of a new `APIDeployment` resource that coordinates rolling out API updates.

Here is an example for a `widgets` resource:

apiVersion: apis.kcp.dev/v1alpha1
kind: APIResourceSchema
metadata:
  name: v220801.widgets.example.kcp.dev
spec:
  group: example.kcp.dev
  names:
    categories:
    - kcp
    kind: Widget
    listKind: WidgetList
    plural: widgets
    singular: widget
  scope: Cluster
  versions:
  - additionalPrinterColumns:
    - description: The current phase
      jsonPath: .status.phase
      name: Phase
      type: string
    - jsonPath: .metadata.creationTimestamp
      name: Age
      type: date
    name: v1alpha1
    schema:
      description: NOTE: full schema omitted for brevity
      type: object
    served: true
    storage: true
    subresources:
      status: {}

The `metadata.name` value must be of the format `<some prefix>.<plural resource name>.<group name>`. From the example above:

- `<some prefix>` is `v220801`
- `<plural resource name>` is `widgets`
- `<group name>` is `example.kcp.dev`

An `APIResourceSchema`'s `spec` is immutable; if you need to make changes to your API schema, you create a new instance.

Conversions between different API versions are not currently supported, but they will be in the future. For the latest status on API conversion, please follow [this issue](https://github.com/kcp-dev/kcp/issues/1671).

Once you've created at least one `APIResourceSchema`, you can proceed with creating your `APIExport`.

### Define your APIExport

An `APIExport` is the way an API provider makes one or more APIs (coming from `APIResourceSchemas`) available to workspaces.

Here is an example `APIExport` called `example.kcp.dev` that exports 1 resource: `widgets`.

apiVersion: apis.kcp.dev/v1alpha1
kind: APIExport
metadata:
  name: example.kcp.dev
spec:
  latestResourceSchemas:
  - v220801.widgets.example.kcp.dev

At a minimum, you specify the names of the `APIResourceSchema`s you want to export in the `spec.latestResourceSchemas` field. The `APIResourceSchemas` must be in the same workspace as the `APIExport` (and therefore no workspace name or path is required here).

You can optionally configure the following additional aspects of an `APIExport`:

- its identity
- what permissions consumers of your exported APIs are granted
- API resources from other sources (built-in types and/or from other `APIExport`s) that your controllers need to access for your service to function correctly

We'll talk about each of these next.

#### APIExport Identity

Each API resource type is defined by an API group name and a resource name. Each API resource type can further be distinguished by an API version. For example, the `roles.rbac.authorization.k8s.io` resource that we frequently interact with is in the `rbac.authorization.k8s.io` API group and its resource name is `roles`.

In a Kubernetes cluster, it is impossible to define the same API `<group>,<resource>,<version>` multiple times; i.e., it does not support competing definitions for `rbac.authorization.k8s.io,roles,v1` or any other API resource type.

In kcp, however, it is possible for multiple API providers to each define the same API `<group>,<resource>,<version>`, _without interference_! This is where `APIExport` "identity" becomes critical.

When you create an `APIExport` and you don't specify its identity (as is the case in the `example.kcp.dev` example above), kcp automatically generates one for you. The identity is always stored in a secret in the same workspace as the `APIExport.` By default, it is created in the `kcp-system` namespace in a secret whose name is the name of the `APIExport`.

An `APIExport`'s identity is similar to a private key; you should never share it with anyone.

The identity's **hash** is similar to a public key; it is not private and there are times when other consumers of your exported APIs need to know and reference it.

Given 2 Workspaces, each with its own `APIExport` that exports `widgets.example.kcp.dev`, kcp uses the identity hash (in a mostly transparent manner) to ensure the correct instances associated with the appropriate `APIResourceSchema` are served to clients. See [Run Your Controller](#Run-Your-Controller) for more information.

#### Permission Claims

An API provider may need to access APIs that come from other APIExports, or that are built in to kcp. This is done by adding `PermissionClaims` for the desired API's group, resource, and identity hash.

# NOT YET FINALIZED / IGNORE FOR NOW

apiVersion: apis.kcp.dev/v1alpha1
kind: APIExport
metadata:
  name: apis.my-corp.dev
spec:
  latestResourceSchemas:
  - v220909-c255fd13.widgets.apis.my-corp.dev
  permissionClaims:
  - group: ""
    resource: "configmaps"
status:
  virtualWorkspaces:
  - url: https://192.168.4.118:6443/services/apiexport/root/apis.my-corp.dev

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:kcp:apiexport:tenancy:maximal-permission-policy
rules:
- apiGroups: ["tenancy.kcp.dev"]
  verbs: ["*"]
  resources:
  - workspaces
  - workspaces/content
  - clusterworkspacetypes
- apiGroups: ["tenancy.kcp.dev"]
  verbs: ["list","watch","get"]
  resources:
  - workspaces/status
  - clusterworkspacetypes/status

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kcp:authenticated:apiexport:tenancy:maximal-permission-policy
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kcp:apiexport:tenancy:maximal-permission-policy
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: apis.kcp.dev:binding:system:authenticated

### Run Your Controller

adsf

### APIBinding
- "imports" all the `APIResourceSchemas` defined in an `APIExport` into its workspace
- Also provides access to the group/resources defined in an `APIExport`'s `PermissionClaims` slice
- APIs are "imported" by accepting `PermissionClaims` within the `APIBinding` for each of the `APIExport`'s group/resources
- The consumer of the `APIExport` must be granted permission to the `bind` verb on that `APIExport` in order to create an `APIBinding`
- An `APIBinding` is bound to a specific `APIExport` and associated `APIResourceSchema`s via the `APIBinding.Status.BoundResources` field, which will hold the identity information to precisely identify relevant objects.
---
An API provider owns the API definitions (`APIResourceSchemas`) and the controllers that act on instances of their custom types. They export the API definitions using `APIExports`.

An API consumer adds APIs from an API provider to their workspace by creating an `APIBinding` instance that references an `APIExport`.

Here is an example `APIBinding` that references the `apis.my-corp.dev` `APIExport` in the `root` namespace, created above.

apiVersion: apis.kcp.dev/v1alpha1
kind: APIBinding
metadata:
  name: whatever
spec:
  reference:
    workspace:
      path: root
      exportName: apis.my-corp.dev
  permissionClaims:
  - resource: "configmaps"
    state: Accepted

### API exporting

### API binding

---
# TODO / notes

- doc when it's ok to delete "old"/no longer used APIResourceSchemas
- identity
- permission claims, how do I correctly reference a resource from another APIExport in a permission claim?
- vw urls
- As a controller, I need to be granted permissions on the APIExport content sub-resource
- example ARS
- how do I correctly reference an APIExport?
- maximalPermissionPolicy
- permisisonClaim acceptance