owned this note
owned this note
Published
Linked with GitHub
# [public] kcp API evolution prototyping
## Overview
Enable API authors to evolve their APIs and make API versions covertible to each other without needing webhooks.
## Motivation
APIs are not static; new features are added, fields are renamed, etc. CustomResourceDefinitions provide a means for converting between different API versions - conversion webhooks. kcp doens't support conversion webhooks because they require Kubernetes `Services` to function, and these are not built in to kcp.
We need to provide a way for API providers that are using kcp's APIResourceSchema, APIExport, and APIBinding APIs to support API version conversion.
## Proposal
At a high level:
1. Add a way to specify conversion logic in an APIResourceSchema, using CEL
2. Modify the kcp fork of Kubernetes to support this CEL-based conversion in CustomResourceDefinitions
### Conversion specification options
#### Background
Given an APIResourceSchema such as:
```yaml=
apiVersion: apis.kcp.dev/v1alpha1
kind: APIResourceSchema
metadata:
name: rev0002.widgets.example.io
spec:
group: example.io
names:
kind: Widget
listKind: WidgetList
plural: widgets
singular: widget
scope: Namespaced
versions:
- name: v1
schema:
description: Widgets do things
properties:
apiVersion:
type: string
kind:
type: string
metadata:
type: object
spec:
properties:
firstName:
type: string
lastName:
type: string
type: object
status:
properties:
phase:
type: string
type: object
type: object
served: true
storage: false
subresources:
status: {}
- name: v2
schema:
description: Widgets do things
properties:
apiVersion:
type: string
kind:
type: string
metadata:
type: object
spec:
type: object
properties:
name:
type: object
properties:
first:
type: string
last:
type: string
status:
properties:
phase:
type: string
type: object
type: object
served: true
storage: false
subresources:
status: {}
```
it defines two API versions, v1 and v2.
v1 has a `firstName` and `lastName` fields directly under `spec`.
v2 groups them under a parent `spec.name` field.
#### Option 1: spec field on APIResourceSchema
```yaml=
apiVersion: apis.kcp.dev/v1alpha1
kind: APIResourceSchema
metadata:
name: rev0002.widgets.example.io
spec:
# other fields omitted for brevity
conversion:
hub: v1
conversions:
- version: v2
fromHub: # v1-to-v2
- field: spec.name.first
rule: self.spec.firstName
- field: spec.name.last
rule: self.spec.lastName
toHub: # v2-to-v1
- field: spec.firstName
rule: self.spec.name.first
- field: spec.lastName
rule: self.spec.name.last
```
#### Option 2: part of each spec.versions[]
```yaml=
apiVersion: apis.kcp.dev/v1alpha1
kind: APIResourceSchema
metadata:
name: rev0002.widgets.example.io
spec:
# other fields omitted for brevity
hubVersion: v1
versions:
- name: v2
schema: {}
conversion:
fromHub: # v1-to-v2
- field: spec.name.first
rule: self.spec.firstName
- field: spec.name.last
rule: self.spec.lastName
toHub: # v2-to-v1
- field: spec.firstName
rule: self.spec.name.first
- field: spec.lastName
rule: self.spec.name.last
```
3. in a separate resource
```yaml
apiVersion: apis.kcp.dev/v1alpha1
kind: APIConversionRules
metadata:
name: rev0002.widgets.example.io
labels:
# if we decide to support multiple
apis.kcp.dev/apiresourceschema: rev0002.widgets.example.io
spec:
hub: v1
conversions:
- version: v2
fromHub: # v1-to-v2
- field: spec.name.first
rule: self.spec.firstName
- field: spec.name.last
rule: self.spec.lastName
toHub: # v2-to-v1
- field: spec.firstName
rule: self.spec.name.first
- field: spec.lastName
rule: self.spec.name.last
```
#### Alternative conversion structures
The examples above specify a hub version and then a list of conversions, 1 per non-hub version. In other words, given hub version v1 and all versions v1, v2, v3, there will need to be 2 conversion entries, 1 each for v2 (converting between v2 and v1 in both directions) and v3 (converting between v3 and v1 in both directions). No conversion entry is needed for v1, because that is the hub version.
We could specify this using different structures, such as:
##### Specify from/to for each rule, 1 field per rule
```yaml
conversions:
- from: v1
rule: self.spec.firstName
to: v2
field: spec.name.first
- from: v1
rule: self.spec.lastName
to: v2
field: spec.name.last
- from: v2
rule: self.spec.name.first
to: v1
field: spec.firstName
- from: v2
rule: self.spec.name.last
to: v1
field: spec.lastName
```
##### Specify from/to for each rule, n fields per rule
```yaml
conversions:
- from: v1
to: v2
rules:
- rule: self.spec.firstName
field: spec.name.first
- rule: self.spec.lastName
field: spec.name.last
- from: v2
to: v1
rules:
- rule: self.spec.name.first
field: spec.firstName
- rule: self.spec.name.last
field: spec.lastName
```
#### How to reference the source object in CEL
All the examples above refer to the source object as `self` in the CEL rules. Another, possibly less confusing option would be to use the name of the source object's version. For example,
```yaml
- from: v1
to: v2
rules:
- rule: self.spec.firstName
field: spec.name.first
- rule: self.spec.lastName
field: spec.name.last
```
would become
```yaml
- from: v1
to: v2
rules:
- rule: v1.spec.firstName
field: spec.name.first
- rule: v1.spec.lastName
field: spec.name.last
```
We potentially could also improve legibility by adjusting the field names slightly, such as:
```yaml
- from: v1
to: v2
rules:
- from: v1.spec.firstName
to: spec.name.first
- from: v1.spec.lastName
to: spec.name.last
```
#### Discussion on options
While options 1 and 2 keep the schema definitions and conversion rules together, they also require that the schema definitions for all API versions **and** all conversion rules (between each non-hub version and the hub version) can fit into a single object in etcd.
Option 3 has the benefit of keeping the conversion rules in a separate resource, so as to avoid restricting the maximum size of an APIResourceSchema any further than the current limitations etcd imposes.
With option 3, we presumably would want to use a naming convention to make it easy to identify which conversion object is linked to which APIResourceSchema. The easiest approach is to require the names to be identical.
One additional considering with option 3 is whether we potentially need to allow conversion rules that exceed the maximum size of a single object in etcd. If that ever become the case, instead of requiring identical names, we could use a label selector to link the conversion resources and an APIResourceSchema. For example, given an APIResourceSchema called `rev0002.widgets.example.io`, we could require that conversion object must have the following label: `apis.kcp.dev/apiresourceschema=rev0002.widgets.example.io`.
#### Decision - Option 3
We have decided to proceed with option 3. This is also aligned with the upstream [Kubernetes KEP for CEL-based admission control](https://github.com/kubernetes/enhancements/pull/3492), which uses a standalone resource for admission rules.
### Supported conversions
#### Simple field-to-field
From (v1):
```yaml
spec:
firstName: bob
```
To (v2):
```yaml
spec:
name:
first: bob
```
Conversion:
```yaml
- from: v1
to: v2
rules:
- from: v1.spec.firstName
to: spec.name.first
```
#### move map (same as simple field-to-field)
From (v1):
```yaml
spec:
colors:
- name: green
feeling: grassy
- name: red
feeling: bold
```
To (v2):
```yaml
spec:
some:
nested:
colors:
- name: green
feeling: grassy
- name: red
feeling: bold
```
Conversion:
```yaml
- from: v1
to: v2
rules:
- from: v1.spec.colors
to: spec.some.nested.colors
```
#### convert single value to list
From (v1):
```yaml
spec:
name: bob
```
To (v2):
```yaml
spec:
names:
- bob
```
Conversion:
```yaml
- from: v1
to: v2
rules:
- field: .spec.name
to: spec.names[0]
- from: v2
to: v1
rules:
- from: v2.spec.names[0]
to: spec.name
preserve:
- spec.names
```
#### List with nested map, field names change
From (v1):
```yaml
spec:
colors:
- name: green
feeling: grassy
- name: red
feeling: bold
```
To (v2):
```yaml
spec:
some:
nested:
awesomeColors:
- realName: green
realFeeling: grassy
- realName: red
realFeeling: bold
```
Conversion:
```yaml
- from: v1
to: v2
rules:
- from: v1.spec.colors
kind: list # maybe not needed if itemRules is present?
to: spec.some.nested.awesomeColors
itemRules:
- from: self.name
to: item.realName
- from: self.feeling
to: item.realFeeling
```
###
From (v1):
```yaml
spec:
colors:
- name: green
feeling: grassy
- name: red
feeling: bold
```
To (v2):
```yaml
spec:
some:
nested:
awesomeColors:
- realName: green
realFeeling: grassy
- realName: red
realFeeling: bold
```
```yaml
- from: v1
to : v2
rules:
- field: spec.colors[i].name
destination: spec.nested.awesomeColors[].realName
```
### move from map to list
From (v1):
```yaml
spec:
colors:
green:
feeling: grassy
red:
feeling: bold
```
To (v2):
```yaml
spec:
colors:
- name: green
feeling: grassy
- name: red
feeling: bold
```
```yaml
- from: v1
to : v2
rules:
- field: spec.colors[key]
destination: spec.colors[]
transformation: {name: key, feeling: self.feeling}
```
### specify other fields outside of destination
From (v1):
```yaml
spec:
colors:
green:
feeling: grassy
red:
feeling: bold
day: monday
```
To (v2):
```yaml
spec:
colors:
- name: green
feeling: grassy
- name: red
feeling: bold
```
```yaml
- from: v1
to : v2
rules:
- field: spec
destination: spec.colors[]
transformation: {name: key, feeling: self.feeling, day: self.day}
```
### Conversion rule validation
When an APIResourceSchema's conversion rules are created, we must compile and check the conversion rules are valid, and reject if not.
The CEL validation is easily doable for simple "from" rules. For "itemRules" we'll have to do additional work, but this should be possible.
The harder problem to solve is how to validate the "to" rules. It's probably best to ensure that a "to" field exists in the schema for the target version.
## Implementation
### Kubernetes changes
APIResourceSchema resources are eventually transformed to CustomResourceDefinitions and served via the apiextensions apiserver part of Kubernetes. Because of this, any modifications to conversion logic have to be made in that section of the code.
The apiextensions apiserver defines a `CRConverterFactory` that is used to supply converters.
```go=
// CRConverterFactory is the factory for all CR converters.
type CRConverterFactory struct {
// webhookConverterFactory is the factory for webhook converters.
// This field should not be used if CustomResourceWebhookConversion feature is disabled.
webhookConverterFactory *webhookConverterFactory
}
```
The following logic determines which converter to supply when conversion is needed:
```go=
var converter crConverterInterface
switch crd.Spec.Conversion.Strategy {
case apiextensionsv1.NoneConverter:
converter = &nopConverter{}
case apiextensionsv1.WebhookConverter:
converter, err = m.webhookConverterFactory.NewWebhookConverter(crd)
if err != nil {
return nil, nil, err
}
converter, err = converterMetricFactorySingleton.addMetrics("webhook", crd.Name, converter)
if err != nil {
return nil, nil, err
}
default:
return nil, nil, fmt.Errorf("unknown conversion strategy %q for CRD %s", crd.Spec.Conversion.Strategy, crd.Name)
}
```
We will have to modify this factory to support a new type of converter, one that is CEL-based. This requires changing the kcp fork of Kubernetes. An option that could reduce long-term maintenace costs (e.g. rebasing) could be the following:
1. Change `CRConverterFactory` from a `struct` to an `interface`
2. Modify `NewCustomResourceDefinitionHandler` to take in a `CRConverterFactory` instead of `serviceResolver` and `authResolverWrapper` (these are only used to construct the default `CRConverterFactory`)
3. Modify `ExtraConfig`
1. Remove `ServiceResolver`
2. Remove `AuthResolverWrapper`
3. Add `CRConverterFactory`
4. Modify `genericcontrolplane` and `cmd/kube-apiserver/app/apiextensions.go` to either take in or set up a `CRConverterFactory` as appropriate.
### CEL evaluation
Before we can perform conversion, we first have to copy the conversion rules from their source workspace (where the APIResourceSchema lives) into the system:bound-crds logical cluster (where the actual CRDs generated from APIResourceSchemas live).
We must modify the APIBinding reconciler so it copies the appropriate conversion object(s) to system:bound-crds as part of the CRD generation process.
## TODO
- Ensuring all CRs eventually get converted to the hub version (or at least we don't allow removing the last APIResourceSchema that supports converting between a CR's version and the hub version)
- Removing old/no longer used APIResourceSchemas
- Removing old/no longer used bound CRDs
- Round tripping safety
- v2 adds new field
- client gets v1, modifies it, issues UPDATE
- must not lose new field
- be able to mark fields for dropping?
Google Drive
Gist
Clipboard
Sign in with Wallet
