[public] kcp API evolution prototyping

Overview

Enable API authors to evolve their APIs and make API versions covertible to each other without needing webhooks.

Motivation

APIs are not static; new features are added, fields are renamed, etc. CustomResourceDefinitions provide a means for converting between different API versions - conversion webhooks. kcp doens't support conversion webhooks because they require Kubernetes Services to function, and these are not built in to kcp.

We need to provide a way for API providers that are using kcp's APIResourceSchema, APIExport, and APIBinding APIs to support API version conversion.

Proposal

At a high level:

Add a way to specify conversion logic in an APIResourceSchema, using CEL
Modify the kcp fork of Kubernetes to support this CEL-based conversion in CustomResourceDefinitions

Conversion specification options

Background

Given an APIResourceSchema such as:






































































apiVersion: apis.kcp.dev/v1alpha1
kind: APIResourceSchema
metadata:
  name: rev0002.widgets.example.io
spec:
  group: example.io
  names:
    kind: Widget
    listKind: WidgetList
    plural: widgets
    singular: widget
  scope: Namespaced
  versions:
  - name: v1
    schema:
      description: Widgets do things
      properties:
        apiVersion:
          type: string
        kind:
          type: string
        metadata:
          type: object
        spec:
          properties:
            firstName:
              type: string
            lastName:
              type: string
          type: object
        status:
          properties:
            phase:
              type: string
          type: object
      type: object
    served: true
    storage: false
    subresources:
      status: {}
  - name: v2
    schema:
      description: Widgets do things
      properties:
        apiVersion:
          type: string
        kind:
          type: string
        metadata:
          type: object
        spec:
          type: object
          properties:
            name:
              type: object
              properties:
                first:
                  type: string
                last:
                  type: string
        status:
          properties:
            phase:
              type: string
          type: object
      type: object
    served: true
    storage: false
    subresources:
      status: {}

it defines two API versions, v1 and v2.

v1 has a firstName and lastName fields directly under spec.

v2 groups them under a parent spec.name field.

Option 1: spec field on APIResourceSchema




















apiVersion: apis.kcp.dev/v1alpha1
kind: APIResourceSchema
metadata:
  name: rev0002.widgets.example.io
spec:
  # other fields omitted for brevity
  conversion:
    hub: v1
    conversions:
    - version: v2
      fromHub: # v1-to-v2
      - field: spec.name.first
        rule: self.spec.firstName
      - field: spec.name.last
        rule: self.spec.lastName
      toHub: # v2-to-v1
      - field: spec.firstName
        rule: self.spec.name.first
      - field: spec.lastName
        rule: self.spec.name.last

Option 2: part of each spec.versions[]





















apiVersion: apis.kcp.dev/v1alpha1
kind: APIResourceSchema
metadata:
  name: rev0002.widgets.example.io
spec:
  # other fields omitted for brevity
  hubVersion: v1
  versions:
  - name: v2
    schema: {}
    conversion:
      fromHub: # v1-to-v2
      - field: spec.name.first
        rule: self.spec.firstName
      - field: spec.name.last
        rule: self.spec.lastName
      toHub: # v2-to-v1
      - field: spec.firstName
        rule: self.spec.name.first
      - field: spec.lastName
        rule: self.spec.name.last

in a separate resource

apiVersion: apis.kcp.dev/v1alpha1
kind: APIConversionRules
metadata:
  name: rev0002.widgets.example.io
  labels:
    # if we decide to support multiple
    apis.kcp.dev/apiresourceschema: rev0002.widgets.example.io
spec:
  hub: v1
  conversions:
  - version: v2
    fromHub: # v1-to-v2
    - field: spec.name.first
      rule: self.spec.firstName
    - field: spec.name.last
      rule: self.spec.lastName
    toHub: # v2-to-v1
    - field: spec.firstName
      rule: self.spec.name.first
    - field: spec.lastName
      rule: self.spec.name.last

Alternative conversion structures

The examples above specify a hub version and then a list of conversions, 1 per non-hub version. In other words, given hub version v1 and all versions v1, v2, v3, there will need to be 2 conversion entries, 1 each for v2 (converting between v2 and v1 in both directions) and v3 (converting between v3 and v1 in both directions). No conversion entry is needed for v1, because that is the hub version.

We could specify this using different structures, such as:

Specify from/to for each rule, 1 field per rule

conversions:
- from: v1
  rule: self.spec.firstName
  to: v2
  field: spec.name.first
- from: v1
  rule: self.spec.lastName
  to: v2
  field: spec.name.last
- from: v2
  rule: self.spec.name.first
  to: v1
  field: spec.firstName
- from: v2
  rule: self.spec.name.last
  to: v1
  field: spec.lastName

Specify from/to for each rule, n fields per rule

conversions:
- from: v1
  to: v2
  rules:
  - rule: self.spec.firstName
    field: spec.name.first
  - rule: self.spec.lastName
    field: spec.name.last
- from: v2
  to: v1
  rules:
  - rule: self.spec.name.first
    field: spec.firstName
  - rule: self.spec.name.last
    field: spec.lastName

How to reference the source object in CEL

All the examples above refer to the source object as self in the CEL rules. Another, possibly less confusing option would be to use the name of the source object's version. For example,

- from: v1
  to: v2
  rules:
  - rule: self.spec.firstName
    field: spec.name.first
  - rule: self.spec.lastName
    field: spec.name.last

would become

- from: v1
  to: v2
  rules:
  - rule: v1.spec.firstName
    field: spec.name.first
  - rule: v1.spec.lastName
    field: spec.name.last

We potentially could also improve legibility by adjusting the field names slightly, such as:

- from: v1
  to: v2
  rules:
  - from: v1.spec.firstName
    to: spec.name.first
  - from: v1.spec.lastName
    to: spec.name.last

Discussion on options

While options 1 and 2 keep the schema definitions and conversion rules together, they also require that the schema definitions for all API versions and all conversion rules (between each non-hub version and the hub version) can fit into a single object in etcd.

Option 3 has the benefit of keeping the conversion rules in a separate resource, so as to avoid restricting the maximum size of an APIResourceSchema any further than the current limitations etcd imposes.

With option 3, we presumably would want to use a naming convention to make it easy to identify which conversion object is linked to which APIResourceSchema. The easiest approach is to require the names to be identical.

One additional considering with option 3 is whether we potentially need to allow conversion rules that exceed the maximum size of a single object in etcd. If that ever become the case, instead of requiring identical names, we could use a label selector to link the conversion resources and an APIResourceSchema. For example, given an APIResourceSchema called rev0002.widgets.example.io, we could require that conversion object must have the following label: apis.kcp.dev/apiresourceschema=rev0002.widgets.example.io.

Decision - Option 3

We have decided to proceed with option 3. This is also aligned with the upstream Kubernetes KEP for CEL-based admission control, which uses a standalone resource for admission rules.

Supported conversions

Simple field-to-field

From (v1):

spec:
  firstName: bob

To (v2):

spec:
  name:
    first: bob

Conversion:

- from: v1
  to: v2
  rules:
  - from: v1.spec.firstName
    to: spec.name.first

move map (same as simple field-to-field)

From (v1):

spec:
  colors:
  - name: green
    feeling: grassy
  - name: red
    feeling: bold

To (v2):

spec:
  some:
    nested:
      colors:
      - name: green
        feeling: grassy
      - name: red
        feeling: bold

Conversion:

- from: v1
  to: v2
  rules:
  - from: v1.spec.colors
    to: spec.some.nested.colors

convert single value to list

From (v1):

spec:
  name: bob

To (v2):

spec:
  names:
  - bob

Conversion:

- from: v1
  to: v2
  rules:
  - field: .spec.name
    to: spec.names[0]
- from: v2
  to: v1
  rules:
  - from: v2.spec.names[0]
    to: spec.name
  preserve:
  - spec.names

List with nested map, field names change

From (v1):

spec:
  colors:
  - name: green
    feeling: grassy
  - name: red
    feeling: bold

To (v2):

spec:
  some:
    nested:
      awesomeColors:
      - realName: green
        realFeeling: grassy
      - realName: red
        realFeeling: bold

Conversion:

- from: v1
  to: v2
  rules:
  - from: v1.spec.colors
    kind: list # maybe not needed if itemRules is present?
    to: spec.some.nested.awesomeColors
    itemRules:
    - from: self.name
      to: item.realName
    - from: self.feeling
      to: item.realFeeling

From (v1):

spec:
  colors:
  - name: green
    feeling: grassy
  - name: red
    feeling: bold

To (v2):

spec:
  some:
    nested:
      awesomeColors:
      - realName: green
        realFeeling: grassy
      - realName: red
        realFeeling: bold

- from: v1
  to : v2
  rules:
  - field: spec.colors[i].name
    destination: spec.nested.awesomeColors[].realName

move from map to list

From (v1):

spec:
  colors:
    green:
      feeling: grassy
    red:
      feeling: bold

To (v2):

spec:
  colors:
  - name: green
    feeling: grassy
  - name: red
    feeling: bold

- from: v1
  to : v2
  rules:
  - field: spec.colors[key]
    destination: spec.colors[]
    transformation: {name: key, feeling: self.feeling}

specify other fields outside of destination

From (v1):

spec:
  colors:
    green:
      feeling: grassy
    red:
      feeling: bold
  day: monday

To (v2):

spec:
  colors:
  - name: green
    feeling: grassy
  - name: red
    feeling: bold

- from: v1
  to : v2
  rules:
  - field: spec
    destination: spec.colors[]
    transformation: {name: key, feeling: self.feeling, day: self.day}

Conversion rule validation

When an APIResourceSchema's conversion rules are created, we must compile and check the conversion rules are valid, and reject if not.

The CEL validation is easily doable for simple "from" rules. For "itemRules" we'll have to do additional work, but this should be possible.

The harder problem to solve is how to validate the "to" rules. It's probably best to ensure that a "to" field exists in the schema for the target version.

Implementation

Kubernetes changes

APIResourceSchema resources are eventually transformed to CustomResourceDefinitions and served via the apiextensions apiserver part of Kubernetes. Because of this, any modifications to conversion logic have to be made in that section of the code.

The apiextensions apiserver defines a CRConverterFactory that is used to supply converters.






// CRConverterFactory is the factory for all CR converters.
type CRConverterFactory struct {
	// webhookConverterFactory is the factory for webhook converters.
	// This field should not be used if CustomResourceWebhookConversion feature is disabled.
	webhookConverterFactory *webhookConverterFactory
}

The following logic determines which converter to supply when conversion is needed:
















var converter crConverterInterface
switch crd.Spec.Conversion.Strategy {
case apiextensionsv1.NoneConverter:
  converter = &nopConverter{}
case apiextensionsv1.WebhookConverter:
  converter, err = m.webhookConverterFactory.NewWebhookConverter(crd)
  if err != nil {
    return nil, nil, err
  }
  converter, err = converterMetricFactorySingleton.addMetrics("webhook", crd.Name, converter)
  if err != nil {
    return nil, nil, err
  }
default:
  return nil, nil, fmt.Errorf("unknown conversion strategy %q for CRD %s", crd.Spec.Conversion.Strategy, crd.Name)
}

We will have to modify this factory to support a new type of converter, one that is CEL-based. This requires changing the kcp fork of Kubernetes. An option that could reduce long-term maintenace costs (e.g. rebasing) could be the following:

Change CRConverterFactory from a struct to an interface
Modify NewCustomResourceDefinitionHandler to take in a CRConverterFactory instead of serviceResolver and authResolverWrapper (these are only used to construct the default CRConverterFactory)
Modify ExtraConfig
Remove ServiceResolver
Remove AuthResolverWrapper
Add CRConverterFactory
Modify genericcontrolplane and cmd/kube-apiserver/app/apiextensions.go to either take in or set up a CRConverterFactory as appropriate.

CEL evaluation

Before we can perform conversion, we first have to copy the conversion rules from their source workspace (where the APIResourceSchema lives) into the system:bound-crds logical cluster (where the actual CRDs generated from APIResourceSchemas live).

We must modify the APIBinding reconciler so it copies the appropriate conversion object(s) to system:bound-crds as part of the CRD generation process.

TODO

Ensuring all CRs eventually get converted to the hub version (or at least we don't allow removing the last APIResourceSchema that supports converting between a CR's version and the hub version)
Removing old/no longer used APIResourceSchemas
Removing old/no longer used bound CRDs
Round tripping safety
- v2 adds new field
- client gets v1, modifies it, issues UPDATE
- must not lose new field
be able to mark fields for dropping?

Andy Goldstein

2022/08/31 23:50:56

```yaml=

And we will offer no help - this is a replacement for conversion webhooks, which developers have to implement on their own, and which work as well (or not) as coded (Edited)

Dr. Stefan Schimanski

2022/09/01 08:16:48

Do we also need some kind of "drop directive" ? I.e. that the old field is removed from unstructured? Or do you want to trust pruning to do that? (Edited)

Ben Luddy

2022/09/01 11:14:19

has

What happens if spec.name is not present? For the CRD schema CEL validation extension, rules are only enforced when the field is present, and optional field validations are supported by anchoring the rule to a parent field and using the has() macro. (Edited)

2022/09/01 11:33:41

I prefer option 1 in order to avoid situations where the conversions become incompatible with the CRD schema (if a CRD update by the ARS-to-CRD transformation controller fails validation, the conversion rules persisted in the ARS would be operating on an earlier revision of the CRD's schemas). (Edited)

2022/09/01 20:49:42

```

Attaching this comment here to keep it separate from the below thread... Are special considerations needed to work with fields nested inside a map or array (e.g. spec.a[5].foobar) as in transition rules (https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/2876-crd-validation-expression-language#transition-rules)? (Edited)

Steve Kuznetsov

2022/08/31 21:02:10

wouldn't this only work iff we also validate that there are no spaces in each? (Edited)

2022/08/31 21:03:51

(and to what extent is kcp in the business of helping people *not* mess this up?) (Edited)

2022/08/31 23:50:09

This is just a contrived example :-) (Edited)

2022/09/01 08:19:13

For clarity: hub version != storage version, right? (Edited)

2022/09/01 13:20:56

It can be, but it is not a requirement that they are identical. (Edited)

2022/09/01 13:24:21

This is slightly different from validations, and we have to be more stringent. (Edited)

We can start by being forgiving for fields that don't exist, maybe? (Edited)

2022/09/01 13:27:41

Probably, although I have a feeling that we'll discover what we need once we start dogfooding the initial implementaiton. (Edited)

2022/09/01 13:35:19

I agree, and that's what I'm planning to do. (Edited)

2022/09/01 13:36:25

(also, CRDs don't get updated in this flow - they're only created) (Edited)

2022/09/01 20:34:56

I worry about making users (writing conversion expressions) responsible for avoiding conversion-time type errors. (Edited)

Craig Brookes

2022/09/02 07:35:10

how limiting would this be? With a webhook you can write straight go code. Would we hit limitations with complex conversions? (Edited)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.