[public] kcp API evolution prototyping

Overview

Enable API authors to evolve their APIs and make API versions covertible to each other without needing webhooks.

Motivation

APIs are not static; new features are added, fields are renamed, etc. CustomResourceDefinitions provide a means for converting between different API versions - conversion webhooks. kcp doens't support conversion webhooks because they require Kubernetes Services to function, and these are not built in to kcp.

We need to provide a way for API providers that are using kcp's APIResourceSchema, APIExport, and APIBinding APIs to support API version conversion.

Proposal

At a high level:

  1. Add a way to specify conversion logic in an APIResourceSchema, using CEL
  2. Modify the kcp fork of Kubernetes to support this CEL-based conversion in CustomResourceDefinitions

Conversion specification options

Background

Given an APIResourceSchema such as:







































































apiVersion: apis.kcp.dev/v1alpha1
kind: APIResourceSchema
metadata:
  name: rev0002.widgets.example.io
spec:
  group: example.io
  names:
    kind: Widget
    listKind: WidgetList
    plural: widgets
    singular: widget
  scope: Namespaced
  versions:
  - name: v1
    schema:
      description: Widgets do things
      properties:
        apiVersion:
          type: string
        kind:
          type: string
        metadata:
          type: object
        spec:
          properties:
            firstName:
              type: string
            lastName:
              type: string
          type: object
        status:
          properties:
            phase:
              type: string
          type: object
      type: object
    served: true
    storage: false
    subresources:
      status: {}
  - name: v2
    schema:
      description: Widgets do things
      properties:
        apiVersion:
          type: string
        kind:
          type: string
        metadata:
          type: object
        spec:
          type: object
          properties:
            name:
              type: object
              properties:
                first:
                  type: string
                last:
                  type: string
        status:
          properties:
            phase:
              type: string
          type: object
      type: object
    served: true
    storage: false
    subresources:
      status: {}

it defines two API versions, v1 and v2.

v1 has a firstName and lastName fields directly under spec.

v2 groups them under a parent spec.name field.

Option 1: spec field on APIResourceSchema





















apiVersion: apis.kcp.dev/v1alpha1
kind: APIResourceSchema
metadata:
  name: rev0002.widgets.example.io
spec:
  # other fields omitted for brevity
  conversion:
    hub: v1
    conversions:
    - version: v2
      fromHub: # v1-to-v2
      - field: spec.name.first
        rule: self.spec.firstName
      - field: spec.name.last
        rule: self.spec.lastName
      toHub: # v2-to-v1
      - field: spec.firstName
        rule: self.spec.name.first
      - field: spec.lastName
        rule: self.spec.name.last

Option 2: part of each spec.versions[]






















apiVersion: apis.kcp.dev/v1alpha1
kind: APIResourceSchema
metadata:
  name: rev0002.widgets.example.io
spec:
  # other fields omitted for brevity
  hubVersion: v1
  versions:
  - name: v2
    schema: {}
    conversion:
      fromHub: # v1-to-v2
      - field: spec.name.first
        rule: self.spec.firstName
      - field: spec.name.last
        rule: self.spec.lastName
      toHub: # v2-to-v1
      - field: spec.firstName
        rule: self.spec.name.first
      - field: spec.lastName
        rule: self.spec.name.last
  1. in a separate resource
apiVersion: apis.kcp.dev/v1alpha1
kind: APIConversionRules
metadata:
  name: rev0002.widgets.example.io
  labels:
    # if we decide to support multiple
    apis.kcp.dev/apiresourceschema: rev0002.widgets.example.io
spec:
  hub: v1
  conversions:
  - version: v2
    fromHub: # v1-to-v2
    - field: spec.name.first
      rule: self.spec.firstName
    - field: spec.name.last
      rule: self.spec.lastName
    toHub: # v2-to-v1
    - field: spec.firstName
      rule: self.spec.name.first
    - field: spec.lastName
      rule: self.spec.name.last

Alternative conversion structures

The examples above specify a hub version and then a list of conversions, 1 per non-hub version. In other words, given hub version v1 and all versions v1, v2, v3, there will need to be 2 conversion entries, 1 each for v2 (converting between v2 and v1 in both directions) and v3 (converting between v3 and v1 in both directions). No conversion entry is needed for v1, because that is the hub version.

We could specify this using different structures, such as:

Specify from/to for each rule, 1 field per rule
conversions:
- from: v1
  rule: self.spec.firstName
  to: v2
  field: spec.name.first
- from: v1
  rule: self.spec.lastName
  to: v2
  field: spec.name.last
- from: v2
  rule: self.spec.name.first
  to: v1
  field: spec.firstName
- from: v2
  rule: self.spec.name.last
  to: v1
  field: spec.lastName
Specify from/to for each rule, n fields per rule
conversions:
- from: v1
  to: v2
  rules:
  - rule: self.spec.firstName
    field: spec.name.first
  - rule: self.spec.lastName
    field: spec.name.last
- from: v2
  to: v1
  rules:
  - rule: self.spec.name.first
    field: spec.firstName
  - rule: self.spec.name.last
    field: spec.lastName

How to reference the source object in CEL

All the examples above refer to the source object as self in the CEL rules. Another, possibly less confusing option would be to use the name of the source object's version. For example,

- from: v1
  to: v2
  rules:
  - rule: self.spec.firstName
    field: spec.name.first
  - rule: self.spec.lastName
    field: spec.name.last

would become

- from: v1
  to: v2
  rules:
  - rule: v1.spec.firstName
    field: spec.name.first
  - rule: v1.spec.lastName
    field: spec.name.last

We potentially could also improve legibility by adjusting the field names slightly, such as:

- from: v1
  to: v2
  rules:
  - from: v1.spec.firstName
    to: spec.name.first
  - from: v1.spec.lastName
    to: spec.name.last

Discussion on options

While options 1 and 2 keep the schema definitions and conversion rules together, they also require that the schema definitions for all API versions and all conversion rules (between each non-hub version and the hub version) can fit into a single object in etcd.

Option 3 has the benefit of keeping the conversion rules in a separate resource, so as to avoid restricting the maximum size of an APIResourceSchema any further than the current limitations etcd imposes.

With option 3, we presumably would want to use a naming convention to make it easy to identify which conversion object is linked to which APIResourceSchema. The easiest approach is to require the names to be identical.

One additional considering with option 3 is whether we potentially need to allow conversion rules that exceed the maximum size of a single object in etcd. If that ever become the case, instead of requiring identical names, we could use a label selector to link the conversion resources and an APIResourceSchema. For example, given an APIResourceSchema called rev0002.widgets.example.io, we could require that conversion object must have the following label: apis.kcp.dev/apiresourceschema=rev0002.widgets.example.io.

Decision - Option 3

We have decided to proceed with option 3. This is also aligned with the upstream Kubernetes KEP for CEL-based admission control, which uses a standalone resource for admission rules.

Supported conversions

Simple field-to-field

From (v1):

spec:
  firstName: bob

To (v2):

spec:
  name:
    first: bob

Conversion:

- from: v1
  to: v2
  rules:
  - from: v1.spec.firstName
    to: spec.name.first

move map (same as simple field-to-field)

From (v1):

spec:
  colors:
  - name: green
    feeling: grassy
  - name: red
    feeling: bold

To (v2):

spec:
  some:
    nested:
      colors:
      - name: green
        feeling: grassy
      - name: red
        feeling: bold

Conversion:

- from: v1
  to: v2
  rules:
  - from: v1.spec.colors
    to: spec.some.nested.colors

convert single value to list

From (v1):

spec:
  name: bob

To (v2):

spec:
  names:
  - bob

Conversion:

- from: v1
  to: v2
  rules:
  - field: .spec.name
    to: spec.names[0]
- from: v2
  to: v1
  rules:
  - from: v2.spec.names[0]
    to: spec.name
  preserve:
  - spec.names

List with nested map, field names change

From (v1):

spec:
  colors:
  - name: green
    feeling: grassy
  - name: red
    feeling: bold

To (v2):

spec:
  some:
    nested:
      awesomeColors:
      - realName: green
        realFeeling: grassy
      - realName: red
        realFeeling: bold

Conversion:

- from: v1
  to: v2
  rules:
  - from: v1.spec.colors
    kind: list # maybe not needed if itemRules is present?
    to: spec.some.nested.awesomeColors
    itemRules:
    - from: self.name
      to: item.realName
    - from: self.feeling
      to: item.realFeeling

From (v1):

spec:
  colors:
  - name: green
    feeling: grassy
  - name: red
    feeling: bold

To (v2):

spec:
  some:
    nested:
      awesomeColors:
      - realName: green
        realFeeling: grassy
      - realName: red
        realFeeling: bold

- from: v1
  to : v2
  rules:
  - field: spec.colors[i].name
    destination: spec.nested.awesomeColors[].realName

move from map to list

From (v1):

spec:
  colors:
    green:
      feeling: grassy
    red:
      feeling: bold

To (v2):

spec:
  colors:
  - name: green
    feeling: grassy
  - name: red
    feeling: bold

- from: v1
  to : v2
  rules:
  - field: spec.colors[key]
    destination: spec.colors[]
    transformation: {name: key, feeling: self.feeling}

specify other fields outside of destination

From (v1):

spec:
  colors:
    green:
      feeling: grassy
    red:
      feeling: bold
  day: monday

To (v2):

spec:
  colors:
  - name: green
    feeling: grassy
  - name: red
    feeling: bold

- from: v1
  to : v2
  rules:
  - field: spec
    destination: spec.colors[]
    transformation: {name: key, feeling: self.feeling, day: self.day}

Conversion rule validation

When an APIResourceSchema's conversion rules are created, we must compile and check the conversion rules are valid, and reject if not.

The CEL validation is easily doable for simple "from" rules. For "itemRules" we'll have to do additional work, but this should be possible.

The harder problem to solve is how to validate the "to" rules. It's probably best to ensure that a "to" field exists in the schema for the target version.

Implementation

Kubernetes changes

APIResourceSchema resources are eventually transformed to CustomResourceDefinitions and served via the apiextensions apiserver part of Kubernetes. Because of this, any modifications to conversion logic have to be made in that section of the code.

The apiextensions apiserver defines a CRConverterFactory that is used to supply converters.







// CRConverterFactory is the factory for all CR converters.
type CRConverterFactory struct {
	// webhookConverterFactory is the factory for webhook converters.
	// This field should not be used if CustomResourceWebhookConversion feature is disabled.
	webhookConverterFactory *webhookConverterFactory
}

The following logic determines which converter to supply when conversion is needed:

















var converter crConverterInterface
switch crd.Spec.Conversion.Strategy {
case apiextensionsv1.NoneConverter:
  converter = &nopConverter{}
case apiextensionsv1.WebhookConverter:
  converter, err = m.webhookConverterFactory.NewWebhookConverter(crd)
  if err != nil {
    return nil, nil, err
  }
  converter, err = converterMetricFactorySingleton.addMetrics("webhook", crd.Name, converter)
  if err != nil {
    return nil, nil, err
  }
default:
  return nil, nil, fmt.Errorf("unknown conversion strategy %q for CRD %s", crd.Spec.Conversion.Strategy, crd.Name)
}

We will have to modify this factory to support a new type of converter, one that is CEL-based. This requires changing the kcp fork of Kubernetes. An option that could reduce long-term maintenace costs (e.g. rebasing) could be the following:

  1. Change CRConverterFactory from a struct to an interface
  2. Modify NewCustomResourceDefinitionHandler to take in a CRConverterFactory instead of serviceResolver and authResolverWrapper (these are only used to construct the default CRConverterFactory)
  3. Modify ExtraConfig
  4. Remove ServiceResolver
  5. Remove AuthResolverWrapper
  6. Add CRConverterFactory
  7. Modify genericcontrolplane and cmd/kube-apiserver/app/apiextensions.go to either take in or set up a CRConverterFactory as appropriate.

CEL evaluation

Before we can perform conversion, we first have to copy the conversion rules from their source workspace (where the APIResourceSchema lives) into the system:bound-crds logical cluster (where the actual CRDs generated from APIResourceSchemas live).

We must modify the APIBinding reconciler so it copies the appropriate conversion object(s) to system:bound-crds as part of the CRD generation process.

TODO

  • Ensuring all CRs eventually get converted to the hub version (or at least we don't allow removing the last APIResourceSchema that supports converting between a CR's version and the hub version)
  • Removing old/no longer used APIResourceSchemas
  • Removing old/no longer used bound CRDs
  • Round tripping safety
    • v2 adds new field
    • client gets v1, modifies it, issues UPDATE
    • must not lose new field
  • be able to mark fields for dropping?
Select a repo