# Adopting Subscriptions API `filters` in Triggers (Experimental Feature)

#### Last Updated: June 28th, 2021

**Knative Eventing Working Group**

| Contributor(s) | Email | Main GitHub Issue |
| -------------- | ------------------- | --- |
| Ahmed Abdalla | [e-mail] | [GitHub Issue](https://github.com/knative/eventing/issues/5204) |

| Approver | Approver Area | Approver Email |
| -------- | ------------- | -------------- |
| TBD | TBD |TBD |

# Motivation / Abstract

1. Adopt the CloudEvents Subscriptions API [`filters`](https://github.com/cloudevents/spec/blob/master/subscriptions-api.md#323-filters) field as an [Experimental Feature](https://github.com/knative/eventing/blob/main/docs/experimental-features.md) in Knative Eventing.
2. Demo a working reference implementation in a MT channel-based broker implementation.
3. Cooperate with the User Experience WG to conduct a user feedback survey and gather feedback from Knative users, PMs and vendors to decide on the future of this feature.

# Background

***Note**: This proposal is a new version that's based on previous work by [Francesco Guardiani](mailto:[e-mail]) which you can find linked [here](https://github.com/knative/eventing/issues/5204).*

`Trigger` API currently supports only exact matching of a string on certain attributes. The expressiveness of this filter method is very poor: No support for `OR`, you cannot match substrings, you cannot do match against multiple possibilites etc which ends up with a very poor UX and even prevents production use cases.

The CloudEvents project recently added a [Subscriptions API](https://github.com/cloudevents/spec/blob/master/subscriptions-api.md) which provides a definition for CNCF CloudEvents event consumers to subscribe to events originating from event producers on behalf of event sources including a [`filters`](https://github.com/cloudevents/spec/blob/master/subscriptions-api.md#323-filters) field that allows subscriptions to specify a set of powerful filter expressions, where each expression evaluates to either true or false for each event.

# Proposal Design / Approach

## Design

### Feature flag

The feature will be enabled using a `new-trigger-filters` flag defined in `config-features`.

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-features
  namespace: knative-eventing
data:
  new-trigger-filters: enabled

### `Trigger` API

The API will add a new `alpha` field with the name `filters` that conforms to the `filters` [API field](https://github.com/cloudevents/spec/blob/master/subscriptions-api.md#filters) defined in the Subscriptions API.

apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
  name: my-service-trigger
spec:
  broker: default
  filters:
  - sql: "source LIKE '%commerce%' AND type IN ('order.created', 'order.updated', 'order.canceled')"
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: my-service

### New `filters` field
* An array of filter expressions that evaluates to true or false. If any filter expression in the array evaluates to false, the event MUST NOT be sent to the `subscriber`.
* Each filter expression follows a "dialect" that defines the type of filter and the set of additional properties that are allowed within the filter expression. If a filter dialect is specified in a subscription that is unsupported by broker, creation or update of the `Trigger` MUST be rejected with an error.

#### Filter Dialects

The new proposed `filters` field will support only the available dialects in CloudEvents Subscriptions API at the time of writing which are:

##### `exact`

CloudEvent attribute String value must match exactly the specified String value.

filters:
  - exact:
      type: com.github.push

##### `prefix`

CloudEvent attribute String value must start with the specified String value (case sensetive).

filters:
  - prefix:
      type: com.github.

##### `suffix`

CloudEvent attribute String value must end with the specified String value (case sensitive).

filters:
  - suffix:
      type: .created

##### `all`

All nested filter expessions must evaluate to true.

filters:
  - all:
      - exact:
          type: com.github.push
      - exact:
          subject: https://github.com/cloudevents/spec

##### `any`

At least one nested filter expession must evaluate to true.

filters:
  - any:
      - exact:
          type: com.github.push
      - exact:
          subject: https://github.com/cloudevents/spec

##### `not`

The nested expression evaluated must evaluate to false.

filters:
  - not:
      - exact:
          type: com.github.push       

##### `sql`

The provided [CloudEvents SQL Expression](https://github.com/cloudevents/spec/blob/master/expression-language.md) must evaluate to true.

filters:
  - sql: "source LIKE '%commerce%' AND type IN ('order.created', 'order.updated', 'order.canceled')"

## Implementation

### `Trigger` API

The `Trigger` API implementation will be provided in knative/eventing repo. The field will be marked as experimental and optional in the golang type.

#### Conflict with the current `filter` field

The current `filter` field will continue to be supported. However, when the feature is enabled and an object includes both `filter` and `filters`, the `filters` field MUST override the `filter` field. This will allow our users to try out the effect of the new `filters` field without compromising existing filters and even controlling it on a `Trigger` object level.

apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
  name: my-service-trigger
spec:
  broker: default
  filter:
    attributes:
      type: dev.knative.foo.bar
      myextension: my-extension-value
  # New CE Subscriptions API filters
  filters:
    - sql: "type == 'dev.knative.foo.bar' AND myextension == 'my-extension-value'"
  subscriber:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: my-service

### MT Channel Broker Reference Implementation

A reference implementation in MT-Broker will be provided to enable our users to evaluate the new `filters` field and have a more accurate users feedback. The `mt-broker-filter` pod is responsible of applying the filtering logic and it will leverage CE go-sdk to parse and filter CESQL expressions.

If the new `filters` and current `filter` fields both are provided in a Trigger object, the new `filters` field MUST override the current `filter` field as described [here](#Conflict-with-the-current-filter-field).

## Prerequisites / Dependencies

* CNCF CloudEvents [Subsriptions API spec](https://github.com/cloudevents/spec/blob/master/subscriptions-api.md).
* CloudEvents SQL [Expression Language spec](https://github.com/cloudevents/spec/blob/master/expression-language.md).
* CloudEvents Go SDK `sql` dialect [implementation](https://github.com/cloudevents/sdk-go/tree/main/sql/v2).

# Integration Checklist

## Operations
TBD
## Observability
TBD
## Test Plan
TBD
## Documentation

Because we want a smooth UX and quick engagement of our users, we need rich docs describing both Eventing experimental features and how to enable them, and the new CE Subscriptions API Filters exeprminetal feature.

The following should be available before the MT-Broker implementation should be approved:

1. Documentation of using the `new-trigger-filters` to enable the new `filters` feature.
2. Documentation of the new `filters` field that's reference in the [Trigger docs](https://knative.dev/docs/eventing/broker/triggers/).
3. Full documentation of CESQL expression language.

## User Experience

In order to decide on the future of the feature, the UX WG can significantly help by conducting users interviews and/or surveys that help eventing decide on the future of experimental feature. Some suggestions of aspects to be covered:

1. Users feedback on the current Trigger `filter` field.
2. Users feedback on the new proposed Trigger `filters` field.
1. Would the new `filters` field improve the UX?
2. Would the new `filters` field reduce the amount of `Trigger` objects compared to using the current `filter` API?
3. How long would it take them to create their first `sql` expression?
4. What were the challenges faced when using the new `filters` API?

# FAQ

## Why are we adding yet another field? Why make the current `filter` field more robust?

The reason there is twofold. First, at the time of developing `Trigger` APIs, there was no Subscriptions API in CloudEvents Project, so it makes sense to experiment with an API that makes us closer to Subscriptions API. Second, we still want to support users workload with the old `filter` field, and give them the possibility to transition to the new `filters` field.

## Why `filters` and not another name that wouldn't be conflicting with the `filter` field?

We've considered naming it `cefilters`, `subscriptionsAPIFilters` or `enhancedFilters` or something less conlficting. But we decided that introducing yet another different name than what's in the Subscriptions API is a step further from the alignment path we want to take towards the Subsriptions API. Instead, we decided it is a good opportunity to conform with the Subscriptions API at least starting from the field name level and leveraging the safety of this being an expermental feature. Meaning once that is implemented, brokers supporting this expermental feature will be able to claim they support CouldEvents Subscriptions API `filters` field.

## How should brokers support the CESQL filter? Will I start writing a cesql parser?

Fortunately, you can use CloudEvents go-sdk or java-sdk, which both support parsing CESQL expressions. Other CloudEvents SDK will get there because the CESQL is already approved and included in the Subscriptions API spec.