# Kuadrant Architectural Overview [Draft]

## Overview

Kuadrant provides a set of policy APIs, that are reconciled by a set of policy controllers and enforced via integration at different points to configure, secure and enhance application connectivity provided via Gateway API. These policy APIs are focused around areas such as DNS, global load balancing and health checks, alongside data plane specific APIs such as rate limiting and auth. Kuadrant builds ontop of the upstream emerging standard for ingress [Gateway API](https://gateway-api.sigs.k8s.io/). From Gateway API it leverages the concept of [policy attachment](https://gateway-api.sigs.k8s.io/references/policy-attachment/) in order to extend the core Gateway API objects with our policy APIs. Kuadrant also integrates with [OCM (Open Cluster Manager)](https://open-cluster-management.io/) as a multi-cluster control plane to enable defining and distributing Gateways across multiple clusters and offers policy APIs that work at this multi-cluster level also.

## Key Architectural Areas

* The Kuadrant architecture is spread across a control plane and also a data plane. Kuadrant can work in both a single and multi-cluster context. Currently for all APIs to work in a single cluster context you need to have OCM installed in that single cluster also.
* The control plane is where policies are exposed and expressed as kubernetes APIs and reconciled by the kuadrant policy controllers.
* The data plane is where kuadrant provided policy enforcement points are installed and integrated with a running gateway.
* Other policies such as rate limit policy and auth policy are reconciled in the control plane and converted into configuration to be consumed by the data plane components.

## 1000m Architecture


**Note:**
Currently some of our policies are focused only around the multi-cluster context (DNSPolicy and TLSPolicy in particular). It is high on our roadmap priority list to ensure our policies can operate at both the single and multi-cluster level.

### Control Plane Components and responsibilites
A control plane component is something responsible for accepting instruction via a CRD based API and ensuring that configuration is manifested into state that can be acted on.

- **[Kuadrant Operator](https://github.com/Kuadrant/kuadrant-operator):**
- Installation of data plane components via their respective operators
- Exposes RateLimitPolicy and AuthPolicy and is currently the policy controller for these APIs
- Configures the Gateway to be able to leverage the data plane enforcement components
- **[Multi-Cluster Gateway Controller](https://github.com/Kuadrant/multicluster-gateway-controller):**
- Exposes DNSPolicy and TLSPolicy
- Configures DNSProviders and TLSProviders
- Focused around use cases involving disributed gateways (for example across clouds or geos)
- Integrates with OCM as the multi-cluster management hub to distribute and observe gateways based on OCM placement and manifestwork APIs.
- **[kuadrant-add-on-manager](https://github.com/Kuadrant/multicluster-gateway-controller/cmd/ocm):**
- Sub component in the gateway controller repo
- Follows the [add-on pattern](https://open-cluster-management.io/concepts/addon/) from OCM
- Responsible for configuring and installing Kuadrant into a target spoke cluster

- **[Limitador Operator:](https://github.com/Kuadrant/limitador-operator)**
- Installs and configures Limitador
- **[Authorino Operator:](https://github.com/Kuadrant/authorino-operator)**
- Installs and configures Authorino

### Data Plane Components and Responsibilities
A data plane component sits in the request flow and is responsible for enforcing policy configuration as a request comes through a gateway

- **Limitador:**
- Complies with the with Envoy rate limiting API to provide rate limiting to the gateway
- **Authorino:**
- Complies with the Envoy external auth API to provide auth integration to the gateway
- **WASM Plugin:**
- Uses the Proxy WASM ABI Spec to integrate with Envoy and provide filtering and connectivity to authorino and limitador for request time enforcement of auth and rate limiting

### Dependencies and integrations

In order to provide its full suite of functionality, Kuadrant has several depenencies

- **[Cert-Manager](https://cert-manager.io/):** (provides the TLS integration)
- required if you want to use TLSPolicy
- **[Open Cluster Manager](https://open-cluster-management.io/):** (provides a multi-cluster control plane)
- required if you want to use multi-cluster gateways
- **[Thanos/Prometheus/Grafana](https://thanos.io):** (provide observability integration)
- optional as we look to leverage the existing technology and provide dashboards etc for ingress

## High Level Multi-Cluster Architecture


Kuadrant has a multi-cluster gateway controller that is intended to run in a OCM provided "Hub" cluster. This cluster is effectively a central management cluster where policy and gateways along with all that OCM offers can be defined and distributed to the managed "spoke" clusters.

## Single cluster
In a single cluster context, the overall architecture remains the same as above, the key difference is that the Hub and Spoke cluster are now a single cluster rather than multiple clusters. This is how we are initially supporting single cluster (note toward future direction we hope to be able to remove the need for OCM in the single cluster context).

## How does Kuadrant leverage OCM

The kuadrant multi-cluster gateway controller is intended to be installed into an OCM hub cluster. The Gateway API resources are also installed into this cluster.
Kuadrant defines a [gateway class](https://gateway-api.sigs.k8s.io/api-types/gatewayclass/) that the multi-cluster gateway controller is the specified controller for. An OCM user can then (with the correct RBAC in place) define a Gateway just as they would in a single cluster setting and this will be reconciled by the multi-cluster gateway controller. We use this gateway definiton to represent a "multi-cluster" gateway and use the status of this gateway resource to store information such as IPs and Hostnames of Load Balancers that are configured in front of an actual gateway.


### Gateway Deployment and Distribution

In order for a multi-cluster gateway to be useful, it needs to be distributed or "placed" on a specific set of hub managed spoke clusters. OCM is responsible for a set of placement and replication APIs. Kuadrant is aware of these APIs, and so when a given gateway is chosen to be placed on a set of managed clusters, Kuadrant multi-cluster gateway controller will ensure the right resources ([ManifestWork](https://open-cluster-management.io/concepts/manifestwork/)) are created in the correct namespaces in the hub. OCM then is responsible for syncing these to the actual spoke cluster and reporting back the status of these resources to the Hub.

In order for the Gateway to be instanstiated, we need to know what gateway provider is being used on the spoke clusters. Admins can then set this provider in the hub via the Gateway class params. Kuadrant will then apply a transformation to ensure the gateway being synced references this spoke gateway provider (istio for example).

It is the OCM workagent that is responsible for syncing down and applying the resources into the managed spoke cluster. It is also responsible for syncing status information back to the hub.

The status information reported back to the Hub is used by the multi-cluster gateway controller to know what LB hosts / IPAddresses to use for DNSRecords that it creates and manages.

More info can be found here https://open-cluster-management.io/concepts/architecture/

## How does Kuadrant integrate with Gateway Providers

The kuadrant data plane, integrates with the Istio gateway provider only currently. To do this, it provides a WASM Plugin that conforms to the Proxy WASM ABI. This WASM Plugin is then executed by the underlying Envoy instance that provides the proxy layer for the Gateway (IE Envoy is the running gateway instance that is responsible for routing traffic).

### Data Flows

There are several different data flows when using Kuadrant.

**Control plane configuration and status reporting**

The initial creation of these APIs (gateways, policies etc) is done by the relevant persona in the control plane just as they would any other k8s resource.
As shown above, in a multi-custer configuration. API definitions are pulled from the Hub and "manifested" into the spokes. The Status of those sycned resources are reported back to the Hub. The same happens for a single cluster, the only difference being the work agent hub controllers are all installed on one cluster.

**3rd party enforcement and Integration**

In order to enforce the policy configuration, components in the control plane and data plane can reach out to configured 3rd parties such as cloud based DNS provider, TLS providers and Auth providers.

**Request Flow**

Requests coming through the gateway instance are intercepted by the WASM plugin installed into the Envoy based gateway provider. This plugin is then responsible for passing on the request to Limitador to be rate limited or Authorino to ensure auth. Each of these component each have the capability to see the request and need to in order to make the required decision. Each of these component can also prevent the request from reaching its backend destination based on user configuration.


## Auth

Authentication is handled by the underly kubernetes cluster. All APIs are CRDs and so authorization is controlled by RBAC in the same way as any kubernetes API.

For Auth between Spoke and Hub see OCM docs https://open-cluster-management.io/concepts/architecture/

### Observability

TODO

## Notes on future direction

TODO expand

Want to separate gateway / ocm integration, policy controllers and policy enforcement

Want to support more gateway providers

Want to progress a Gateway Policy Provider API/Interface

TODO Future arch diagram