# Multi-tenant KEDA HTTP Addon design

_June 22, 2021_
_[@arschles](https://github.com/arschles)_

**This document is a work in progess.**

This document is an extension of [issue #183](https://github.com/kedacore/http-add-on/issues/183) in the KEDA HTTP Addon repository. In this issue, [@yaron2](https://github.com/yaron2) requested that the interceptor and external scaler become multi-tenant, so that many `Deployment`s could be controlled by a single scaler and routed by a single interceptor (or many replicas of the same interceptor `Deployment`).

## Current design

Currently, when a new `HTTPScaledObject` is created, the HTTP Addon operator creates a new external scaler and interceptor. As implied above, there is currently a 1:1 mapping from interceptor/scaler to application, which is defined as a `Deployment` & `Service`. The interceptor watches the `Deployment` and routes to the `Service`. The scaler requests the size of the pending HTTP request queue on each interceptor by the following process:

1. request the IP address of each interceptor pod from the Kubernetes control plane API
2. make a request to each interceptor to get the size of the pending HTTP request queue
3. calculate the sum of all the queues
4. report the sum to KEDA

Both the interceptor and scaler are configured to interact with the app `Service`/`Deployment` by environment variables. These variables are set by the operator.

## Proposed design changes

Since both of these components are going to become multi-tenant, the operator should not create them when a new `HTTPScaledObject` is created. Instead, both the interceptor and external scaler should be made aware of a new application `Service`/`Deployment` by the creation of a new `HTTPScaledObject`.

### Scaler changes

Similar to current functionality, the scaler must still report metrics to KEDA. Instead of currently reporting a single metric -- the sum of pending HTTP queue sizes across all interceptors -- it must report one metric per application (in the same namespace as it runs). For example, if there are 7 applications in the same namespace, the scaler needs to report 7 metrics to KEDA.

This change has implications for the interceptor, which are outlined below.

### Interceptor Changes

The interceptor has two major changes. First, it needs to report more granular pending queue sizes to the scaler -- pending queue sizes per application. Second, it needs to be able to route requests dynamically based on the properties of a single request.

#### Interceptor routing table

When it receives an incoming request, the interceptor must look at the hostname (i.e. the `Host` header) and decide to which `Service` to forward the request. This new functionality implies a lookup table, which we'll call a routing table.

The routing table is a simple key/value storage system that maps from `Host` header to backing `Service`. The interceptor will only route to `Service`s in the same Kubernetes namespace.

This routing table should be stored in a central location -- likely the set of `HTTPScaledObject`s in the same namespace -- and cached in memory of each interceptor process.

#### Granular metrics

The scaler requires that the interceptor report pending queue sizes per application, rather than total pending queue size. Since the interceptor will have a routing table and will dynamically route requests, it will also keep count of the pending queue size _per `HTTPScaledObject`_ -- using the routing table -- rather than globally.

Once the interceptor is changed to keep one counter per application, it that information will be added into the scaler -> interceptor RPC call, rather than just the single global number.

### Operator changes

The operator's major functionality is currently to create and configure a new scaler and interceptor for a new `HTTPScaledObject`. This design calls for multi-tenant scalers and interceptors, so that behavior will be removed. Instead, the operator will be responsible for storing and updating the routing table and notifying the interceptors about changes to the routing table.

#### Changes to the `HTTPScaledObject`

`HTTPScaledObject`s have half the information needed for the routing table. They have the `Service`/`Deployment` to forward to, but not the `Host` information. That information will need to be added to the `spec` section of `HTTPScaledObject`s.

#### Assembling the routing table

When a new `HTTPScaledObject` is created, modified, or removed, the operator is currently notified and a reconcile loop is initialized (indeed, a `Reconcile` function is called).

When any change happens to the set of `HTTPScaledObject`s in a namespace, the operator will calculate and store the routing table. Storage will be (eventually) be pluggable such that the cluster operator can choose where to store the table. By default, it will be stored in a `ConfigMap` in the same namespace.

#### Updating the routing table

On each update to the routing table, the operator will serialize the routing table into a `ConfigMap` with a well-known name.

Interceptors will watch Kubernetes events on the same `ConfigMap` and, on every updated or added event, will deserialize the data therein and save the new routing table to their in-memory copy.

Interceptors will also fetch a complete copy of the `ConfigMap` every `$DURATION`, deserialize the contents, and save the new routing table to their in-memory copy. This functionality ensures that, if the watch stream is broken or there are other network issues, any given interceptor will either converge to the latest routing table or fail after no more than `$DURATION`.

#### Verifying that the routing table is updated

>This section is a work in progress. It is tracked in https://github.com/kedacore/http-add-on/issues/225, and the contents of this section may be out of date.

After an `HTTPScaledObject` is created, deleted or updated, it would be helpful for cluster operators to verify that the change has been propagated across the fleet of interceptors.

The operator will convey this information in the `status` field of any given `HTTPScaledObject`. That functionality will ensure that a cluster operator can verify that a specific application is up to date.

>When an `HTTPScaledObject` is deleted, the operator won't have a `status` field to update. In the case that an interceptor does not update its routing table, it will, at worst, route to the backing application for a short time after an `HTTPScaledObject` is deleted until it updates its routing table.

After the operator gets an update to the set of `HTTPScaledObject`s (i.e. a new one, or a change to an existing one), and it issues the "ping update" request, it can determine when all interceptors finished their update by comparing the unique IDs (i.e. the [hostname](https://kubernetes.io/docs/concepts/containers/container-environment/#container-information)) of each pod in the `Deployment` to the unique IDs to those that have made the fetch-routing-table RPC. The pseudocode for this process is as follows:

pods = interceptor_deployment.get_pods()
unique_id = uuid.new()
for pod in pods:
    pod.send_ping_update_request(unique_id)

respondent_pods = accept_ping_update_requests()

TODO