# Design: Infrastructure provisioning
**Author**: @juho

One of the goals for FTL is to provision any infrastructure needed to run the code based on the code itself. As we implement this as a feature, we need to define how infrastructure provisioning fits into the FTL deployment workflow.

The two main parts of this design doc are how infrastructure is physically provisioned, and how the provisioning workflow is scheduled.

## Requirements

### All infrastructure and resources should follow the same pattern
This includes resources in the kubernetes, in a cloud or in any propertiary company resources.

### The provisioning should be generic and adjustable to fit most commonly used provisioning systems
We should not tie ourselves to any specific cloud infrastructure, or provisioning tool. The provisioning logic needs to be extendable / configurabe with plugins to match any commonly used provisioners and patterns.

### We need to be able to plan our changes before applying them
We need to provide a way to view the changes a new deployment would apply before actually deploying to a cluster.

### Multiple pieces of FTL infrastructure can be created in a single step
We might want to patch multiple infrastructure changes in a single update. This is especially useful if we want to use the pattern of one Terraform / Cloudformation / Pulumi stack per module.

### We should make it possible to onboard existing infrastructure as a code
Many organizations already have IaC systems in place. We should make it possible to integrate with them if needed. This might mean making a commit to a IaC repository and waiting for a build to pass instead of directly changing resources.
Though, this should not be the recommended way to use FTL, is should be supported to make any migration to FTL easier.

### Support for multiple versions of a module at the same time
We should be able to run multiple versions of a module at the same time, assuming that the resources needed to run them are not conflicting. If there is a conflict between resources, we can fail the deployment.

## Design

### Domain model
We model the state of the FTL resources as a DAG of resource dependencies. Changes to a module as a result of a deployment are modelled as a separate DAG of deployment steps. Separating the two allows us to logically separate the deployment planning from deployment execution.

#### Resource Graph
Resource graph is a graph of dependencies between resources, describing the current state of the FTL cluster and its infrastructure.

##### Graph Nodes
In the Resource Graph, each "node" represents a resource, which is part of the infrastructure used to run FTL modules. Resources are abstract, FTL specific, representations of underlying resources.

Examples of resources are
- database
- topic
- cron schedule
- a specific instance of a module
- The FTL cluster itself, which serves as the foundational resource for everything else

Each node is uniquely identified by combination of the module id, resource id, and the deployment id that last modified the resource. eg. `module-foo:database-bar:deployment-1` This allows us to run resources from previous deployment simultaneously with newer ones if needed.

##### Graph Edges
In the Resource Graph, edges represent dependencies between resources. These dependencies may include constraints set by the dependent resource to control how the related resource can be updated.

If a resource in the resource graph is not connected to the root `FTL` node, it can be unprovisioned and removed. To ensure we can create deployments where dependencies are established before the dependent resources, we keep the deployment state separately before writing the result of a module deployment to the graph as an atomic update once all resources have been provisioned.

Constraints are used to manage how modules are updated to new versions. Constraint is a requirement specific to a resource type, that an instance of the given resource can be evaluated against. So, for resource type `A`:

func (r A) Fulfills(constraint Constraint) bool

An example of a constraint would be a signature of a verb module A calls from module B. If a new deployment of module B chages this signature, it does not fullfill the existing constraint.

in this case, the fulfills for Module resource type would look something like

func (m *Module) Fulfills(constraint Constraint) bool {
  typed := constraint.(ModuleConstraint)
  return m.hasVerbWithSignature(typed.RequiredVerbSignature)
}

These constraints impact how the deployment execution graph is created, and their use varies depending on the resource type. In the Resource Graph, every node must always meet all the constraints of its incoming edges for the graph to remain valid.

##### Example
For an example of a resource graph, consider schema

module A {
  database postgres foodb

  verb foo(...) ...
    +calls A.bar
    +database calls A.foodb

  verb bar(...) ...
}

module B {
  database mysql bardb
  
  verb foo(...) ...
    +calls A.bar
    +database calls B.bardb
}

The resource graph describing this could be something like this, assuming the deployment id is `dep-1`

rectangle "FTL"
rectangle "module[A::dep-1]"
rectangle "module[B::dep-1]"
rectangle "database[A:foodb:dep-1]"
rectangle "database[B:bardb:dep-1]"

"module[B::dep-1]" --> "module[A::dep-1]"
"module[A::dep-1]" --> "database[A:foodb:dep-1]"
"module[B::dep-1]" --> "database[B:bardb:dep-1]"
"FTL" --> "module[A::dep-1]"
"FTL" --> "module[B::dep-1]"

#### Deployment Graph
The Deployment DAG is an execution graph that outlines deployment operations. It acts as the connection between the deployment planner and the deployment executor. We model each node in this graph as an update to a single module, with the updated schema as the input. The updates in the schema are applied atomically and the resource graph is updated only when the full change is applied to the underlying system.

Only one update to a module can be running at the same time. At the time of execution, the update is decomposed into sequential tasks to be run, and after they have succeeded the overal resouce graph is updated to match the new state.

If due to an update, a previous version of a module is no longer needed, its removal is scheduled as a separate cleanup deployment.

When a new module is scheduled for a deployment, it is queued behind any previous deployments of it, any of its direct dependencies, or modules depending directly on it.

For example, if we receive a request to deploy module A, module B, and module C depending on A, and finally D depending both on A and B, we would get this deployment graph:

rectangle A
rectangle B
rectangle C
rectangle D

A -> C
A -> D
B -> D

where A needs to be deployed first, after which C and B can be deployad in parallel, and once B is done, D can be finally deployed.

#### Deployment Tasks
Deployment tasks are run sequentially as a part of a module deployment. They are calculated as a function of the previous resource graph, and the new schema to be deployed:

func Deployment(newSchema *Schema, state *ResourceGraph) []Task

If we are planning a deployment, the we will calculate what the resource graph would lokk like after pending dependent deployments, and then use that graph as the resource graph for the plan.

Examples of tasks that can be part of a deployment in the future:
- Setting up a MySQL database
- Run a database migration and a backfill
- Deploying a new version of a module
- Removing an unused version of a module

The exact structure of these operations will be defined later.

The deployment executor will handle tasks in a deployment where the previous task has finished. If all tasks of the deployment are finished, the executor updates the resource graph.

If any older versions of modules are removed from the graph due to an update, a separate deployment is queueud up to remove the old version of a module.

Each resource type has a planner that translates changes in the resource graph into deployment tasks, based on the old and new states of the resource and any constraints. So, if type `WithConstraints[A]` is a struct with a resource of type `A` and its `Constraint`s

func (r A) Plan(new WithConstraints[A], old []WithConstraints[A]) []Task

Here are some examples of potential planning rules:
- If a new module does not provide a verb needed by existing modules, keep the old module.
- If we are trying to reduce database capacity, fail the plan
- If we are changing DB schema, run any needed backfills and migrations as separate deployment operations (as a rough idea what could be done, something to be defined much more concretely in the future)

#### Plan
A Plan for a deployment is a list of tasks that that deployment will generate. Additionally, plan can contain information of which deployments it is blocked by, as well as any module removals it will generate as a separate deployment. Finally, we can materialise a plan by querying the provisioner plugins for a detailed underlying plan.

### System Components

actor "ftl deploy"
node "ftl-provisioner" {
   node "executor" 
   node "planner"
}

cloud K8S {
    node "ftl-controller"
    node "ftl-runner-1"
    node "ftl-runner-2"
}
    
file "ftl-provisioner-aws"
file "ftl-provisioner-sql"
file "ftl-provisioner-compute"

cloud aws
cloud DBs

"planner" --> executor
"ftl deploy" --> "planner" : schema + artefacts
"executor" --> "ftl-provisioner-aws" : delegates
"executor" --> "ftl-provisioner-sql" : delegates
"executor" --> "ftl-provisioner-compute" : delegates
"planner" --> "ftl-provisioner-aws" #8F8F8F : plans
"planner" --> "ftl-provisioner-sql" #8F8F8F : plans
"planner" --> "ftl-provisioner-compute" #8F8F8F : plans
"ftl-provisioner-compute" --> "ftl-controller" : updates
"ftl-provisioner-aws" --> "aws" : updates
"ftl-provisioner-sql" --> "DBs" : updates
"ftl-controller" <--> "ftl-runner-1"
"ftl-controller" <--> "ftl-runner-2"

#### ftl-provisioner
We want to separate the deployment and runtime systems to ftl-provisioner (admin and deployment concerns) and ftl-controller (runtime concerns) so that we can scope permissions accordingly.

Operations mutating the ftl-cluster or infrastructure will be sent to ftl-provisioner, while ftl-controller handles anything needed to keep the system running.

ftl-provisioner should never be called from ftl-controller or any part of ftl handling customer traffic. If ftl-provisioner goes down, it should not affect the runtime behaviour of the cluster in any way. Furthermore, we want to logically isolated ftl-provisioner from any customer traffic for security reasons.

ftl-provisioner runs both the deployment planner and deployment executor.

#### ftl-provisioner plugins
While the execution graph is an abstract representation of changes needed, ftl-provisioning plugins provide a way to translate those changes to actual infrastructure changes in the underlying platform.

the plugins are executables located in the `$PATH` of the `ftl-provisioner`. When the provisioner starts up, it discovers all `ftl-provisioner-<plugin>` binaries in its `$PATH`. Each resource is configured to be handled by a specific provisioner in the cluster config, where one resource can potentially support more than one resource type.

The plugins should be short living, stateless executables. If the provisioning system needs a state, like terraform or cloudformation do, that should be stored within provisioner specific infrastructure. The plugins use the underlying provisioner to interract with its state. When a change is requested from the plugin, it will return an opaque token referring to the state change, that the executor can use to query the state of the change.

The input to plugins will be the graph diff of the module's resource sub-graph after the deployment operation has been applied, filtered to only contain the resources that the provider supports. This allows the plugins to construct a desired state if they use a declarative provisioner to apply changes.

The plugin executions should be idempotent so that it is safe to retry a change if a previous execution fails.

When the plugin succeeds, it can return implementation specific metadata for each changed resource that we will store to the resource graph with the updated resource.

##### planner
When the `ftl-provisioner` receives a schema for deployment, and places it in the deployment graph based on the dependencies of the module.

The planner the takes deployments with no dependencies and creates the tasks for the deployment based on the schema to be deployed and the state of the resource graph.

Note, we should update the schema to match before the scheduled deployment is finished, so that any clients are compiled always against the latest schema.

If we want to show the user what underlying changes would be done by this deployment, we can call `ftl-provisioner-plugins` with this deployment graph using a `Plan()` functionality of the plugin. This will return an implementation specific detailed plan from each plugin and compose them to be show to the caller.

#### executor
The deployment executor listens to tasks to be executed in any running deployment, and if the task is not assigned to any executor, picks it up. It will then either call a provisioner plugin to apply the changes to the underlying infrastructure, or call the ftl-controller to change the FTL state.

When a plugin initiates a change, it will return a change id that the deployment executor stores to the task metadata. This id will be then used to poll the status of the provisioning, and once the plugin reports it as a success, the task is updated as succeeded, and any potential metadata returned to from the plugin stored with the task metadata.

### local dev experience
`ftl dev` workflow will not change due to these changes. When running in local mode, we will include an embedded provisioner in the binary, capable of provisioning local resources. Otherwise, the actual provisioning workflow will be the same as in a remote cluster.

### DX Changes

#### ftl plan
We shall add a new command `ftl plan` to describe changes an update would do before actually deploying it. We shall not support saving a plan for later execution at this point.

To create a plan, instead of creating changes to the deployment graph, we report these changes to the caller. We can later further improve this by giving the caller the option to return provisioner specific plans by constructing the changed reource graph and using that as an input to a planning call to the provider plugins. These can then run a provider specific plan if that is supported, which can be reported to the end user.

## Alternatives considered

### Have ftl-provisioners listen to deployment events from the controller
We could keep deploying to the controller, and then have ftl-provisioner services listening events from the controller to provision any required resources.

The problem with this is that it would make planning changes challenging. If we want to `plan` our changes, we need to be able to talk to the provider without starting a new deployment.

Furthermore, this patterns allows us to isolate the `ftl-provisioner` from the cluster where `ftl-controller` is running further, if we decide to do so. With this pattern, we could prevent any outgoing traffic from the cluster running our business logic to the provisioning service with higher privileges to make changes in the system. If `ftl-provisioner` was compromised, it would not be able to instruct the provisioner to do potentially harmful changes.

### Crossplane updating infra from K8s updates
We could run Crossplane in K8s to update infrastructure. However, this would tie FTL to K8s, and the cluster running business logic would need privileges to make infra changes. It would also make it difficult to use existing gitops workflows with FTL if needed.

### Use graph rewriting rules to transform the resource graph
We could drop the need for secondary deployment graph and tasks if we had a set of graph rewrite rules with side effects transforming the graph with deployments as resources towards the stable deployed end state. This, however, would make producing plans difficult, as there would not be a clear link of graph transformations and deployments. Additionally, graph rewriting is an unusual approach for many engineers, which would make it difficult to maintain the rewrite rules consistently.
The benefits of being a well researched area and offering a more declarative execution model were deemed not to be enough to warrant this approach.

### Use a more granular task execution graph instead of deployment graph to offer higher deployment concurrency
In the current model, a deplyment adding a new verb can block deployments of dependant modules, even if the interface they need does not change.

We could detect dependencies at task execution level, and parallelise the deployment tasks much more granularly. However, like with graph rewriting, this would make planning much more difficult, as deployments can affect resources from a same module in parallel.

## Next Steps
- spike the process using an in memory datastore, iterate on the APIs and the algorihm, and see that it can handle more complex deployment workflows.
- initially implement the provisioner as part of the `ftl-controller`. Once the basic workflow is implemented, we should look at splitting it as a separate service with a more holistic design for different ftl systems.

## Appendix A: Step by step case study
Let's see a series of deployments would flow through this system. We assume that we have two plugins: `ftl-provisioner-aws` to call Cloudformation, and `ftl-provisioner-module` to provision new module deployments in the ftl cluster.

Note that the code here is just pseudo-code to try to illustrate the idea. The actual implementation might be quite different, and this example simplifies especially the plan creation a lot.

### Creating a new module
#### Plan
Consider an initial deployment of a schema:

module A {
  database postgres foodb

  verb foo(a int) int
    +database calls A.foodb
}

When we receive the deployment request, with deployment id `1` the first step would be to query the deployment graph, and see if `A` or any of its dependencies are being deployed. As the deployment tree is empty, we place this schema to the deployment tree as the first node:

rectangle A

Next, we take the only node from the deployment tree, and transform it to a set of deployment tasks for deployment `1`. We do this by extracting the resource graph from this schema, as a function of the schema itself:

rectangle "module[A::1]"
rectangle "database[A:foodb:1]"

"module[A::1]" -> "database[A:foodb:1]"

Then we will query the current resource tree, containing only the root `FTL` node, to find that all the resources are new.

After this, we will plan the tasks. We see that there are changes to two different resource types: `database`, and `module`. As module depends on `database`, we will add a task to create the database in the task queue first, passing the current list of databases, and old list of databases (empty) as inputs. As `ftl-provisioner-aws` has registered itself to handle databases, we will call it:

task1: call ftl-provisioner-aws [database[A:foodb:1]] []

Next, we add a task to handle the ftl module update with the corresponding plugin:

task2: call ftl-provisioner-ftl [module[A::1]] []

Both tasks are marked as "WAIT" and the task table will look something like this

DEPLOYMENT | STEP | STATE | TASK                     | INPUTS
--------------------------------------------------------------
1          | 1.   | WAIT  | call ftl-provisioner-aws | ...
1          | 2.   | WAIT  | call ftl-provisioner-ftl | ...

#### Execution
When the executor sees a task waiting in a deployment with previous tasks completed, it assigns it to iself and marks it as running:

DEPLOYMENT | STEP | STATE   | TASK                     | INPUTS
----------------------------------------------------------------
1          | 1.   | RUNNING | call ftl-provisioner-aws | ...
1          | 2.   | WAIT    | call ftl-provisioner-ftl | ...

It then executes the task. In this case by calling `ftl-provisioner-aws` with the new and previous sets of database resources in the module:

ftl-provisioner-aws [database[A:foodb:1]] []

This plugin converts the current state into a cloudformation script:

{
  "AWSTemplateFormatVersion" : "2010-09-09",

  "Resources" : {
    "A:foodb" : {
      "Type" : "AWS::RDS::DBInstance",
      "Properties" : {
        ...
      }
    },
}`

Then it would create a stack for A, a change set for the change above, and execute the change-set. The ChangeSetID would be returned to the ftl-provisioner.

The provisioner would then poll the state of the execution by periodically calling ftl-provisioner-aws with the id, until the execution is marked as finished.

Next, the same would happen to the ftl-provisioner, which would call the ftl controller to deploy a new version of the module.

After this, the deployment is complete, and we will update the resource tree with the resources from the module:

rectangle "FTL"
rectangle "module[A::1]" 
rectangle "database[A:foodb:1]" 

"FTL" -> "module[A::1]" 
"module[A::1]" -> "database[A:foodb:1]"

At that point, the deployment is removed from the deployment graph.

### Adding a module depending on the previous one
Now, assume that a new module B is deployed, calling the verb from module A

module B {
  verb bar(a int) int
    +calls A.foo
}

Like before, we see that there are now pendinf deployments, add the schema to the deployment graph, convert that into a single task to run `ftl-provider-ftl` for the new module, and finally, we will update the resource graph to be:

rectangle "FTL"
rectangle "module[A::1]" 
rectangle "module[B::2]" 
rectangle "database[A:foodb:1]" 

"FTL" -> "module[A::1]" 
"FTL" -> "module[B::2]" 
"module[A::1]" -> "database[A:foodb:1]"
"module[B::2]" -> "module[A::1]": "verb: foo(int) int"

Note the additional edge created to the previously deployed module A, with a constraint of having the verb that `B` is calling.

### Deploying a change with a resource removal
Now, suppose the following version of module A is deployed:

module A {
  verb: foo(int) int
}

As there are existing modules, we need to check if the new modules fulfills the constraints they have. Here, the constraint we are checking against is `{verb: foo(int) int}`, and so, we call

(r Module) func fulfills(constraint = {verb: foo(int) int}) bool

Which in this case returns true, as `r` has verb `foo(int) int`

As the new module fulfills the constraint of module `B`, we can move the incoming edge to point to the new module. We execute the tasks, this time calling

ftl-provisioner-aws [] [database[A:foodb:1]]

to remove the old database.

When we update the resource graph, we will end up with this graph:

rectangle "FTL"
rectangle "module[A::1]" 
rectangle "module[A::3]" 
rectangle "module[B::2]" 

"FTL" -> "module[A::3]" 
"FTL" -> "module[B::2]" 
"module[B::2]" -> "module[A::3]": "verb: foo(int) int"

As `module[A::1]` is now orphan, we will add an additional deployment to the deployment graph for removing `module[A::1]`, when executed, unprovisions the old version of `A` and removes it from the graph:

rectangle "FTL"
rectangle "module[A::3]" 
rectangle "module[B::2]" 

"FTL" -> "module[A::3]" 
"FTL" -> "module[B::2]" 
"module[B::2]" -> "module[A::3]": "verb: foo(int) int"

### Deploying an incompatible module change
Suppose we are now deploying this module change:

module A {
  verb: foo(int, int) int
}

This time,when constructing the task list, we notice that the new `A` does not fulfil the constraints imposed by `B` and so, we can not replace it. This time, when calling `ftl-provisioner-ftl` the call will be:

ftl-provisioner-ftl [module[A::3], module[A::4]] [module[A::3]]

After this change is executed in a deployment, the final graph will be

rectangle "FTL"
rectangle "module[A::3]"
rectangle "module[B::2]"
rectangle "module[A::4]"
"FTL" -->  "module[A::4]"
"FTL" --> "module[B::2]"
 "module[B::2]" --> "module[A::3]": "verb: foo(int) int"

### Scheduling concurrent dependent changes
Now, assume we get a request to deploy the following change:

module A {
  verb: foo(int, int, int) int
}

Like above, this would create a deployment graph to replace the old module `module[A::4]` with a new version `module[A::5]`

Here, the deployment graph is

rectangle "A[5]"

Assume we now get a request to deploy a new version of module B, before this module A version is deployed, where the new module B calls the verb `foo(int, int, int) int`.

We will need to query the existing deployment graph for any modifications to the downstream dependencies of `B` (this is `A[5]`) and place that as a sependency of deployment of `B`:

rectangle "A[5]"
rectangle "B[6]"
"A[5]" -> "B[6]"

This means that `A[5]` deployment needs to finish before `B[6]` can begin.

Note, that when `A[5]` finishes, the previous module version can be removed, and this inserts a separate deployment to remove the old version:

rectangle "B[6]"
rectangle "Cleanup[A]"
"B[6]" -> "Cleanup[A]"

Where the execution of `Cleanup[A]` will remove any orphans in the module and run the provisioners with updated resources