# Design: Schema Versioning
**Author**: @sdouglas

## Description (what)

Schema versioning allows FTL to track changes in schema over time, and can be used to determine if two modules compiled at different times are actually compatible with each other. It will also form the basis for other features such as server side code generation.

This document does not deal with operational details of running with multiple versions of schemas, it is intended to be a foundational building block that can be used to implement these features.

## Motivation (why, optional)

Currently we store that a given module references another module, but we have no way of knowing which version of that module they were expecting. We can keep out schema consistent by disallowing incompatible changes on the server, however if a module is deployed that was not compiled against the most recent schema we have no way of knowing if that module is still compatible. This will be especially important when you have multiple environments such as staging, as different environments will have different schema versions.

## Goals

* Specify how schema are versioned
* Specify the compatibility contract for modules

### Non-Goals (optional)

* This does not address running multiple modules of different versions at the same time.

## Design (how)

All FTL schema will be assigned a version, which is a SHA256 hash of the schema proto file. When a module is compiled and it's schema references another schema then module level metatada will be added to the schema that include the schema version for the referenced module.

When a module is deployed FTL will perform compatibility checking. Two types of checks are performed, persistent and ephemeral. Persistent checks apply to decls that are used in persistent data, such as pubsub and databases. Ephemeral checks are applied to types that are used in other places.

Persistent checks are applied against any persistent Decls (such as PubSub), and Data Decls used by these Decls.

Persistent checks are as follows:

* The type of the persistent Decl cannot be changed
* The types of any fields in a persistent data Decl cannot be changed
* Fields cannot be removed
* Any new fields must be optional

Any violation of these rules will result in the deployment failing. This can be avoided by providing an approptiate migration, however migrations are outside the scope of this document.

Ephemeral checks are applied to exported verbs, both in the new deployment, and verbs called by the new deployment. Ephemerial checks validate the following:

-

A used decl is defined as the following:

* A verb referenced in the 'calls' metadata
* The topics referenced from a subscription
* Data objects used by the above
* Data objects used by persistent decls such as topics.

Persistent checks are stricter than ephemeral checks.

Basic compatibility guarentees are:

* Used Verbs cannot change their signature, and must still be present
* Used Topics cannot change their signature, and must still be present
* Used data declsl must exist
* Field types in data decls cannot change

Internal compatbility rules:
* If the data decl on the server is missing fields they must be marked as optional in the referenced schema

External compatbility rules:
* New fields on data decls used as the request to a verb must be optional

No matter the directory any additional fields that are not present in a given module must be ignored by FTL serialization. This means that both optional fields can always be added, and non-optional fields can be added to output objects such as verb responses and topics.

FTL schemas will be looked up by the blob store by SHA256 hash. Currently this is a database, however in future it will be an OCI registry. In order to make sure that a FTL server will always have a copy of a given referenced schema deployments can include these schemas as content blobs in their deployment artifacts. As the schemas are always addressed by hash including it as a content blob will mean that the FTL server will be able to look it up.

### Required changes (how)

* Add additional metadata to a schema to record referenced versions
* Bundle referenced schema in the deployment to ensure the server always has a copy
* Update deployment validation logic to take the schema version into account
* Write lots of tests for the above