Database schema version ranges

Synapse currently has the concept of a database "schema version", which is used to forestall problems where a database update is performed, but the deployment is then rolled back to an earlier version, causing incompatibility.

This is implemented via the synapse.storage.prepare_database.SCHEMA_VERSION constant and the schema_version.version database column.

The problem with this approach is that it makes it impossible to roll Synapse back to an earlier version. In practice, we often take a "rolling" approach to schema changes. For example, consider the situation where a database table (say, old_table) becomes redundant; we might do the following:

Synapse version N: original version
Synapse version N+1: remove code that reads from old_table, but keep writing to it in case of rollback to version N.
Synapse version N+2: remove code that writes to old_table, but don't yet remove the table in case of rollback to version N+1.
Synapse version N+3: remove old_table completely.

The precise details vary, but such a pattern of gradually evolving database code in a way that maintains compatibility with one or two earlier versions is not uncommon.

Ideally, we would complain about attempts to roll back from N+2 to N, whilst allowing roll-back from N+1 to N or N+2 to N+1; however the schema_version implementation makes that impossible.

Proposal

We change the semantics of schema_version.version to indicate the oldest SCHEMA_VERSION that the database is compatible with.
At startup, Synapse continues to abort if schema_version.version is greater than synapse.storage.prepare_database.SCHEMA_VERSION.
However, schema_version.version is no longer set to prepare_database.SCHEMA_VERSION (we probably need a different constant for it).

In the above example:

Synapse version N+1 would have prepare_database.SCHEMA_VERSION=N+1, (and would add its delta files to delta/<N+1>) to reflect that it uses the database differently to version N, but would set schema_version.version=N, since the database will still be compatible with Synapse version N.
Likewise, Synapse version N+2 would set prepare_database.SCHEMA_VERSION=N+2, and must set schema_version.version=N+1, to denote that we have broken compatibility with version N.