Design: Federation Protocol

Author: @juho

Description

This proposal defines a Federation Protocol for FTL, allowing an FTL cluster to:

• Pull or generate external FTL schemas from another realm.
• Enable local development by syncing external schemas.
• Call verbs from an external realm without manual imports.

Motivation

• Enables cross-realm interoperability between FTL clusters.
• Supports distributed services where different realms contribute to the overall system.
• Allows integration with non-FTL systems like gRPC and REST.

Example 1: Generating a Realm from gRPC Reflection

Some systems expose gRPC APIs but don’t provide FTL schemas. Using gRPC reflection, an FTL cluster can:

• Expose gRPC service definitions as an FTL realm.
• Call remote gRPC services like native FTL verbs.
• Stay in sync with schema updates automatically.

Example 2: Generating a Realm from Another FTL Cluster

For environments with multiple FTL clusters (e.g., different teams, cloud regions):

• Reference schemas and verbs from another cluster instead of duplicating them.
• Sync changes automatically when the external cluster updates.
• Enable seamless cross-cluster service calls.

Design

Schema

We shall represent external realms in the schema as top level schema elements. All but one realm in schema are marked as +external with a metadata marker.

For example, for grpc realm it could look something like

external realm bank {
  module balance {
    export verb withdraw(...) ...
  }
  ...
}

realm <cluster's realm> {
  builtin module builtin { ... }

  module echo { 
    export verb echo(echo.EchoRequest) echo.EchoResponse
      +calls balance.withdraw@bank
    ...
  }
}

The schema of an +external realm is fetched from a remote git repo. Each realm is reponsible syncing its schema in file in git repo representing the public API of the realm.

The endpoint the cluster uses to connect to the remote FTL realm is passed as a config value with key <name>-realm-endpoint

When referring to schema elements from other realms, a @<realm> suffix is used in the referring modules.

Exporting verbs from FTL cluster

To support exporting verbs to be called from other realms, we change the export annotation to support optional export level. For example, in a Go verb, we can define a verb to be exported from the realm like this:

//ftl:verb export:realm
func Echo(...)...

We will export a public schema, which is all export:realm verbs and any data used by them by writing a proto-text file to the project root on every change to the public API. This file will be committed with the code changes so it can be consumed by other realms.

It should be generated by every build FTL does.

To avoid dealing with transitive dependencies, we will also add checks that the public API of a realm does not expose data types defined in external realms.

Declaring external realms

Locally, we will declare external realms in the ftl-project.toml file, with a separate section for each realm.
The configuration will include the git repo to fetch the external realm schema from, as well as the commit used to fetch the schema to keep builds deterministic. For now, we assume that the ftl-project.toml is at the root of the external repo. The external schema file in that repo should be defined in ftl-project.toml

We will add a new CLI command for adding new realms:

ftl realm add --git-repo=[e-mail]:bank/accounts.git --git-branch=main

This adds an entry to ftl-project.toml for a new realm. We shall also store the latest commit from the branch in the ftl-project.toml

This way the config with external realms looks like

[realms]
  [egress]
      [accounts]
        git-repo = "[e-mail]:bank/accounts.git"
        git-branch = "main"
        git-commit = "aac675450010d2b8fc0528239e49e56fd94b5309"
      [taxman]
        ...

To update the commits to the latest, we will add a command

ftl realm update

Finally, we shall add commands for managing realms like ftl realm list and ftl realm remove

The realms are placed in [egress] block to allow us to record schemas of the calling services in the future. Though, not implemented now, in the future we can also write the calling schema to the target repo when adding a realm dependency to maintain 2-way graph of schemas across realm boundaries.

Local dev experience

For now, any calls to a remote realm locally will fail, but we can make the DX for this better in the future by allowing fake or remote connection based development against these APIs.

Connectivity

When a verb in FTL cluster calls an external FTL verb, it is routed to a remote ftl-ingress service for that realm. The ingress service then routes the call to the runner of the verb.

Non FTL realms

For any realm not implemented in FTL (like external APIs or a set of gRPC services), the following are needed:

• an FTL proxy working as the ftl-ingress endpoint for any external calls. This proxy needs to be able to translate FTL calls to any native format that might be used internally in the realm.
• a git repo with up to date public schema for the realm. For example for gRPC, we need a process that reads the reflection edpoints, or original proto files, and translates those to FTL schema that is then committed to the repo where FTL clusters can consume it

Runtimes

The runtimes need to be changed to take the realm name into account when generating clients, by using the realm appropriately in the client package path.

Diagram

Rejected Alternatives

Manually Importing schemas to local modules

We could ask users to import proto files for remote grpc services or FTL modules they want to call as local modules.
However, this would be too much maintenance for engineers.

Updating the schema of external clusters between the cluster directly

The ftl clusters could sync the schema of remote clusters at runtime, but that would require connectivity to a cluster for developers for local development, and would make dealing with breaking changes difficult.