Brief: Raft Integration in FTL Services

# Brief: Raft Integration in FTL Services **Author**: @worstell  ### Cron #### Requirements: * Execute scheduled jobs exactly once * Distribute jobs across multiple nodes (balance load) * Support automatic failover #### Raft Integration: * Sharded groups to distribute load * Leader election to ensure only one node schedules jobs * State replication to synchronize job schedules across nodes * Snapshotting for recovery ### Lease #### Requirements: * Leases are only granted to one holder at a time * Leases automatically expire after a set TTL * Support automatic failover (lease is re-acquired if a node fails) * Distribute workload when many clients are acquiring leases #### Raft Integration: * Sharded groups to distribute load * Leader election to manage lease allocations * State replication to sync active leases ### Provisioner #### Requirements: * Resources are provisioned exactly once * Maintain provisioning state across multiple instances * Support automatic failover #### Raft Integration: * Leader election to prevent duplicate provisioning * State replication to synchronize provisioning status ### Controller #### Requirements: * Single coordinator for deployments * Manage runner lifecycles (tracking, spawning, and terminating instances) * Support automatic failover #### Raft Integration: * Leader election to coordinate deployment updates * State replication to synchronize runner states and deployment versions across controllers * Typed events to track runner registrations, heartbeats, and deletions ### Schema Service #### Requirements: * Act as source of truth for the schema state, return consistent view of schema across instances * Publish schema updates as events #### Raft Integration: * Leader election to apply schema updates via one node * State replication to sync schema state across nodes * Consensus protocol to guarantee consistency