# Language Plug-Ins
**Author**: Matt Toohey

## Description

Separate out support for each language, so that languages can be added without changing core FTL code.

## Motivation

Supporting a new language is messy and spread across many parts of the codebase.

## Goals

- Language support separated from core FTL code
- Anyone can add language support without changing FTL code
- Each runner should not need support for all languages, just the language needed for the module.

### Non-Goals

- Separating our supported languages into a separate codebase
- Remote building (language plug-in assumes access to file system)
- No changes to how we do our integration tests that involve go/java/kotlin
- No change to how ftltest works for each language
- FTL will not connect language plugins with how the LSP features work in this initial scope.
- FTL will not provide any help templating for new modules or when building. This may be helpful as we add more languages and might be worth revisiting later.

## Design (how)
#### Language Discovery
FTL will look for each language plugin by searching for executables in the `$PATH` which follow the pattern `ftl-language-<name>`.

#### Launching

`ftl-language-<name>`

For each module, FTL will launch a language plugin. The following environment variables will be passed in:
- `FTL_BIND`: The port to bind to
- `FTL_NAME`: The name to use for the plugin for logging. It usually corresponds to the name of the module but this should not be assumed to be true (eg. `ftl new` does not know the module name when launching the plugin)

If language support requires a lot of RAM, then the language plugin may want to centralize the processing in it's own daemon.

If FTL loses communication with a plugin it can launch a new plugin to handle that module.

#### Communication
FTL will then communicate with the language plugin over gRPC. See the gRPC protocol below.

#### Build updates
Once FTL sends a build streaming call to the plugin, it is up to the language plugin to watch for updates that trigger builds.
This allows for:
- hot reloading
- easier handling of watching for files that may also be modified by the build process (eg: go.mod)

#### Deploying
Clusters need to know what image to provision runners with for a module.
Each time a language plugin builds a module, it passes back the name of the docker image to be used. The docker image must be based on FTL runner image for that version of FTL.

FTL will then store the runner image name along with ModuleRuntime in the schema.

When creating kubernetes deployments, FTL will apply the current FTL version tag to the runner image name. It is assumed that language plugin providers will have a runner image available with the same version tag as FTL.

### Sequence Diagrams
#### Creating a module

CLI->FTL: ftl new
FTL->Plugin: GetCreateModuleFlags
Plugin-->FTL: GetCreateModuleFlagsResponse
FTL->Plugin: CreateModule
Note right of Plugin: Generate files for a new module
Plugin-->FTL: CreateModuleResponse
FTL->CLI: exit

#### Building a module

Note right of FTL: Initial set up
FTL->Plugin: ModuleConfigDefaults
Plugin-->FTL: ModuleConfigDefaultsResponse
FTL->Plugin: GetDependencies
Plugin-->FTL: DependenciesResponse

Note right of FTL: FTL waits until dependencies have been built
FTL->Plugin: Build (contextId = A)
Plugin-->FTL: LogMessage: "compiling"
Plugin-->FTL: LogMessage: "extracting schema"
Plugin-->FTL: BuildSuccess (contextId = A)

#### Building a module with automatic rebuilds

Note right of FTL: Initial set up
FTL->Plugin: ModuleConfigDefaults
Plugin-->FTL: ModuleConfigDefaultsResponse
FTL->Plugin: GetDependencies
Plugin-->FTL: DependenciesResponse

Note right of FTL: FTL waits until dependencies have been built
FTL->Plugin: Build (contextId = A, rebuildAutomatically = true)
Plugin-->FTL: BuildSuccess (contextId = A, isAutomaticRebuild = false)

Note left of Plugin: After a while, detects file changes
Plugin-->FTL: AutoRebuildStarted (contextId = A)
Plugin-->FTL: BuildSuccess (contextId = A, isAutomaticRebuild = true)

Note right of FTL: Dependency's schema was updated
FTL->Plugin: BuildContextUpdated (contextId = B)
Plugin-->FTL: BuildContextUpdatedResponse
Plugin-->FTL: BuildSuccess (contextId = B, isAutomaticRebuild = false)

Note left of Plugin: After a while, detects file changes
Plugin-->FTL: AutoRebuildStarted (contextId = B)
Note left of Plugin: While building, discover a new dependency
Plugin-->FTL: BuildFailure (contextId = B, isAutomaticRebuild = true, invalidateDependencies = true)
FTL->Plugin: GetDependencies
Plugin-->FTL: DependenciesResponse
FTL->Plugin: BuildContextUpdated (contextId = C)
Plugin-->FTL: BuildContextUpdatedResponse
Plugin-->FTL: BuildSuccess (contextId = C, isAutomaticRebuild = false)

### gRPC Protocol

// ModuleConfig contains the configuration for a module, found in the module's ftl.toml file.
message ModuleConfig {
  // Name of the module
  string name = 1;
  // Absolute path to the module's directory
  string dir = 2;
  // The language of the module
  string language = 3;

  // Absolute path to the directory containing all of this module's build artifacts for deployments
  string deploy_dir = 4;
  // Build is the command to build the module.
  optional string build = 5;
  // Build lock path to prevent concurrent builds
  string build_lock = 6;
  // The directory to generate protobuf schema files into. These can be picked up by language specific build tools
  optional string generated_schema_dir = 7;
  // Patterns to watch for file changes
  repeated string watch = 8;

  // LanguageConfig contains any metadata specific to a specific language.
  // These are stored in the ftl.toml file under the same key as the language (eg: "go", "java")
  google.protobuf.Struct language_config = 9;
}

// ProjectConfig contains the configuration for a project, found in the ftl-project.toml file.
message ProjectConfig {
  string dir = 1;
  string name = 2;
  bool no_git = 3;
  bool hermit = 4;
}

message GetCreateModuleFlagsRequest {
  string language = 1;
}

message GetCreateModuleFlagsResponse {
  message Flag {
    string name = 1;
    string help = 2;
    optional string envar = 3;
    // short must be a single character
    optional string short = 4;
    optional string placeholder = 5;
    optional string default = 6;
  }
  repeated Flag flags = 1;
}

// Request to create a new module.
message CreateModuleRequest {
  string name = 1;
  // The root directory for the module, which does not yet exist.
  // The plugin should create the directory.
  string dir = 2;

  // The project configuration
  ProjectConfig project_config = 3;

  // Flags contains any values set for those configured in the GetCreateModuleFlags call
  google.protobuf.Struct Flags = 4;
}

// Response to a create module request.
message CreateModuleResponse {}

message ModuleConfigDefaultsRequest {
  string dir = 1;
}

// ModuleConfigDefaultsResponse provides defaults for ModuleConfig.
//
// The result may be cached by FTL, so defaulting logic should not be changing due to normal module changes.
// For example, it is valid to return defaults based on which build tool is configured within the module directory,
// as that is not expected to change during normal operation.
// It is not recommended to read the module's toml file to determine defaults, as when the toml file is updated,
// the module defaults will not be recalculated.
message ModuleConfigDefaultsResponse {
  // Default relative path to the directory containing all build artifacts for deployments
  string deploy_dir = 1;

  // Default build command
  optional string build = 2;

  // Build lock path to prevent concurrent builds
  optional string build_lock = 3;

  // Default relative path to the directory containing generated schema files
  optional string generated_schema_dir = 4;

  // Default patterns to watch for file changes, relative to the module directory
  repeated string watch = 5;

  // Default language specific configuration.
  // These defaults are filled in by looking at each root key only. If the key is not present, the default is used.
  google.protobuf.Struct language_config = 6;
}

message DependenciesRequest {
  ModuleConfig module_config = 1;
}

message DependenciesResponse {
  repeated string modules = 1;
}

// BuildContext contains contextual information needed to build.
//
// Plugins must include the build context's id when a build succeeds or fails.
// For automatic rebuilds, plugins must use the most recent build context they have received.
message BuildContext {
  string id = 1;
  // The configuration for the module
  ModuleConfig module_config = 2;
  // The FTL schema including all dependencies
  schema.Schema schema = 3;
  // The dependencies for the module
  repeated string dependencies = 4;
  // Build environment provides environment variables to be set for the build command
  repeated string build_env = 5;
}

message BuildContextUpdatedRequest {
  BuildContext buildContext = 1;
}

message BuildContextUpdatedResponse {}

// Error contains information about an error that occurred during a build.
// Errors do not always cause a build failure. Use lesser levels to help guide the user.
message Error {
  enum ErrorLevel {
    INFO = 0;
    WARN = 1;
    ERROR = 2;
  }

  enum ErrorType {
    FTL = 0;
    // Compiler errors are errors that are from the compiler. This is useful to avoid duplicate errors
    // being shown to the user when combining errors from multiple sources (eg: an IDE showing compiler
    // errors and FTL errors via LSP).
    COMPILER = 1;
  }

  string msg = 1;
  ErrorLevel level = 4;
  optional Position pos = 5;
  ErrorType type = 6;
}

message Position {
  string filename = 1;
  int64 line = 2;
  int64 startColumn = 3;
  int64 endColumn = 4;
}

message ErrorList {
  repeated Error errors = 1;
}

// Request to build a module.
message BuildRequest {
  // The root path for the FTL project
  string project_root = 1;

  // The path to the directory containing all module stubs. Each module stub is in a subdirectory.
  string stubs_root = 2;

  // Indicates whether to watch for file changes and automatically rebuild
  bool rebuild_automatically = 3;

  BuildContext build_context = 4;
}

// AutoRebuildStarted should be sent when the plugin decides to start rebuilding automatically.
//
// It is not required to send this event, though it helps inform the user that their changes are not yet built.
// FTL may ignore this event if it does not match FTL's current build context and state.
// If the plugin decides to cancel the build because another build started, no failure or cancellation event needs
// to be sent.
message AutoRebuildStarted {
  string context_id = 1;
}

// BuildSuccess should be sent when a build succeeds.
//
// FTL may ignore this event if it does not match FTL's current build context and state.
message BuildSuccess {
  // The id of build context used while building.
  string context_id = 1;
  // Indicates whether the build was automatically started by the plugin, rather than due to a Build rpc call.
  bool is_automatic_rebuild = 2;
  // Module schema for the built module
  schema.Module module = 3;
  // Paths for files/directories to be deployed
  repeated string deploy = 4;

  // Errors contains any errors that occurred during the build
  // No errors can have a level of ERROR, instead a BuildFailure should be sent
  // Instead this is useful for INFO and WARN level errors.
  ErrorList errors = 5;
}

// BuildFailure should be sent when a build fails.
//
// FTL may ignore this event if it does not match FTL's current build context and state.
message BuildFailure {
  // The id of build context used while building.
  string context_id = 1;
  // Indicates whether the build was automatically started by the plugin, rather than due to a Build rpc call.
  bool is_automatic_rebuild = 2;

  // Errors contains any errors that occurred during the build
  ErrorList errors = 3;

  // Indicates the plugin determined that the dependencies in the BuildContext are out of date.
  // If a Build stream is being kept open for automatic rebuilds, FTL will call GetDependencies, followed by
  // BuildContextUpdated.
  bool invalidate_dependencies = 4;
}

// Every type of message that can be streamed from the language plugin for a build.
message BuildEvent {
  oneof event {
    AutoRebuildStarted auto_rebuild_started = 2;
    BuildSuccess build_success = 3;
    BuildFailure build_failure = 4;
  }
}

message GenerateStubsRequest {
  // The directory path to generate stubs into
  string dir = 1;
  // The schema of the module to generate stubs for
  schema.Module module = 2;
  // The module's configuration to generate stubs for
  ModuleConfig module_config = 3;

  // Native module configuration is the configuration for a module that uses the plugin's language, if
  // the main moduleConfig provided is of a different language. It is provided as a mechanism to derive
  // language specific information. For example, the language version.
  optional ModuleConfig native_module_config = 4;
}

message GenerateStubsResponse {}

message SyncStubReferencesRequest {
  ModuleConfig module_config = 1;

  // The path of the directory containing all module stubs. Each module is in a subdirectory
  string stubs_root = 2;

  // The names of all modules that have had stubs generated
  repeated string modules = 3;
}

message SyncStubReferencesResponse {}

// LanguageService allows a plugin to add support for a programming language.
service LanguageService {
  // Ping service for readiness.
  rpc Ping(xyz.block.ftl.v1.PingRequest) returns (xyz.block.ftl.v1.PingResponse) {
    option idempotency_level = NO_SIDE_EFFECTS;
  }

  // Get language specific flags that can be used to create a new module.
  rpc GetCreateModuleFlags(GetCreateModuleFlagsRequest) returns (GetCreateModuleFlagsResponse);

  // Generates files for a new module with the requested name
  rpc CreateModule(CreateModuleRequest) returns (CreateModuleResponse);

  // Provide default values for ModuleConfig for values that are not configured in the ftl.toml file.
  rpc ModuleConfigDefaults(ModuleConfigDefaultsRequest) returns (ModuleConfigDefaultsResponse);

  // Extract dependencies for a module
  // FTL will ensure that these dependencies are built before requesting a build for this module.
  rpc GetDependencies(DependenciesRequest) returns (DependenciesResponse);

  // Build the module and stream back build events.
  //
  // A BuildSuccess or BuildFailure event must be streamed back with the request's context id to indicate the
  // end of the build.
  //
  // The request can include the option to "rebuild_automatically". In this case the plugin should watch for
  // file changes and automatically rebuild as needed as long as this build request is alive. Each automactic
  // rebuild must include the latest build context id provided by the request or subsequent BuildContextUpdated
  // calls.
  rpc Build(BuildRequest) returns (stream BuildEvent);

  // While a Build call with "rebuild_automatically" set is active, BuildContextUpdated is called whenever the
  // build context is updated.
  //
  // Each time this call is made, the Build call must send back a corresponding BuildSuccess or BuildFailure
  // event with the updated build context id with "is_automatic_rebuild" as false.
  //
  // If the plugin will not be able to return a BuildSuccess or BuildFailure, such as when there is no active
  // build stream, it must fail the BuildContextUpdated call.
  rpc BuildContextUpdated(BuildContextUpdatedRequest) returns (BuildContextUpdatedResponse);

  // Generate stubs for a module.
  //
  // Stubs allow modules to import other module's exported interface. If a language does not need this step,
  // then it is not required to do anything in this call.
  //
  // This call is not tied to the module that this plugin is responsible for. A plugin of each language will
  // be chosen to generate stubs for each module.
  rpc GenerateStubs(GenerateStubsRequest) returns (GenerateStubsResponse);

  // SyncStubReferences is called when module stubs have been updated. This allows the plugin to update
  // references to external modules, regardless of whether they are dependencies.
  //
  // For example, go plugin adds references to all modules into the go.work file so that tools can automatically
  // import the modules when users start reference them.
  //
  // It is optional to do anything with this call.
  rpc SyncStubReferences(SyncStubReferencesRequest) returns (SyncStubReferencesResponse);
}