# Future Flamenco Features

## MongoDB to relational database

Moving forward I want to get rid of MongoDB. It's been bundled with Flamenco Manager for ease of deployment, but it's never gotten really stable.

### Database Choice

- ❓ **PostgreSQL:** good, fast, real database, but requires installation. I've never done that on anything but Linux, so not sure about how simple/hard that is. [Pure Go bindings](https://github.com/jackc/pgx), making cross-compilation easy.
- ❓ **SQLite:** ~~Go bindings require C compiler, making cross-compilation hard.~~ [Pure Go bindings](https://pkg.go.dev/modernc.org/sqlite) available. Requires no installation. Easy to start from scratch, but can make Manager upgrades slightly harder (need to copy database + config file around).

### Database Interface

- ✅ **Plain SQL:** probably good enough for us, and lowers learning barrier. Easier to make fast queries.
- **ORM:** could make it easier to support multiple databases in the future.

### Data Migration

- Worker data: migration might not even be necessary, given that workers automatically reregister anyway. Would loose blacklists.
- Any task claimed by the Manager needs to either be "given back" to the Server or migrated. Or we decide that it's not going to be a drop-in replacement, and that jobs will simply have to be resubmitted.

## Move Server functionality to Manager

Requires quite a few different topics.

### Job Compiler: Scripting

Conversion of **job description to concrete tasks** to execute needs to be reimplemented on the Manager. This also requires conversion of Python code to something else. This is probably a good time to reconsider the way this is done.

A possible direction would be to use some form of **scripting engine** in order to have a user-expandable set of scripts. These could make it possible to declare custom job types, which is something that is desirable by a lot of studios.

Suitable (i.e. pure-Go) scripting engines:
- ✅ [Goja](https://github.com/dop251/goja): ECMAScript, so known language but could be a big library, or
- [Tengo](https://github.com/d5/tengo): Its own language, but small & fast.

Alternative: use a declarative way to define job types, which could be visually represented as a node tree (like Scratch). This could be expressed in YAML, or maybe somehow in ECMAScript or Tengo.

### Authentication

Flamenco Manager should no longer require Blender Cloud for authentication. Either it should have its own user database, or use an OAuth server directly, or even LDAP. Probably best to have some **pluggable auth structure**.

Note that this is independent of the authentication of the Manager itself, when communication with Flamenco Server. I think the current approach can be kept.

### Projects & Users

**Big fat TODO:** how to deal with users (authentication, authorization) and project membership.

### Communication with Blender

It probably is a good idea to create a **new add-on** for this, so that it can be disconnected further from the Blender Cloud service.

Communicating directly with Blender, the Manager could **expose its supported job types and their parameters**. This can then be used to dynamically generate a UI for submitting jobs.

### Communication with Workers

- ❓gRPC: modern protocol, high performance, has a constant connection so communication can go two ways. Also supports streaming messages, which could be useful for handling logging.
- ❓REST/OpenAPI3: allows generating web UI for exploration & documentation, can also be used from a webbrowser.
- [demo repo](https://github.com/MarioCarrion/todo-api-microservice-example/tree/074bbb9f4d0f79e5bced943c10c56013705969a9)

### Task life cycle: updates from Workers

Task updates have to be handled solely by the Manager, instead of relying on the Server. This means that the task state machine has to be reimplemented on the Manager.

### Task & Job management: updates from Server

Flamenco Server will have to handle task updates differently. It will get dumber and not handle any automatic task status changes. It should just forward such requests from the web interface to the Manager.

There should be APIs for task/job management and an API for monitoring task/job updates/progress/logging/etc.

### Web interface

The Manager web interface will have to become considerably more advanced, in order to take over all the job & task management functions of the Server.

### Expand job & task metadata

Jobs and tasks should have metadata describing the project-level task that they relate to, the shot, the assets, etc.

There should be an API to query these things, and to report related data (job/task aggregated status, render times, etc.).

## Flamenco Server

We have to consider **reimplementing Flamenco Server** as standalone service, as a cross-Manager dashboard and not much more. It will be much easier to work on and deploy than to try and keep the current implementation afloat.

## Technical Debt

Flamenco Manager was my first nontrivial Go program. Since then I've learned a lot about the language, and I want to reimplement/refactor some Manager parts.

Flamenco Worker was an attempt to use Python's then-new `async` system. In hindsight this was a bad decision, and caused more trouble than it's worth. Time permitting, I'd like to **reimplement the Worker in Go**, as this will make cross-platform builds much easier, and also will ease deployment for users. In my spare time I've already started working on a [Go implementation of the Worker](https://gitlab.com/dr.sybren/flamenco-worker-go).

As for the "command" implementation in the Worker, the new Go implementation will follow the same approach as the Python implementation (i.e. hard-coded functions for executing commands, with knowledge about what's happening instead of just a CLI runner).