Flamenco 3 (blog post draft)

Almost 5 years after the release of Flamenco 2.0, development has started on a new version: Flamenco 3**.

In this blog post Sybren Stüvel describes the current vision on Flamenco 3, with the intent to discuss this with anyone interested, get feedback, and improve the design.

Flamenco is the Open Source render farm software, developed by Blender Animation Studio. It is aimed at performing tasks such as frame rendering, and video encoding. It supports any Blender version, hardware configuration and any other software with a command line interface.

Target Audience

Flamenco is meant for smaller animation studios and individuals at home. Think of roughly 1-10 artists using it, and 1-100 computers attached to the farm to execute tasks. Blender Animation Studio uses various desktop machines for the render farm when they're not used by the artists, and some developer machines are powerful enough to run Flamenco and work on coding Blender at the same time.

Design Principles

The following principles guide the design of Flamenco 3:

Blender.org Project

Flamenco is a true blender.org project. This means that it's Free and Open Source, made by the community, lead by Blender HQ. Its development will fall under the umbrella of the Pipline, Assets & IO module.

Minimal Authentication & Organisation

Because Flamenco is aimed at small studios and individuals, it won't offer much in terms of user authentication, nor the organisation of users into groups. The first version will likely just have a "name" field when submitting jobs, and people are trusted to not mess with other people's jobs.

Minimize External Components

Running Flamenco should be extremely simple. This means that it should depend on as few external packages as possible. Apart from the Flamenco components themselves, all you need to install is Blender, ffmpeg, and a database.

The downside of this is that development might take longer (because some things that an external service could solve need to be implemented).

No Errors, Guide Users To Success

Instead of stopping with a "no database configured" error, Flamenco should show a helpful interface in which you're guided towards a working system.

Customisabe

Studio pipeline developers / TDs should be able to customise the behaviour of Flamenco. They should be able to create new job types, and adjust existing job types to their needs. For this, Flamenco 3 will most likely be using JavaScript to convert a job definition like "render this blend file, frames 1-100" into individual tasks for computers to execute.

These custom scripts should also be picked up by the Blender add-on that's used to submit files to Flamenco, so that there is just one place where customisations take place.

Work offline

Like Blender itself, Flamenco should be able to fully work offline. That is, work without internet connection. If any future feature should need such a connection, that feature should always be optional, and be disabled by default.

Data Storage

Data should be stored as plain files whenever possible. Where a higher level of coordination is required, a database can be used.

Infrastructure & Supported Platforms

Setting up a render farm is not as simple as pushing a button, but Flamenco aims to keep things as simple as possible. What you need to run Flamenco is:

• One or more computers to do the work, i.e. running Flamenco Worker. These machines also need Blender, of course, and ffmpeg for video processing.
• A computer to run the central software, Flamenco Manager. This could be one of the above computers, or a dedicated one. At the moment it also needs a PostgreSQL database (but see my questions below).
• A local network with file sharing already set up, so that the above computers can all reach the same set of files.

Since Blender Animation Studio fully runs on Open Source software, Linux is the main platform Flamenco is developed for. Windows and macOS will also be supported, but will need help from the community to get tested & developed well.

Development Plans

Sybren has been working on a proof of concept, which has now been published in the official Flamenco repository on developer.blender.org. Anna Sirota, Pablo Vazquez, and Francesco Siddi are also part of the team.

It will take several months of development to produce a minimal viable product.

My Questions To You

The goal of this post is not only to inform, but also to get feedback.

• What do you think about the design principles described above?
• If you considered using Flamenco 2 before, but didn't, what were the main hurdles?
• How do you feel about having to install Blender, FFmpeg, and a database? And then tell Flamenco where to find those?