Notes on The Codex CI
=====================

## Goals of a CI System

The main goal of a CI system is ensure that the system is always in a releasable state. This is the main reason tests run on every commit, and that's also the reason why tests are run on branches - to make sure that those, too, are releasable.

## The Codex CI System

The Codex CI system:

* ensures Codex is releasable by compiling the codebase and running its test suite;
* builds and pushes runnable images for usage by end users;
* builds and pushes runnable images for usage in distributed testing.

## Nim Compiler Versioning

The Nim compiler is required at three main points in the Codex lifecycle:

1. during development;
2. during CI, where Codex needs to be compiled before tests can run;
3. during builds of the Docker images, where again Codex is compiled as part of a multistage build.

Currently, the versions of the compiler used at each of these stages can, and do, diverge.

**In local development**, the version that gets used is, by default, the one adopted in the [nimbus-build-system](https://github.com/status-im/nimbus-build-system/tree/master/vendor) (1.6.16 at the time of this writing), but can be overriden locally with a `NIM_COMMIT` parameter.

**In CI**, [the version is pinned by the `ci.yml` file](https://github.com/codex-storage/nim-codex/blob/master/.github/workflows/ci.yml#L10), and is at 1.6.14 at the time of this writing.

**In Docker builds**, the version is again the one from nimbus-build-system, again at 1.6.16 at the time of this writing, and cannot currently be overridden.

This has several shortcomings:

1. you cannot currently use a custom compiler in Docker images;
2. even if enabled this by, say, adding an extra "nim_version" parameter that affects only the Docker builds, the nim compiler versions in Docker image, local development, and CI would still be allowed to diverge. Indeed, you'd have to keep all these parameters in sync, demanding extra care and cognitive overhead (manual work) even in development branches. I argue that this is for no good reason, as one would hardly ever (if ever) want these versions to be different.

This is particulary painful when one wants to experiment with a custom compiler on their own branch, and wants to run both CI and dist tests on it. Since Nim is still a very experimental language and we oftentimes need to touch the compiler, this will likely happen more times in the future.

Being able to branch and experiment with newer versions of the compiler is also important, and I foresee we will face the same problems soon as we try to run this against `2.x`.

## Proposal

To remedy the aforementioned issues, I propose that:

* CI, Docker images, and local development should always reflect the same compiler version;
* the compiler version is made to be a part of the Codex codebase, not scooped implicitly from nimbus-build-system.

This would allow one to easily create a branch, set their own version of the nim compiler, and then commit, knowing that:

1. anyone that forks that branch will be using that same version of the compiler;
2. any results from CI will be derived using that same version of the compiler;
3. any Docker image built from that branch will be built using that same version of the compiler.

I made a [pull request](https://github.com/codex-storage/nim-codex/pull/680) which suggests [one very simple way](https://github.com/codex-storage/nim-codex/blob/d7e62bb0c203a8b6ee4e15702d26e68d91ef4d8f/env.sh#L4) to implement this. This is not the only way. However, provided that the we agree on the goals, implementation can always be improved.

## Supporting a Compiler Compatibility Matrix

Dmitriy pointed out that although we will select a version of Nim as the "preferred" or "LTS" one, we still want to be able to assert that we support several versions of the compiler.

I am going to assume here that:

* compatibility means that the code compiles and the the test suite runs on CI for a given targeted Nim version;
* Docker images do not need compatibility builds, as end-users are not required to install the compiler.

With those assumptions in mind, my current PR does not prevent this from being implemented in a very simple fashon. One could easily reintroduce `NIM_COMMIT` from a matrix and run the CI for each, which would then pick up the desired versions.

The reason I've removed this from there right now is because:

1. we're not running CI against a matrix right now (there is only one version);
2. given that we have only one version right now, the only thing that input was achieving was causing CI to diverge from everything else.