# CI / CD discussion

Conceits:

* We're all overburdened right now
* The people most capable of rewriting our CI infra are precisely those who are the most overburdened
* Short-term our best option is probably to ask for more minutes and defer for 4-5 months while we investigate our next move

Benefits to Travis (it's worth discussing):

* It's what we're already using
* It does support some features that other CI don't
* Nested virtualization
* POWER and SystemZ support, funnily enough

Drawbacks of Travis:

* Got acquired by a private equity firm, fired senior engineers
* Not receiving much new feature development
* Stability has been bad at times (has been ok lately)
* Slow
* Situation unlikely to improve much

Benefits of switching to GitLab CI *OR* GitHub Actions:

* Both highly acclaimed, considered to be the best CI services available
* Both offer effectively unlimited CI minutes
* GitLab has a 50,000 minute allowance
* Github unrestricted
* Probably faster hardware than Travis
* Vastly better documentation
* Vastly better APIs
* Ability to trigger some CI tasks only when certain files / directories change
* Ability to add custom runners to the pool

Benefits of Github Actions:

* Likely the best "integration" of any mentioned tool with Github
* Likely the biggest ecosystem of any mentioned tool
* Large marketplace of custom "actions"
* Can write your own re-usable "actions" to do complicated custom things
* Focus on configuration of "actions" rather than templated code
* Likely to be a dominant CI ecosystem due to Github's wide reach
* Easier to find in-the-wild examples

Downsides of Github Actions:

* Does not re-run jobs individually, when a job fails, you need to re-run the entire workflow

Benefits of Gitlab CI:

* Has a *lot* of neat features, many that even GHA doesn't have
* Downside: A lot of the most useful features require buying into the whole ecosystem (repo-hosting, issues, etc.)
* e.g. Built-in "merge when CI passes" button
* Lots of niceties for "microservice-y" architectures
* e.g. Events in one repo can trigger the CI pipelines of other repos or call directly into their CI pipelines to run parts of them
* In GHA each repo is silo'd
* FOSS use comes with a free premium account, which means the entire suite of workflow tools https://about.gitlab.com/pricing/gitlab-com/feature-comparison/
* Most of which aren't very useful without buying into the whole ecosystem
* We have discussed moving away from Redmine, so this is possibly worth considering, but it's a whole other discussion
* (GitLab does support issue relations, confidential issues, fancy roadmaps, kanban boards, etc.)
* GitLab is an open source company, most of their code is open-source

The GitLab CI infrastructure is really powerful and I'm pretty pleased with
what we're able to do with it for libvirt and QEMU upstream. It is especially
good if your project is making full use of other GitLab features, most
especially merge requests, but it is useful even if you don't.

Historically GitLab has had unlimited CI minutes for any public project,
but they're intending to introduce a fairly low cap, with elevated minutes
for projects that apply for their open source program. Alternatively though
you can bring your own compute resources and register them as a "runner"
for your project, so you get the full use of the CI system integration with
GitLab, without having to worry about limits.

I'm using GHA because...:

- Ecosystem (lot of actions available and easy to create)
- Setup: just drop a yml file and you're ready, it will use it, so it's
easy to replicate settings from one repo to another by just copying
the yaml file.
- Above, also makes it easy to find something you like in another repo
and setup something similar to yours
- GH Pages... an action can run and publish to another branch so you can
build quickly updated pages with more powerful builders than the
standard jekyll (you can even run your own jekyll with extra plugins,
etc).

To add some more Red Hat projects using GH actions:

* https://github.com/quarkusio/quarkus
* https://github.com/quarkusio/quarkus-images
* https://github.com/graalvm/mandrel
* https://github.com/graalvm/mandrel-packaging
So far for me the only benefit of travis over GH is that travis provides aarch64 runners while GH does not.

So if your project is hosted on GitHub it's totally worth considering it.

We use Gitlab CI at work extensively and I'm using Github Actions in my free time.

Gitlab CI is much more mature, as expected, but isn't infallible. While the extend syntax is good nowadays, the matrix stuff with Github Actions is IMO better.

Gitlab CI also has some weird kinks. For example, when specifying the environment of a job (i.e. so that Gitlab displays the SHA that is deployed to Prod) it breaks our Terraform pipeline. No joke, and nobody has any idea why.

Github Actions also has a lot more "support" from the community, with e.g. custom actions, templates and so on. Gitlab CI has some templates as well, but most of the stuff is DIY.

Gitlab CI is more lower level, where you can decide which docker image to build on. Github Actions is more like a VM, where you can install what you want, but a "docker image" is encapsulated in a custom actions. So you could write a custom "action" called "base image" that basically does the same as Gitlab CIs Job Image specifier, but it's more work.

Overall I think I'd use Gitlab CI if I want to selfhost, but Github Actions if I'd plan to use the cloud variant of either.

Short summary from my experience:

GitLab CI:

[+] CI runners can be self-hosted (even if you use cloud GitLab), helps with resource-hungry jobs
[+] Very feature-rich pipeline config, lot of complex things (caching for example) can be configured to "just work"

[-] Large pipelines can be hard to express in YAML, need to make heavy use of anchors to make it maintainable
[-] Docker runners are much better supported than native runners

Github Actions:

[+] Good cross-platform support, trivial to cover Linux+Windows+MacOS with the same CI script
[+] Actions can be fetched from a community marketplace, lot of readily-available things

[-] Anything that is not covered by an existing action / YAML config needs to be implemented in TypeScript
[-] Less capable in expressing complex pipeline topology

In these repos we actually dynamically generate the parallel portion of our test pipeline in GHA, by creating a JSON array from "tox -l" with all the environments we want to run and feeding that into the second stage of the pipeline. This allows your builds to become nearly arbitrarily complex without making maintenance of the tooling itself a bottleneck:
https://github.com/oasis-roles/ansible_collection_system
https://github.com/oasis-roles/ansible_collection_osp

In the world of Molecule we moved to GHA somewhat recently:
https://github.com/ansible-community/molecule/
https://github.com/ansible-community/molecule-docker/
https://github.com/ansible-community/tox-ansible/
etc

Here's where I build Vagrant VMs with GHA:
https://github.com/greg-hellings/packer-templates/

And here's where I use it to package Windows builds of some utilities and release them when the time is right:
https://github.com/devroles/mingw_sword_package

I didn't find it [GHA] very hard to set up for a simple project. My preferred approach to CI is to set up as much of it as possible in the project itself, so all I have to teach the CI system itself to do as little as possible (this makes porting between CI systems, and running the tests locally, easier). I have a project in Github because it's part of an org that's in Github: https://github.com/os-autoinst/openQA-python-client it's a Python library, and as I usually do I have the testing set up mostly through tox: https://github.com/os-autoinst/openQA-python-client/blob/master/tox.ini so it can equally well be run in most any CI system or just on your local system. To teach Github Actions to run it, this was all that was needed: https://github.com/os-autoinst/openQA-python-client/blob/master/.github/workflows/tox.yml (it runs on Ubuntu because I don't think Fedora or RHEL are choices :>)

The difficulty of setup is going to vary a lot depending on how much work you need the CI system to do, of course. But I like things that make it simple to do simple stuff, and GHA passes that test. Getting that working took me about half an hour.

I much prefer Gitlab CI. In most cases building my applications needs 2 commands, so just specifying the docker image + the 2 or 3 build steps works really well. Actions are a lot more abstract, so I found it much more a PITA. Gitlab also allows you to make custom runners, which I used to clone specific proxmox templates and run builds in there, since the build environment was super complicated and specific.

I also host my own runners, which is a lot faster than any hosted variant I tried so far and only costs me 30 bucks a month for 15 people to have all pipelines complete in 3 minutes.

###### tags: `CI/CD`