# 20200601 The future of the FCOS pipeline

## Kickoff email

Hi Everyone!

I am gathering everyone here together to discuss the future of the build pipeline
for Fedora CoreOS. Currently we use Jenkins to orchestrate our build pipeline running
in an OpenShift project within apps.ci.centos.org. The problems our team are facing at
the moment are two fold:

- Future Infrastructure:

The current set up (jenkins on top of openshift in apps.ci.centos.org) has treated us
well. There has been the occasional infrastructure issue that need some attention, but
it hasn't been too bad. What does concern us is the age of the software (currently OpenShift
3.6), which is pretty out of date at this point. I believe I have heard some rumblings of
an OpenShift 4 environment to replace it with, but I'm not sure where that stands.

- Multi-arch

We currently only build x86_64 artifacts. We schedule builds on top of an x86_64 OpenShift
cluster using the Kubernetes plugin for jenkins. Access to OpenShift clusters for each
architecture (aarch64, ppc64le, s390x) would be ideal to achieve parity with what we are
currently doing for x86_64, but probably not likely to happen. We need to figure out how
we can incorporate the great work that Jakub and team have been doing [1] so that our
multi-arch artifacts are first class citizens. There are a few other upstream tickets
related to this too [2] [3].

My goal here is to try to schedule some time for us all to discuss possible answers to the
above questions and to brainstorm on what the best future build pipeline for FCOS looks like.
I will try to set up some time early next week to discuss this and see if we can make progress
on this front.

Dusty

[1] https://lists.fedoraproject.org/archives/list/[e-mail]/message/CLZZE3UKOTLDQQRMDC2Z4MKBOIND64BO/
[2] https://github.com/coreos/fedora-coreos-tracker/issues/262
[3] https://pagure.io/fedora-infrastructure/issue/8370

## Monday 06/01 Meeting Notes

### Future OpenShift Infrastructure in CentOS CI

- Update: CPE team is working on updating OpenShift in CentOS CI
- [link](https://lists.centos.org/pipermail/ci-users/2020-May/002088.html)
- bstinson: we have the new OCP 4 cluster running right now and we're starting
to migrate high value tenants to the new cluster (target end of the month)

### Multi-Arch Options

- Notes:
- jligon: Other parties we may want to bring in to the multi-arch conversation
- Dennis Gilmore
- Andy McCrae
- bstinson: we're looking at scheduling jobs on other clusters (different architecture) from jenkins
- bstinson: we're not sure about our timelines for being able to do multi-arch in CentOS CI. Also managing multiple clusters is a lot of burden.
- jakub: having a multi-arch cluster is something we're lacking on the community side and having it would help more than just FCOS
- cverna: managing multiple clusters is hard to manage; common solution for everyone would be best if we end up doing it
- pingou: do we have multi-arch hardware available?
- bstinson: shopping around for a cloud hosting provider for multi-arch would be the best option for us
- jligon: we're working on the BU side to try to establish relationships to support cloud hosted multi-arch hardware
- dusty: can we work with cloud providers to get free credits?
- pingou: if we go with UPI aren't we just managing things ourselves again?
- bstinson:
- jakub: getting a VM on different arch should be easy, but getting managed openshift for non x86_64 is probably not possible
- dusty: is managing the cluster via UPI still too much for us to manage ourselves?
- bstinson: still a lot of work to manage an openshift cluster even if we're abstracted away from the hardware
- cverna: for aarch64 we have the hardware and we need a cluster for OSBS so we'll have that
- mohan: multi-arch team has a few new s390x machines that we might be able to get some VMs on
- jakub: what is really needed to get FCOS building multi-arch?
- jlebon: access to hardware is the biggest thing
- jlebon: plumbing through multi-arch support to COSA and the pipeline
- jakub: 1st step for all of this might be to do multi-arch builds of COSA and pushing to quay
- dustymabe: do we use SSH to VMs? do we try to re-use koji?
- jlebon: whatever we do we'll want to make sure we can still run a COSA via a container
- jlebon: we can set up a VM as a jenkins worker and then we can run things in a container or directly on the machine
- dustymabe: does Fedora have any ppc64le VMs that they could give out?
- mohan: Fedora hasn't given out an ppc64le VMs for special projects at this point
- jakub: we should be able to get a VM from one of the universities to use for this if we asked
-