# Gating podman and its friends github repo using Zuul

## Problem
* In tripleo, OpenStack services are deployed in containers.
* Container lifecycle, healthcheck and execution managaed via systemd using paunch.
* Paunch under the hood uses podman.
* Updating a new version of podman does not gurantee backward compatibility often leads to performance issue
* Above scenario is out of the scope of podman testing suite as it requires real deployment.
**Every time podman versions are bumped, that is a manual event**

### Repo/Epic link:
* podman - https://github.com/containers/libpod
* Epic: https://tree.taiga.io/project/tripleo-ci-board/epic/1271

### Current workflow in RDO
* container team releases a new version of podman on github and build the new rpm on centos build system.
* Tripleo team updates the podman version in rdoinfo/buildsys-tags/cloud7-openstack-{release}-testing for a particular release
* it triggers following dist-git jobs defined in rdo-jobs(https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/distgit-jobs.yaml):
* rdoinfo-tripleo-master-centos-7-scenario001-standalone
* rdoinfo-tripleo-master-testing-centos-7-multinode-1ctlr-featureset010
* rdoinfo-tripleo-master-testing-centos-7-standalone-featureset052
* Example review: podman 1.5 update in RDO-> https://review.rdoproject.org/r/#/c/21994/
* Above jobs are defined here: https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/projects.yaml
* Once the above job passed, the same podman build got retagged in CBS with cloud7-openstack-{release}-testing.
* Now TripleO CI job, podman comes from dlrn-deps repo which is maintained here: https://github.com/redhat-openstack/rdoinfo/blob/master/deps.yml
* Then it passed it to the promotion pipeline from where we find the breakage and promotion issue

## Goal
Having a tripleo-ci standalone job with selinux enabled getting triggered using Zuul
on each podman pull requests will provide an early feedback to TripleO developers.

## Design Proposal

### Few considerations:
* we need a lightening fast job
* The longest job on podman is special_testing_rootless TEST_REMOTE_CLIENT:true -> 31 mins
* We need a job with CentOS7 and RHEL8 with selinux

### Proposal
* Integrate podman github repo with Zuul:
* Install the SF github app to podman - Zuul will then receive event and be able to report CI result
* https://softwarefactory-project.io/docs/user/zuul_user.html#install-a-github-app
* https://github.com/apps/softwarefactory-project-zuul
* Add podman to zuul rdo tenant in softwarefactory-project.io/r/config
* Create a project pipeline in rdo config
* Trigger zuul job on github PR using comments
* Make the pipeline to listen on each pull requests
* Define new jobs in rdo-jobs
* Set of playbooks needed:
* Create mock environment
* Build the podman rpm
* Run this script: https://github.com/containers/libpod/blob/master/contrib/build_rpm.sh
* Build the buildah rpm
* https://github.com/containers/buildah/blob/567141716e5802c556a53e3ec6021105e5e17a09/tests/test_buildah_rpm.sh#L80
* Create the repo
* rewrite the release-file to include the above repo
* For example: https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/run-distgit.yaml#L111
* Then trigger the standalone job
* Once the job finishes, get back the results
* In job results, Highlight whether the package is built and consumed successfully
* Add the job to cockpit to monitor and find a way to highlight the failure and performance

### Requirements from podman team:
* In order to integrate software factory github zuul app with containers/podman github repo
* We need permission from containers/podman repo owner

### Open Questions (Got answered):
As per epic the proposal was to gate each pull request with a fastest tripleo job (basic standalone). It brings some questions.
* Why to add a tripleo Job against podman pull requests?
* Does the current podman test suite does not bringing any value to us?
* ( Question for Emilien )What coverage is missing in podman test suite for openstack cases.
* The way we manage containers lifecycle with systemd is done in openstack/paunch. But we are never sure that it won't break in the next release.
* The way we handle the container healthchecks with systemd timers is also done by paunch and again we don't want it to break.
* Also, TripleO is a heavy user of "podman exec" and we have found performances issues with it, that weren't seen in the podman CI.
* At least, we could have a job with SElinux enabled, so we would catch much earlier than now the AVCs.
* Note that TripleO has been the first real user of Podman and its dependencies.
* ( Question for Emilien )Would augmenting the podman testcases themselves be a better way to test podman for tripleo vs. running a tripleo deployment?
* No because their testcases aren't really designed to deploy podman within an ecosystem of other tools like we do with TripleO. It would be a lot of social and technical work to do, while we can just use tripleo for "free".
* Since container team know podman better as TripleO is just a consumer and running tripleo job against each pull request does give any value to podman developers?
* Who is going to maintain this Job, Who is going to look after the failures?
* Putting the job in cockpit and montior from there?
* TripleO deployment have it's own sort of failures, means failures may not be due to podman PR, how podman/container team will deal with this?
* Do we want to run tripleo jobs on each file changes in podman?
* Do we need to design some experimental jobs?
* Will single job will cover all the issues we have seen with podman till now?
* How we can handle failures which occurs randomly(few failures in multiple runs) with just running a single job?

## TODO
* Findout podman friends
* Explore distro-zuul-jobs
* build-test-packages and distro-zuul-jobs similarity