Full end to end CI for ceph-ansible

Problem

ceph-ansible is used in TripleO deployment using ceph-ansible.
It basically runs in scenario 1 and 4 jobs where a released ceph-ansible rpm is consumed.

Repo link

• ceph-ansible: https://github.com/ceph/ceph-ansible

Epic Link: https://tree.taiga.io/project/tripleo-ci-board/epic/1053

Current Workflow

New version of ceph-ansible lands in CentOS Build System and is consumed through storage repo in TripleO CI job. Currently Ceph developers, updates the release file for all release with the new base repo url from ceph-nautilus testing tag of the ceph repo and then tests it in scenario-1 and scenario-04.

For example: https://review.opendev.org/#/c/501810/ updates the centos testing repo for ceph
and https://review.opendev.org/#/c/501987/ a patch against tripleo heat templates to trigger the tripleo based scenario-01 and scenario-03 got triggered.

It is a manual process and it requires a lot of effort.

Vision

In order to catch issues in ceph-ansible which affect ooo deployments before they get into ceph-ansible releases.

Proposal

trigger one of the existing tripleo jobs on every pull-request for ceph-ansible and publish back the results into a pull-request comment

Work down till now

Integrate ceph-ansible github repo with Zuul

• Install the SF github app to ceph-ansible - Zuul will then receive event and be able to report CI result

  • https://softwarefactory-project.io/docs/user/zuul_user.html#install-a-github-app
  • https://github.com/apps/softwarefactory-project-zuul

• Add Ceph-ansible to zuul rdo tenant in softwarefactory-project.io/r/config

  • https://softwarefactory-project.io/r/#/c/14203/

• Create a project pipeline in rdo config

  • Add github-check pipeline and noop job
    • https://review.rdoproject.org/r/#/c/17456/
  • Add ceph-ansible Zuul job configuration
    • https://review.rdoproject.org/r/#/c/17600/

• Trigger zuul job on github PR using comments

  • https://github.com/ceph/ceph-ansible/pull/3576

Current Jobs defined for ceph-ansible Integration

• tripleo-ceph-integration
  • https://github.com/rdo-infra/rdo-jobs/blob/06f406bfb2ba8df46d8271aa0689d11af95b5e4c/zuul.d/ceph-ansible.yaml
  • What above jobs are doing?
    • https://github.com/rdo-infra/rdo-jobs/blob/06f406bfb2ba8df46d8271aa0689d11af95b5e4c/playbooks/tripleo-ceph-integration/pre.yaml
      • Installing required packages and adding zuul user to mock group
    • https://github.com/rdo-infra/rdo-jobs/blob/06f406bfb2ba8df46d8271aa0689d11af95b5e4c/playbooks/tripleo-ceph-integration/run.yaml
      • Generating rpm using make rpm
      • Using createrepo to create the repo
    • https://github.com/rdo-infra/rdo-jobs/blob/06f406bfb2ba8df46d8271aa0689d11af95b5e4c/playbooks/tripleo-ceph-integration/post.yaml
      • Copy rpms and artifacts to zuul root log directory

Work pending

• Since rpm is already generated and repo is there, we need to pass this repo to other job
  • where rpm will be gets consumed in standalone scenario 1 & 4 jobs
    • https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/run-distgit.yaml#L111
• Build test packages needs to be improved if depends-on of ceph-ansible patch used in rdo reviews or vice versa
  • [better] Improve build-test-packages to build the rpm itself there as just like normal jobs by parsing git commit message
• Need to find a list of files on which we need to trigger the job on each ceph-ansible PR
• Add the job in cockpit for monitoring and maintaince

Some reuse of the tools

• Reuse distro-zuul-jobs to all DLRN or rpm related operations
  • https://pagure.io/zuul-distro-jobs

Open Questions:

• Do we want to run tripleo-jobs on each PR?
  • IF yes, then we need to find list of files going to change on which tripleo jobs will be triggered?
  • all ceph-ansible stable-4 branches trigger tripleo train
  • all ceph-ansible stable-3 branches trigger tripleo queens (or rocky)
• Except scenario 1 & 4, is there any other fastest job we can run?
• What about adding a post pipeline which got triggered automatically if a new release of ceph-ansible comes?

Design Approach on passing yum repos file to TripleO CI Job [needs discussion with team]

• Create a new release file and override the existing tripleo-ci release file

  • https://github.com/rdo-infra/rdo-jobs/blob/master/playbooks/run-distgit.yaml#L111
  • Since release files are less updates
  • It might be error prone.

• By Passing the rpm created by parent job as a zuul artifacts to the child job where tripleo-ci job will run

  • https://zuul-ci.org/docs/zuul/user/jobs.html#user-jobs-parent-results
  • https://zuul-ci.org/docs/zuul/user/jobs.html#return-artifacts

• Introducing a custom var in release file which can be setted with the reponame created from the ceph-ansible/podman
  repo and enable it in the release file and enable the repo

• Get the pull request and build the rpm there and install it from here in the job.