# Validation-Framework CI ###### tags: `Design` Proposal to modify how the validation-framework is currently working in CI. Noting that the validation-framework itself is a DF/tripleo responsibility, the actual tests and results from those tests are owned by other projects and DFG's ## TripleO Validations https://docs.openstack.org/tripleo-validations/5.1.2/readme.html And if you don't know, now you know. ## Fix the TripleO gate turn off validate_services that are currently running in upstream check or gate patch that can break the tripleo gate [patch1](https://review.opendev.org/#/c/705051/) [patch2](https://review.opendev.org/#/c/705178/) [patch3](https://review.opendev.org/#/c/705177/) ## Execute validations in the promotion pipeline Since the validations are owned by non-tripleo projects ensure the validations are run in the promotion pipelines where non-tripleo content is validated prior to entering upstream. :::info Note, I see validations were turned on upstream but not in the promotion jobs, this should never be the case. ::: ***Compare: name: tripleo-ci-centos-7-scenario010-standalone*** * upstream - https://review.opendev.org/#/c/705051/3/zuul.d/standalone-jobs.yaml * promotion job - https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/standalone-jobs.yaml#L153-L166 The pattern should be exactly reversed, the validations should be turned on in promotion but not upstream check or gate. ## Define a single scenario and standalone to test the validation framework itself. * The validation team can pick a current scenario and copy it to a new number. This way they can edit the services they want as needed. [tripleo deployment scenarios](https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/ci/environments) * Define a standalone job that executes the above scenario in upstream and promotion. * https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml * https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/standalone-jobs.yaml :::info Name the job "tripleo-ci-validations-something-something" ::: :::success This should give the validation framework team the room and flexibility to test the framework as they see fit while also assuring the tripleo gate does not fail. ::: ## Future points. The CI team is building out a [component pipeline](https://hackmd.io/5uYRmLaOTI2raTbHWsaiSQ) As we define jobs in this pipeline I think it would be a great idea to add as much of the validation framework as possible here per component. #### Customizing component pipeline jobs We will be adjusting the tempest tests per component to heavily hit only the specific component under test. E.G. if the we're running a compute component job expect a lot of: * tempest.api.compute * tempest.scenario.test_volume It would be a really good idea to follow the same pattern for the validation-framework. * add strict validations back in * ensure validations are configurable * ensure failing validations can work w/ a skip list (may already be avail.) #### The component pipeline will run both upstream and internally w/ osp. A mock up of a component pipeline dashboard is viewable [here](http://tripleo-cockpit.usersys.redhat.com/d/Wj-ao_sWz/full-component-pipeline?orgId=1) ## Summary / tldr It's not appropriate for validations to fail in the tripleo upstream gate unless the validation framework itself has failed. * Limit the exposure upstream * Create a specific job for the validation team to work with upstream. * Heavily test validations in the promotion pipeline * Incorporate validations into the component pipeline planning.