Validation-Framework CI

tags: Design

Proposal to modify how the validation-framework is currently working in CI.
Noting that the validation-framework itself is a DF/tripleo responsibility, the actual tests and results from those tests are owned by other projects and DFG's

TripleO Validations

https://docs.openstack.org/tripleo-validations/5.1.2/readme.html
And if you don't know, now you know.

Fix the TripleO gate

turn off validate_services that are currently running in upstream check or gate patch that can break the tripleo gate patch1 patch2 patch3

Execute validations in the promotion pipeline

Since the validations are owned by non-tripleo projects ensure the validations are run in the promotion pipelines where non-tripleo content is validated prior to entering upstream.

Note, I see validations were turned on upstream but not in the promotion jobs, this should never be the case.

Compare:
name: tripleo-ci-centos-7-scenario010-standalone

The pattern should be exactly reversed, the validations should be turned on in promotion but not upstream check or gate.

Define a single scenario and standalone to test the validation framework itself.

Name the job "tripleo-ci-validations-something-something"

This should give the validation framework team the room and flexibility to test the framework as they see fit while also assuring the tripleo gate does not fail.

Future points.

The CI team is building out a component pipeline

As we define jobs in this pipeline I think it would be a great idea to add as much of the validation framework as possible here per component.

Customizing component pipeline jobs

We will be adjusting the tempest tests per component to heavily hit only the specific component under test. E.G. if the we're running a compute component job expect a lot of:

  • tempest.api.compute
  • tempest.scenario.test_volume

It would be a really good idea to follow the same pattern for the validation-framework.

  • add strict validations back in
  • ensure validations are configurable
  • ensure failing validations can work w/ a skip list (may already be avail.)

The component pipeline will run both upstream and internally w/ osp.

A mock up of a component pipeline dashboard is viewable here

Summary / tldr

It's not appropriate for validations to fail in the tripleo upstream gate unless the validation framework itself has failed.

  • Limit the exposure upstream
  • Create a specific job for the validation team to work with upstream.
  • Heavily test validations in the promotion pipeline
  • Incorporate validations into the component pipeline planning.
Select a repo