molecule on tripleo repos

Notes for upcoming meeting, feel free to add your comments or questions.

Keep in mind that this document refers only to molecule scenario testing, not our integration testing.

Who owns molecule scenarios?

  • Who is expected to fix broken scenarios?
    • @zbr: same as other tests like unittests or linters: the repository owner.
    • @zbr: based on the last year scenario failures, I do not remember when it was last time when we got hit by a test framework bug. Most of the time it was an external factor like os image change, container engine changes, nodepool image change or code change but not the test framework itself.
  • What if it proves to be bug in molecule or a plugin?
    • @zbr: happy to help, including with a speedy hotfix if needed

Consolidate base container images

Last week we realized that many molecule jobs were using outdated centos8 images from docker.io, instead of stream ones. Images from docker.io/centos images are not even published by redhat, but the ones from quay.io/centos/centos are.

The tags that we care about are name steam8 and centos7, with the remark that centos8 one should be avoided as being a dead end.

Based on images above, @zbr created last week some with preinstalled Python deps and pushed them to https://quay.io/repository/pycontribs/centos?tab=tags – they are considerably bigger than original ones (4-5x) but using them does provide a performance boost for testing as we avoid reinstalling these when running.

For most tests using pre-build images is fine but it should be noted that some particular ones where we may want to test the pure images.

We currently do not have an automated process to update these base images, but producing one would not be a real issue. We already used the previous pycontribs images for more than an year without problems.

Centralize supported platforms?

As we have more than a dozen of repos, every time we want to update the images uses for testing we have to make changes to many repos. Last week showed us that is very easy to endup with a plentora of various images on each repository.

Molecule allows to centralize platforms used for testing at repository level but does not (yet) have a way to retrieve the list of platforms from another repository.

While it should not be hard to add a feature like this, I have some serious concerns: update of official platform list may break jobs on various repos as we cannot really ensure that we do cross-repo testing all the time.

Example of cases where platform list may need update: adding a strem9, or switching to other images or registry. Assume we have 100 molecule scenarios spread across our repos, what if changing platforms breaks 10% of them? – and we do not even know which one are these when we make the change.

  • @zbr maybe switch process may need to happen gradually, per repo

Best way to call molecule on CI

There are currently not less than 3 methods for calling molecule scenarios, each one with some pros and cons. Can we look which one can become the only way we use it?

(gchamoul) Note that our current way to run the molecule tests only work on the CI but not locally due to container volume constraints

  1. pytest-molecule plugin, which exposes molecule scenarios as pytests.
    • Advantage of being able to chain run multiple scenarios and to get a nicely formated html report with the output of each run.
    • Downside of this is that calling a single scenario from the command line requires taming pytest
    • Listing scenarios locally using pytest --collect-only, but they are mixed with unittests, requiring adding some extra params to limit them.
  2. The tripleo-ansible custom way
    • a bit similar with pytest-molecule approach but requires manual activation of a custom venv to run
    • no single command to run a scenario locally
    • no single command to list all scenarios
  3. tox-ansible (newest) - expose each molecule scenario as a virtual tox environment.
    • tox -a will list all scenarios
    • tox -e scenarioname to run a single scenario, can use comma to chain multiple ones
    • disadvantage: some limited html reporting, not as nice as pytest-html ones.
    • unable to produce/combine junit reports (not that we ever used that)
    • doesn't take in consideration project using molecule shared configuration at the project level (gchamoul - working on a patch)
    • as a pro, seems to be the path of least resistance for local testing, and having harmony / familiarity between both CI and local dev could be advantageous (dpeacock)

Open questions

  • …
Select a repo