# Image Tests

This sets out a plan to simplify contributions to `osbuild-composer` in general, and image definitions in particular.

Context: https://coreos.slack.com/archives/C0235DZB0DT/p1644845793594899

## Aims

These are the primary aims of our testing strategy.

### Manifest Generation is Stable

Changes to image definitions do not change the manifets generation unintentionally. This is achieved by having a pinned set of manifests generated from a pinned set of inputs + rpmrepo snapshots committed to `osbuild-composer`. On every CI run these are regenerated and verified to be unchanged. If someone intentionally changes the produced manifests, they would need to regenerate the pinned manifests as part of their patch.

#### Proposed Improvement

Regenerate all manifests whenever the rpmrepo snapshots are updated by schutzbot so these stay up-to-date and in sync. Ensure that the set of manifests thus generated is complete for all distros / architectures / image types.

### Image Info is Stable

Changes to `osbuild` do not change the image-info produced by a given manifest. This was achieved through pinning image-info together with the above mentioned manifests in `osbuild-composer`, but this proposal moves away from that.

#### Proposed Improvement

Keep pinned manifests in `osbuild` together with their associated image-info. On every CI run, regenerate the image info and verify it is unchanged. At regular intervals have schutzbot sync the pinned manifests from `osbuild-composer` to `osbuild` and regenerate the image-info. Long-term, extend CI to run these tests on more architectures than we currently do.

Crucially, `osbuild-composer` would no longer care about image-info at all, as it is the wrong layer of the stack, it only cares that the images boot and behave corectly, not about the precise contents.

### No Manual Image Info Regeneration

Currently, when either rpmrepo snapshots should be updated, or manifest generation changes, or image info itself changes, a developer would have to manually regenerate all the image info, which would mean provisioning machines and wait for many hours.

This would lower the bar to entry to composer contribution in general and image definition contribution in particular.

#### Proposed Improvement

The purpose of image info is to verify that osbuild does not change behavior. Keep that tool in osbuild itself, and let the bot perform all routine regeneration. The only exception is when image info itself is changed, in which case the devloper doing the change would need to regenerate all the image info currenly pinned in the repository (at least in the current model).

## Secondary Aims

Image info could and should be used as a development tool in addition to a CI tool. We should maintain this possibility and all the current workflows, but it should be optional, not something everyone have to understand. The example brought up was when manifest generation was changed in composer, to ensure that the resulting image info remained unchanged, despite the underlying manifests changing. This is a valid, though not common, usecase that we should make easy when necessary.

Secondly, regenerating image info on non-default architectures (the ones we don't have in CI) gives us a sanity check that these things have been checked to build at least once. I think this would be better serevd by making it possible / easy to run our integration tests manually against a provisioned machine, in a similar way to how image generation can be done.

## Plan

- copy image-info and related scripts to `osbuild`
- set up a bot to copy over all pinned manifests from `osbuild-composer` to `osbuild` and regenerate the image-info for these new manifests
- set up `osbuild` CI to verify that the image-info does not change between runs (similar to what is currently done in composer)
- drop image-info tests from composer (together with pinned data)
- drop image-info and related scripts from `osbuild-composer`
- hook up the manifest generation in `osbuild-composer` to a bot that recreates it on snapshot updates