# Test Manifests and Image Info

## Summary

Testing and CI in osbuild-composer does *too much work*. Test cases and image-info take too much time to create and keep up-to-date when changing image definitions. We need to separate the concerns and responsibilities of osbuild-composer and osbuild.

We've talked a few times about moving image-info outside of osbuild-composer to a repository that can be maintained separately. The idea for this separation (and the CI changes in the end) is that osbuild-composer testing should only be concerned with manifest creation, image uploads, API calls, job queueing, etc., but image _creation_ should be the the concern of osbuild's testing.

The changes are based on the following assumptions:
- osbuild-composer shouldn't "care" if the same manifest produces two different images with different image-info. This should be the concern of osbuild.
- osbuild-composer CI shouldn't be building every image type every time it runs if the manifest for that type hasn't changed.
- Test cases with image-info are useful for catching changes in images, but the current way we create and use them is cumbersome and makes it very likely that we ignore changes.
- **A better contribution and CI experience is possible.**

## Steps

### 1. Remove image-info from osbuild-composer

- The test cases in the osbuild-composer repository ([test/data/manifests](https://github.com/osbuild/osbuild-composer/tree/main/test/data/manifests)) should only contain the manifest and the list of packages (`rpmmd`). No `image-info`.
- `image-info` files are moved to a separate repository (or `osbuild/osbuild`). The new repository should contain the ID of an osbuild-composer commit that they correspond to and scripts for building all images and generating the image info for that commit.
- Test case generator scripts in osbuild-composer should not require building images and instead just generate the manifest and package list (this can be done currently with the `--keep-image-info` flag).
- Unit tests and GitLab CI tests in osbuild-composer remain the same.

At the end of this step we will have the same test coverage that we do now, but image-info updates wont slow down osbuild-composer PRs. The plain manifest generation for the test cases will take a trivial amount of time (relative to image-info) and will be much easier to run by external contributors and without special hardware.

### 2. Test the image-info

- Set up CI for the new image-info repository to build each image using the manifests in osbuild-composer and compare the image-info.
- Some hueristics will be needed to ignore some changed values, treat others as warnings, and produce errors when necessary.
- Periodically update the image-info by updating the osbuild-composer commit ID and regenerating all test cases.

At the end of this step we will have a higher test coverage than we do now. Image info changes will be caught and reported automatically.

### 3. Automate the boring stuff

- Set up automation that periodically and on-demand can update the image-info database.
- Tie it to osbuild-composer PRs so that the automation can run on a PR (when needed) and produce a report of changes.
- Tie it to osbuild-composer PRs to update the test cases in the PR itself on-demand (append manifest changes to the open PR).

### 4. Limit image builds in osbuild-composer CI

*This is a much bigger change that will require more thought and planning*

- Build images in osbuild CI based on the manifests in osbuild-composer (or the image-info database).
- Store the artifacts in an accessible location (s3 bucket? RH infra?).
- In osbuild-composer CI, **do not** build images that are already available in storage. Identify by manifest ID and download for boot testing, upload testing, etc.
- Manifests that have been changed in the PR (or not yet built in storage) should still be rebuilt and tested as they are now.

This will **significantly** speed up osbuild-composer CI. Changes that don't affect image defintions (e.g., API changes) should require no image builds at all.