# How OKD gets built

### Source code

All software starts at the source code. In OKD's case there are multiple repos under the https://github.com/openshift organization.

Lets pick one for example - https://github.com/openshift/origin-branding. Its a simple image, consisting of:

* `manifests/0000_10_origin-branding_configmap.yaml` - a configmap telling console to use OKD branding and linking to OKD documentation
* `Dockerfile` - instructions for CI how to build that image
Note `io.openshift.release.operator=true` label - that tells OKD's version operator to manage this manifest and reconcile it if changes happen.

All changes are built, tested and promoted by our CI, which is [Prow](https://github.com/kubernetes/test-infra). Note the green checkmark near the commit banner, it has a post-submit job:
https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/branch-ci-openshift-origin-branding-master-images/1369684984721313792

When the commit lands in the branch, CI fetches the latest release configured for this branch (4.8 atm):

2021/03/10 16:22:10 Tagging openshift/release:golang-1.10 into pipeline:root
2021/03/10 16:22:10 Tagging origin/4.8:base into pipeline:base
2021/03/10 16:22:10 Tagged shared images from origin/4.8:${component}...

Then it builds the image:

2021/03/10 16:22:10 Building src
2021/03/10 16:23:20 Build src succeeded after 1m8s
2021/03/10 16:23:20 Building branding
2021/03/10 16:24:31 Build branding succeeded after 1m9s

We don't run any tests for this image, so the images get tagged:

2021/03/10 16:24:31 Tagging branding into stable

and get promoted to official 4.8 nightlies imagestream:

2021/03/10 16:24:31 Promoting tags to origin/4.8:${component}: branding

Some components, like `okd-machine-os`, also require tests to pass in order to merge the PR:
https://github.com/openshift/okd-machine-os/pull/102
* `e2e-aws`
* `e2e-aws-upgrade`
* `e2e-gcp`
* `e2e-vsphere`

The first two tests are required to pass, so PRs which break AWS and/or upgrades won't be accepted.

### Building a release payload

During promotion CI keeps promoted images in the `release` imagestream on the CI server. When it detects that a new image has been promoted, CI makes a snapshot of this images (copying it to a new `4.x-<date>` imagestream) and creates a new release running `oc adm release new`.
This commands ensures that:
* all images referred to in the operators exist
* image references are rewritten so that the release would be self-sufficient (no external images required to run it)
* image metadata is parsed to set git information so that every component can be tracked back to a particular commit:

$ oc adm release info quay.io/openshift/okd:4.7.0-0.okd-2021-03-07-090821 --commit-urls
Name:           4.7.0-0.okd-2021-03-07-090821
Digest:         sha256:2597be07a16ae6ef8df7752c039190bf5b238f81f88aca129d0c2d542239b537
Created:        2021-03-07T09:12:53Z

...

Images:
  NAME                                           URL 
  aws-ebs-csi-driver                             https://github.com/openshift/aws-ebs-csi-driver/commit/f6a71bf783f3b7429a8c2fbbc6f3d586e6a7eb60
  aws-ebs-csi-driver-operator                    https://github.com/openshift/aws-ebs-csi-driver-operator/commit/97f73eb7de4b474497169771239dcd81990d3f2e
  aws-machine-controllers                        https://github.com/openshift/cluster-api-provider-aws/commit/5ffc5f422a80169b342c19f50b22a0c4883cc0e3

Users don't necessarily have to wait for their fix to land in nightlies, as they could create a new
release payload themselves. Refer to https://github.com/openshift/okd/blob/master/CONTRIBUTING.md for more information.

### Sharing images with OCP
OKD project is heavily relying on OCP images, tests and infrastructure.

In OKD 3.x days OCP images were based on RHEL images - and due to licensing it could not be used in community distributions like OKD.
So openshift 3.x commits were built using centos on CI, promoted to OKD - and then rebuilt using RHEL images internally and released in OCP.
This workflow led to a duplication of work - QE had to verify two slightly different distributions in order to ensure developer CI is valid and images we release to customers are functional.

The solution to this problem is [Universal Base Image](https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image) - a set of RHEL images, freely distributed and supported by Red Hat. These images are used as base for CI images and official OCP images. Now CI builds OCP image based on UBI, tests it in OCP, promotes it to the OCP release image streams and if OCP tests pass it mirrors them to OKD:

$ OKD_IMAGE=$(oc adm release info --image-for=coredns registry.ci.openshift.org/origin/release:4.7)
$ echo $OKD_IMAGE
registry.ci.openshift.org/origin/4.7-2021-03-20-070531@sha256:a51b153b85c92c76017eee22582630b3ba6585ee839cf9b23a18f8598c326254
$ OCP_IMAGE=$(oc adm release info -a auth.json --image-for=coredns registry.ci.openshift.org/ocp/release:4.7.0-0.ci-2021-03-20-094925)
$ echo $OCP_IMAGE
registry.ci.openshift.org/ocp/4.7-2021-03-20-094925@sha256:09cdac06b17046aa92511c6095bb814bcd8effe6dc5bab5523b1e65db6533fd6
$ diff <(oc image info -a auth.json ${OCP_IMAGE}) <(oc image info ${OKD_IMAGE})
1c1
< Name:        registry.ci.openshift.org/ocp/4.7-2021-03-20-094925@sha256:09cdac06b17046aa92511c6095bb814bcd8effe6dc5bab5523b1e65db6533fd6
---
> Name:        registry.ci.openshift.org/origin/4.7-2021-03-20-070531@sha256:a51b153b85c92c76017eee22582630b3ba6585ee839cf9b23a18f8598c326254

Once the image is promoted, the release controller updates OKD `release` imagestream and a new OKD nightly is created.

Note that OCP ci images have their own set of tests, and if these don't pass images are not copied to OKD - if the change is not suitable for OCP it's not suitable for OKD either.

### OKD release controller

The release controller also runs OKD-specific tests to ensure that end-to-end test on a prepared image passes.

The https://amd64.origin.releases.ci.openshift.org/#4.7.0-0.okd page lists all OKD 4.7 nightlies, shows nightly acceptance status (Ready/Rejected), failed tests and the upgrade graph. Each release page also can show a changelog between releases, CI job testing status and instructions how to extract installer pinned to this release.

When we're ready to make a new stable release we follow this procedure - https://hackmd.io/LIK_IOnER-6kuIQWAVn5dQ?both, which can be summarized as:

* Mirror content from CI registry to quay.io
* Tag nightly into `stable-4` channel
This would make release controller update the upgrade graph and run upgrade tests from previous stable to a proposed nightly.
If the test passes user cluster would see the upgrade button in the console.

* We also publish the release to github, so installer is extracted from the payload, signed and uploaded to github

OKD, unlike OCP, is a rolling release distribution, so it has a single `stable-4` stream. Nightlies are produced for every OCP stream (`4.8.0-0.okd`, `4.7.0-0.okd` etc), but we can't keep up with OCP tempo and support different stable streams. Once 4.7 nightlies have been promoted to it a new stable 4.6 release could not be promoted to stable - this will make a 4.6 -> 4.7 -> 4.6 upgrade graph, which is invalid.

### Issue tracking
We use Github issues tracker at https://github.com/openshift/okd/issues to handle OKD-specific issues.

As OKD is relying on a lot of OCP images, most issues we hit with OKD are affecting OCP too. For instance, OKD is using unmodified console image, so any UI problem would be reproducible on OCP too. Some images are however OKD-specific, for instance `machine-os-content` is based on Fedora CoreOS (while OCP is based on RHEL CoreOS).

So, if you're sure the issue you're hitting is happening in OCP too please use OCP bugzilla to track it - this would get direct developer attention and speed up the fixing process.

Most issues require a lot of additional information to figure out the details, so OKD includes two tools to collect necessary information:
* log bundles produced by installer
* `oc adm must-gather`

Linking those log bundle files to your report will help us triage the issue quicker.

### Sibling distribution for community

Hopefully this release overview guide has been useful to de-mystify some details of how OKD gets built and tested.
To re-iterate:

* OKD is a community distribution, we make decisions on Workgroup calls in the open
* As a community distribution every part is open to be modified by community members
* OKD 4 is not OCP 4 upstream/downstream/midstream/otherstream. It reuses multiple OCP images, adding OKD-specific one.
In order to describe this situation we coined the term "sibling distributions".
* OKD release process is heavily automated and mostly happens in a hands-off manner.

Cheers :)