# 20221006_small-tasks-for-rhcos-pipeline

## Namespaces in virt cluster

At the end, we'll update: https://gitlab.cee.redhat.com/coreos/team-operations/-/blob/main/INFRASTRUCTURE.md

Virt cluster: https://console-openshift-console.apps.ocp-virt.prod.psi.redhat.com/

- `rhcos`: used for new ART pipeline
- `rhcos-art`: used for old pipeline for x86
- `rhcos-devel`: used for new devel pipeline

## Steps for ART migration

- Azure secret might need some mutating
- owner: mnguyen, ART
- Mike converted our devel pipeline azure secret, ART secret might need the same done
- adapt ART processes for new bucket layout
- owner?
- https://issues.redhat.com/browse/ART-5258
- set up pipecfg repo
- owner: ravanelli
- only set up 4.13 and 4.12 streams for now; e.g. comment out the others
- turn off 4.12 and move to new pipeline, repeat for all other streams until 4.9
- owner: work with ART
- for 4.11 and older we need to merge back in the rhcos-4.x-new branch into rhcos-4.x
- clean up leaked resources during testing:
- aliyun: `us-east-1`
- `m-0xif605c71en89f41i74`
- `m-0xi6zmehimxs78s26si8`
- `m-0xidvmnlrwtrqtjj93p0`
- `m-0xifv8wlc16ulz2kpchs`
- aws:
- `us-east-1`
- `ami-059a45e77659c1a7c`
- `ami-045a8d5554f78886b`
- `ami-0773fcf4dbb7da853`
- `us-gov-west-1`
- `ami-06b9b31519b571ecc`
- `ami-095328d282430a044`
- `ami-0f1068cb86d6bcf56`
- `ami-0c8c677979b3c3e1b`
- gcp: project: `rhcos-cloud`
- images:
- `rhcos-413-86-202211212146-0-gcp-x86-64`
- `rhcos-413-86-202211221557-0-gcp-x86-64`
- `rhcos-411-86-202211241840-0-gcp-x86-64`
- `rhcos-410-84-202211251255-0-gcp-x86-64`
- objects:
- `gs://rhcos/rhcos/rhcos-413-86-202211212146-0-gcp-x86-64.tar.gz`
- `gs://rhcos/rhcos/rhcos-413-86-202211221557-0-gcp-x86-64.tar.gz`
- `gs://rhcos/rhcos/rhcos-411-86-202211241840-0-gcp-x86-64.tar.gz`
- `gs://rhcos/rhcos/rhcos-410-84-202211251255-0-gcp-x86-64.tar.gz`

### Questions for ART

- MAYBE DONE - triggering jobs
- from Slack thread: ART wants a service account to be able to trigger builds automatically?
- https://issues.redhat.com/browse/COS-1906
- jlebon has potential solution in: https://github.com/coreos/fedora-coreos-pipeline/pull/769
- what's the best way to turn off streams in old pipeline?
- answer: ART has their own way to do this
- bucket layout adaption work dependency/ordering?
- answer: no hard dependency; should be able to deploy 4.13, 4.12 before it's fully supported
- release browser adaptation for new bucket layout
- owner: jmarrero
- figure out what to do for the internal release browser
- https://issues.redhat.com/browse/COS-1908

## Now

- backport what is necessary to older cosa
- owners: jlebon, dustymabe
- DONE for 4.9 through 4.12
- still need to do 4.8 if that's desired
- testing output of new pipeline
- owners: travier, mnguyen
- need to work with ART to figure out how we can test what is output by the new pipeline

## Later

- drop `oscontainer-push-old-registry-secret`
- it was determined we don't need it; steps:
- update `rchos-devel` pipeline `oscontainer-push-registry-secret`
- delete `rhcos-devel` pipeline `oscontainer-push-old-registry-secret`
- delete old registry secret entries from bitwarden
- delete supporting code from the pipeline
- fold the cosa `-new` branches back into the canonical branches
- get rid of RHCOS hacks in the pipeline: https://github.com/coreos/fedora-coreos-pipeline/blob/6a36ceb54703b29f5c729e89c2f36cdd8b67b1fc/docs/config.yaml#L31-L43
- investigate plume call in release job for RHCOS
- owners: jlebon
- fix upgrade tests to work with RHCOS
- fold compression setting back into image.yaml
- owners: jlebon
- https://issues.redhat.com/browse/COS-1905
- fix UEFI tests on old RHCOS
- https://github.com/coreos/fedora-coreos-pipeline/commit/fab14eb5f4dd183d954a041db3ac5e066808fdc6
- remember to have ART update their azure secret to token based so they only need one file.

- one thing we do in the FCOS pipeline is pass the SRC_CONFIG_COMMIT along to the forked jobs (multi-arch and kola tests). However, that doesn't convey "yumrepos" git information.
- We can use the branch tip for now and fix that later
- `kola-testiso-denylist.yaml`
- owners: renata
- don't create jobs that are FCOS-specific
- figure out public (behind VPN) access for Jenkins interface

## DONE

- DONE configurable s3 bucket location
- DONE nuke `official` variable from the upstream pipeline code
- DONE ensure that just building OSTree and pushing oscontainer works
- DWM: I think this works now
- JL: I concur
- DONE basic hotfix support
- OVERRIDE_PIPECFG_URL, OVERRIDE_PIPECFG_REF parameters for build, build-arch jobs
- add support for `hotfix.disable_secureboot_tests_hack`
- document hotfix process for ART
- safety check for not clobbering canonical artifacts when running hotfix runs
- add support for a `hotfix.name` to namespace pushed things (container images, S3 bucket path)?
- JL: https://github.com/coreos/fedora-coreos-pipeline/pull/740
- DONE investigate power kola failure
- owners: mike, ~~michael~~, Gursewak
- https://jenkins-rhcos.apps.ocp-virt.prod.psi.redhat.com/job/build-arch/9/
- https://issues.redhat.com/browse/COS-1817
- DONE set the limit for memory patch
- owners: dustymabe
- PR: https://github.com/coreos/coreos-ci-lib/pull/116
- DONE set the limit for memory patch
- owners: dustymabe
- PR: https://github.com/coreos/coreos-ci-lib/pull/116
- DONE add and use `skip_artifacts` as opposed to `additional_artifacts`
- owner: renata
- https://coreos.slack.com/archives/C03MX36D7C4/p1665089688549459
- PR: https://github.com/coreos/fedora-coreos-pipeline/pull/685
- DONE - possibly update the azure image upload secret for rhcos-devel
- owner: mike
- it uses a different format than what the FCOS pipeline supports
- https://issues.redhat.com/browse/COS-1818
- dustymabe: new secret updated, now need to try it in the pipeline
- DONE - add support for uploading azure artifacts in FCOS pipeline
- owner: dustymabe
- PR: https://github.com/coreos/fedora-coreos-pipeline/pull/699
- DONE - support for uploading powervs artifacts in FCOS pipeline
- owner: dustymabe
- PR: https://github.com/coreos/fedora-coreos-pipeline/pull/699
- DONE - figure out what to do in release job for fcos vs rhcos different handling of GCP
- owners: dustymabe, jlebon
- Fixed in https://github.com/coreos/fedora-coreos-pipeline/commit/f2d2628ba8ea0e96819b312aa800a8adba842203
- DONE - Monitor PR for oscontainer (old-style)
- owners: dustymabe, jmarerro
- https://github.com/coreos/coreos-assembler/pull/3111 was merged!
- DONE pushing manifest listed container images to `registry.ci.openshift.org/rhcos-devel/machine-os-content` doesn't work
- owner: jlebon
- https://coreos.slack.com/archives/C03MX36D7C4/p1665110385638539?thread_ts=1665091862.119999&cid=C03MX36D7C4
- https://github.com/openshift/release/commit/1725585
- Colin and Jonathan discussed this in https://coreos.slack.com/archives/C03MX36D7C4/p1665110484881309?thread_ts=1665091862.119999&cid=C03MX36D7C4
- PRs:
- https://github.com/coreos/coreos-assembler/pull/3129
- https://github.com/coreos/coreos-assembler/pull/3130
- https://github.com/coreos/fedora-coreos-pipeline/pull/698
- DONE AWS China/GovCloud support
- owners: dustymabe
- For china we don't upload there but tell users to do it themselves:
- https://docs.openshift.com/container-platform/4.11/installing/installing_aws/installing-aws-china.html#installation-aws-regions-with-no-ami_installing-aws-china-region
- PR: https://github.com/coreos/fedora-coreos-pipeline/pull/716
- DONE cosa image templating
- need top-level knob with stream-level override (for 4.12)
- get rid of all cosa imagestreams for now
- owners: jlebon
- DONE - file locking for build extensions container
- owners: jmarrero
- PR: https://github.com/coreos/coreos-assembler/pull/3152
- DONE - get a powervs-devel secret and store it in bitwarden
- (https://issues.redhat.com/browse/COS-1811)
- DONE - add generic replication for powervs and aliyun (like we have for AWS)
- owners: renata
- PR: https://github.com/coreos/fedora-coreos-pipeline/pull/746
- DONE - beaker reservation time and rhcos group access
- owner: mnguyen, dustymabe
- working with dennis gilmore on getting these applied
- DWM: we should have this all worked out now
- DONE - fold hacks into `main` branch of f-c-pipeline
- DONE: PR: https://github.com/coreos/fedora-coreos-pipeline/pull/753
- DONE - bring up devel pipeline in new `rhcos-devel` namespace
- DONE: dustymabe did this on 11/15
- DONE - add `cloud-replicate` job so it can be run separately in the RHCOS pipeline
- owners: dustymabe
- DONE: PR: https://github.com/coreos/fedora-coreos-pipeline/pull/758
- DONE - aliyun `rchos-devel` bucket getting deleted periodically in aliyun devel account
- This causes aliyun cloud image upload to fail every few days or so
- `aliyun oss mb oss://rhcos-devel` # to workaround
- Tickets:
- https://issues.redhat.com/browse/DPP-11564
- https://issues.redhat.com/browse/COS-1881
- DONE - empty `rhcos` namespace
- owner: dustymabe
- DONE - get secrets into a bitwarden vault into copy/paste form
- owner: work with ART
- DONE - populate ART namespace with secrets
- owner: work with ART
- DONE - turn *off* 4.13 stream in old pipeline
- owner: work with ART
- DONE - bring up pipeline with `./deploy --pipecfg` set to pipecfg repo
- owner: work with ART
- DONE - run 4.13 stream builds, fix issues, repeat until satisfied
- owner: work with ART
- DONE - support buildupload private ACL
- owners: jlebon
- PR: https://github.com/coreos/fedora-coreos-pipeline/pull/766

### Hotfix sugar for ART strawman

For simple case, add new sugar:

hotfix:
    cosa_overrides:
        rpms:
            - RPM URL
            - Koji/Brew URL
        yumrepos:
            - Brew repofile URL
            - https://gitlab.cee.redhat.com/coreos/redhat-coreos.git@hotfix-JIRA-1234:my.repo

For complex case, two general approaches:
1. branch redhat-coreos repo and branch internal mirror of openshift/os repo
2. branch internal mirror of openshift/os repo and import everything into it

https://gitlab.cee.redhat.com/coreos/redhat-coreos.git@hotfix-JIRA-1234

- `git checkout <starting point>`
- `git checkout -b hotfix-JIRA-1234`
- two ways to modify `config.yaml` for packages
- in the simple case of just a few RPMs different:
hotfix:
cosa_overrides:
rpms:
- RPM URL
- Koji/Brew URL
yumrepos:
- Brew repofile URL
- https://gitlab.cee.redhat.com/coreos/redhat-coreos.git@hotfix-JIRA-1234:my.repo
- if you want more control, you can e.g. add a yumrepo definition to the repo and create a `manifest.yaml`:
source_config:
yumrepos: $pipecfg-url@hotfix-JIRA-1234
url: $pipecfg-url
ref: hotfix-JIRA-1234
hotfix:
manifest_override: $pipecfg-url@hotfix-JIRA-1234

UX:
- need one new yumrepo file new.repo
- add new.repo to this branch
- specify hotfix.cosa_overrides.yumrepos
- https://gitlab.cee.redhat.com/coreos/redhat-coreos.git@hotfix-JIRA-1234:my.repo

Implementation:
- pipeline downloads URLs into overrides/yum.repos.d
- cosa knows to add repo ID into manifest override and generate lockfile from it

all located in redhat-coreos, or whatever internal repo we choose:
- config.yaml
- yum repos
- contentsets.yaml

overrides/rootfs
overrides/rpm
overrides/yum.repos.d
- my.repo
- other.repo