---
# System prepended metadata

title: 20221006_small-tasks-for-rhcos-pipeline

---

# 20221006_small-tasks-for-rhcos-pipeline

## Namespaces in virt cluster

At the end, we'll update: https://gitlab.cee.redhat.com/coreos/team-operations/-/blob/main/INFRASTRUCTURE.md

Virt cluster: https://console-openshift-console.apps.ocp-virt.prod.psi.redhat.com/

- `rhcos`: used for new ART pipeline
- `rhcos-art`: used for old pipeline for x86
- `rhcos-devel`: used for new devel pipeline

## Steps for ART migration

- Azure secret might need some mutating
    - owner: mnguyen, ART
    - Mike converted our devel pipeline azure secret, ART secret might need the same done
- adapt ART processes for new bucket layout
    - owner? 
    - https://issues.redhat.com/browse/ART-5258
- set up pipecfg repo
    - owner: ravanelli
    - only set up 4.13 and 4.12 streams for now; e.g. comment out the others
- turn off 4.12 and move to new pipeline, repeat for all other streams until 4.9
    - owner: work with ART
    - for 4.11 and older we need to merge back in the rhcos-4.x-new branch into rhcos-4.x
- clean up leaked resources during testing:
    - aliyun: `us-east-1`
        - `m-0xif605c71en89f41i74`
        - `m-0xi6zmehimxs78s26si8`
        - `m-0xidvmnlrwtrqtjj93p0`
        - `m-0xifv8wlc16ulz2kpchs`
    - aws:
        - `us-east-1`
            - `ami-059a45e77659c1a7c`
            - `ami-045a8d5554f78886b`
            - `ami-0773fcf4dbb7da853`
        - `us-gov-west-1`
            - `ami-06b9b31519b571ecc`
            - `ami-095328d282430a044`
            - `ami-0f1068cb86d6bcf56`
            - `ami-0c8c677979b3c3e1b`
    - gcp: project: `rhcos-cloud`
        - images:
            - `rhcos-413-86-202211212146-0-gcp-x86-64`
            - `rhcos-413-86-202211221557-0-gcp-x86-64`
            - `rhcos-411-86-202211241840-0-gcp-x86-64`
            - `rhcos-410-84-202211251255-0-gcp-x86-64`
        - objects:
            - `gs://rhcos/rhcos/rhcos-413-86-202211212146-0-gcp-x86-64.tar.gz`
            - `gs://rhcos/rhcos/rhcos-413-86-202211221557-0-gcp-x86-64.tar.gz`
            - `gs://rhcos/rhcos/rhcos-411-86-202211241840-0-gcp-x86-64.tar.gz`
            - `gs://rhcos/rhcos/rhcos-410-84-202211251255-0-gcp-x86-64.tar.gz`

### Questions for ART

- MAYBE DONE - triggering jobs
    - from Slack thread: ART wants a service account to be able to trigger builds automatically?
    - https://issues.redhat.com/browse/COS-1906
    - jlebon has potential solution in: https://github.com/coreos/fedora-coreos-pipeline/pull/769
- what's the best way to turn off streams in old pipeline?
    - answer: ART has their own way to do this
- bucket layout adaption work dependency/ordering?
    - answer: no hard dependency; should be able to deploy 4.13, 4.12 before it's fully supported
- release browser adaptation for new bucket layout
    - owner: jmarrero
    - figure out what to do for the internal release browser 
    - https://issues.redhat.com/browse/COS-1908

## Now

- backport what is necessary to older cosa
    - owners: jlebon, dustymabe
    - DONE for 4.9 through 4.12
        - still need to do 4.8 if that's desired
- testing output of new pipeline
    - owners: travier, mnguyen
    - need to work with ART to figure out how we can test what is output by the new pipeline

## Later

- drop `oscontainer-push-old-registry-secret`
    - it was determined we don't need it; steps:
        - update `rchos-devel` pipeline `oscontainer-push-registry-secret`
        - delete `rhcos-devel` pipeline `oscontainer-push-old-registry-secret`
        - delete old registry secret entries from bitwarden
        - delete supporting code from the pipeline
- fold the cosa `-new` branches back into the canonical branches
- get rid of RHCOS hacks in the pipeline: https://github.com/coreos/fedora-coreos-pipeline/blob/6a36ceb54703b29f5c729e89c2f36cdd8b67b1fc/docs/config.yaml#L31-L43
    - investigate plume call in release job for RHCOS
        - owners: jlebon
    - fix upgrade tests to work with RHCOS
    - fold compression setting back into image.yaml
        - owners: jlebon
        - https://issues.redhat.com/browse/COS-1905
    - fix UEFI tests on old RHCOS
        - https://github.com/coreos/fedora-coreos-pipeline/commit/fab14eb5f4dd183d954a041db3ac5e066808fdc6
- remember to have ART update their azure secret to token based so they only need one file.

- one thing we do in the FCOS pipeline is pass the SRC_CONFIG_COMMIT along to the forked jobs (multi-arch and kola tests). However, that doesn't convey "yumrepos" git information.
    - We can use the branch tip for now and fix that later
- `kola-testiso-denylist.yaml`
    - owners: renata
- don't create jobs that are FCOS-specific
- figure out public (behind VPN) access for Jenkins interface

## DONE

- DONE configurable s3 bucket location
- DONE nuke `official` variable from the upstream pipeline code
- DONE ensure that just building OSTree and pushing oscontainer works
    - DWM: I think this works now
    - JL: I concur
- DONE basic hotfix support
    - OVERRIDE_PIPECFG_URL, OVERRIDE_PIPECFG_REF parameters for build, build-arch jobs
    - add support for `hotfix.disable_secureboot_tests_hack`
    - document hotfix process for ART
    - safety check for not clobbering canonical artifacts when running hotfix runs
    - add support for a `hotfix.name` to namespace pushed things (container images, S3 bucket path)?
    - JL: https://github.com/coreos/fedora-coreos-pipeline/pull/740
- DONE investigate power kola failure
    - owners: mike, ~~michael~~, Gursewak
    - https://jenkins-rhcos.apps.ocp-virt.prod.psi.redhat.com/job/build-arch/9/
    - https://issues.redhat.com/browse/COS-1817
- DONE set the limit for memory patch
    - owners: dustymabe
    - PR: https://github.com/coreos/coreos-ci-lib/pull/116
- DONE set the limit for memory patch
    - owners: dustymabe
    - PR: https://github.com/coreos/coreos-ci-lib/pull/116
- DONE add and use `skip_artifacts` as opposed to `additional_artifacts`
    - owner: renata
    - https://coreos.slack.com/archives/C03MX36D7C4/p1665089688549459
    - PR: https://github.com/coreos/fedora-coreos-pipeline/pull/685
- DONE - possibly update the azure image upload secret for rhcos-devel
    - owner: mike
    - it uses a different format than what the FCOS pipeline supports
    - https://issues.redhat.com/browse/COS-1818
    - dustymabe: new secret updated, now need to try it in the pipeline
- DONE - add support for uploading azure artifacts in FCOS pipeline
    - owner: dustymabe
    - PR: https://github.com/coreos/fedora-coreos-pipeline/pull/699
- DONE - support for uploading powervs artifacts in FCOS pipeline
    - owner: dustymabe
    - PR: https://github.com/coreos/fedora-coreos-pipeline/pull/699
- DONE - figure out what to do in release job for fcos vs rhcos different handling of GCP
    - owners: dustymabe, jlebon
    - Fixed in https://github.com/coreos/fedora-coreos-pipeline/commit/f2d2628ba8ea0e96819b312aa800a8adba842203
- DONE -  Monitor PR for oscontainer (old-style)
    - owners: dustymabe, jmarerro
    - https://github.com/coreos/coreos-assembler/pull/3111 was merged!
- DONE pushing manifest listed container images to `registry.ci.openshift.org/rhcos-devel/machine-os-content` doesn't work
    - owner: jlebon
    - https://coreos.slack.com/archives/C03MX36D7C4/p1665110385638539?thread_ts=1665091862.119999&cid=C03MX36D7C4
    - https://github.com/openshift/release/commit/1725585
    - Colin and Jonathan discussed this in https://coreos.slack.com/archives/C03MX36D7C4/p1665110484881309?thread_ts=1665091862.119999&cid=C03MX36D7C4
    - PRs:
        - https://github.com/coreos/coreos-assembler/pull/3129
        - https://github.com/coreos/coreos-assembler/pull/3130
        - https://github.com/coreos/fedora-coreos-pipeline/pull/698
- DONE AWS China/GovCloud support
    - owners: dustymabe
    - For china we don't upload there but tell users to do it themselves:
        - https://docs.openshift.com/container-platform/4.11/installing/installing_aws/installing-aws-china.html#installation-aws-regions-with-no-ami_installing-aws-china-region
    - PR: https://github.com/coreos/fedora-coreos-pipeline/pull/716
- DONE cosa image templating
    - need top-level knob with stream-level override (for 4.12)
    - get rid of all cosa imagestreams for now
    - owners: jlebon
- DONE  - file locking for build extensions container
    - owners: jmarrero
    - PR: https://github.com/coreos/coreos-assembler/pull/3152
- DONE - get a powervs-devel secret and store it in bitwarden
    -  (https://issues.redhat.com/browse/COS-1811)
- DONE - add generic replication for powervs and aliyun (like we have for AWS)
    - owners: renata
    - PR: https://github.com/coreos/fedora-coreos-pipeline/pull/746
- DONE - beaker reservation time and rhcos group access
    - owner: mnguyen, dustymabe
    - working with dennis gilmore on getting these applied
    - DWM: we should have this all worked out now
- DONE - fold hacks into `main` branch of f-c-pipeline
    - DONE: PR: https://github.com/coreos/fedora-coreos-pipeline/pull/753
- DONE - bring up devel pipeline in new `rhcos-devel` namespace
    - DONE: dustymabe did this on 11/15
- DONE - add `cloud-replicate` job so it can be run separately in the RHCOS pipeline
    - owners: dustymabe
    - DONE: PR: https://github.com/coreos/fedora-coreos-pipeline/pull/758
- DONE - aliyun `rchos-devel` bucket getting deleted periodically in aliyun devel account
    - This causes aliyun cloud image upload to fail every few days or so
    - `aliyun oss mb oss://rhcos-devel` # to workaround
    - Tickets:
        - https://issues.redhat.com/browse/DPP-11564
        - https://issues.redhat.com/browse/COS-1881
- DONE - empty `rhcos` namespace
    - owner: dustymabe
- DONE - get secrets into a bitwarden vault into copy/paste form
    - owner: work with ART
- DONE - populate ART namespace with secrets
    - owner: work with ART
- DONE - turn *off* 4.13 stream in old pipeline
    - owner: work with ART
- DONE - bring up pipeline with `./deploy --pipecfg` set to pipecfg repo
    - owner: work with ART
- DONE - run 4.13 stream builds, fix issues, repeat until satisfied
    - owner: work with ART
- DONE - support buildupload private ACL
    - owners: jlebon
    - PR: https://github.com/coreos/fedora-coreos-pipeline/pull/766

### Hotfix sugar for ART strawman

For simple case, add new sugar:

```yaml
hotfix:
    cosa_overrides:
        rpms:
            - RPM URL
            - Koji/Brew URL
        yumrepos:
            - Brew repofile URL
            - https://gitlab.cee.redhat.com/coreos/redhat-coreos.git@hotfix-JIRA-1234:my.repo
```

For complex case, two general approaches:
1. branch redhat-coreos repo and branch internal mirror of openshift/os repo
2. branch internal mirror of openshift/os repo and import everything into it








https://gitlab.cee.redhat.com/coreos/redhat-coreos.git@hotfix-JIRA-1234

- `git checkout <starting point>`
- `git checkout -b hotfix-JIRA-1234`
- two ways to modify `config.yaml` for packages
    - in the simple case of just a few RPMs different:
        hotfix:
          cosa_overrides:
            rpms:
              - RPM URL
              - Koji/Brew URL
            yumrepos:
              - Brew repofile URL
              - https://gitlab.cee.redhat.com/coreos/redhat-coreos.git@hotfix-JIRA-1234:my.repo
    - if you want more control, you can e.g. add a yumrepo definition to the repo and create a `manifest.yaml`:
        source_config:
          yumrepos: $pipecfg-url@hotfix-JIRA-1234
          url: $pipecfg-url
          ref: hotfix-JIRA-1234
        hotfix: 
          manifest_override: $pipecfg-url@hotfix-JIRA-1234
          

UX:
- need one new yumrepo file new.repo
- add new.repo to this branch
- specify hotfix.cosa_overrides.yumrepos
    - https://gitlab.cee.redhat.com/coreos/redhat-coreos.git@hotfix-JIRA-1234:my.repo

Implementation:
- pipeline downloads URLs into overrides/yum.repos.d
- cosa knows to add repo ID into manifest override and generate lockfile from it
        
all located in redhat-coreos, or whatever internal repo we choose:
- config.yaml
- yum repos
- contentsets.yaml

overrides/rootfs
overrides/rpm
overrides/yum.repos.d
  - my.repo
  - other.repo
