# 20200723 FCOS-initiatives-releng-infra

# Multi-arch POC

https://github.com/coreos/fedora-coreos-tracker/issues/541

Need to get aarch64 node for testing.
- Preferrably decent amount of RAM with virt capabilities (/dev/kvm)

# More reliable package layering

https://github.com/coreos/fedora-coreos-tracker/issues/400
https://github.com/coreos/fedora-coreos-tracker/issues/401

- The problem:
- In a nutshell the problem is because packages in our base layer are locked and if the base layer is built from a different "package snapshot" than the latest updates repo then we get errors because rpm-ostree really doesn't want to replace base layer packages.

- Solutions:
- Using a smaller repo of curated packages that we bless for use with FCOS
- we respin this repo when we spin FCOS
- we maintain versions of it that match the versions we built different FCOS builds with
- Making more than the latest version of packages available for fedora
- POSITIVE: helps all of FCOS and Silverblue (and possibly IoT) users that use package layering
- could we keep the existing mirror set up, but transition older packages into an archive repo once they drop out of bodhi
- the archive repo could be hosted in S3 or something so we don't change our relationship with our existing mirrors
- embed information in the ostree at compose time about the yum repo that was used and peruse https://kojipkgs.fedoraproject.org/compose/updates/ to try to find a match with what was used for the base layer and use that for package layering instead

Email to teams:

Hello Silverblue and IoT teams. The FCOS team got together with Fedora releng last week to discuss the issue regarding package layering that periodically plagues us (https://github.com/coreos/fedora-coreos-tracker/issues/400). The solution we believe will help all OSTree based editions involves creating an archive repo where any package that has made it to the Fedora stable repositories can be accessed at a later time. In general, we think this should solve the problem because we should be able to install packages that won't require updating the base layer.

Goals:

- help solve the same problem for Fedora CoreOS, Fedora Silverblue, and Fedora IoT
- don't add to mirror network requirements
- i.e., store/host the content somewhere else. AWS is a candidate here.
- keep traditional systems behavior the same
- don't enable archive repo(s) by default on non-ostree distributions

Since it can take a long time to create repos for large package sets we may end up creating more than one repo that gets updated at different cadences. For example:

- One that gets updated weekly
- all packages obsoleted before X date (large package set)
- One that gets updated nightly
- all packages obsoleted after X date (small package set)

This is still a work in progress and the design may take a few turns as we work out the details and/or find new information.

Having this new repository will help us in Fedora CoreOS as we have a stable stream that lags behind Fedora stable repos. It should also help Fedora Silverblue when they move to a release cadence that doesn't match the bodhi updates repos. I'm not sure how much of a problem this currently is in Fedora IoT, but I imagine Fedora IoT has similar problems.

Thoughts?

Dusty Mabe

[fedora@ip-10-0-1-143 rpms]$ time ~/koji-download-ts.sh
Downloading new packages since 1597881868 til now (1597892897).
Downloading: 0ad-0.0.23b-15.fc32.x86_64.rpm
File 0ad-0.0.23b-15.fc32.x86_64.rpm already downloaded, skipping
Downloading: apt-doc-2.1.10-1.fc32.noarch.rpm
Downloading: apt-apidoc-2.1.10-1.fc32.noarch.rpm
Downloading: apt-utils-2.1.10-1.fc32.x86_64.rpm
Downloading: apt-libs-2.1.10-1.fc32.x86_64.rpm
Downloading: apt-devel-2.1.10-1.fc32.x86_64.rpm
Downloading: apt-2.1.10-1.fc32.x86_64.rpm
Downloading: python3-ara-1.4.3-1.fc32.noarch.rpm
Downloading: python3-ara-tests-1.4.3-1.fc32.noarch.rpm
Downloading: python3-ara-server-1.4.3-1.fc32.noarch.rpm
Downloading: ara-1.4.3-1.fc32.noarch.rpm
Downloading: ara-doc-1.4.3-1.fc32.noarch.rpm
Downloading: compat-golang-github-mholt-caddy-devel-1.0.5-1.fc32.noarch.rpm
Downloading: golang-github-caddyserver-caddy-devel-1.0.5-1.fc32.noarch.rpm
Downloading: caddy-1.0.5-1.fc32.x86_64.rpm
Downloading: cinnamon-devel-doc-4.6.7-1.fc32.noarch.rpm
Downloading: cinnamon-4.6.7-1.fc32.x86_64.rpm
Downloading: cinnamon-control-center-4.6.2-1.fc32.x86_64.rpm
Downloading: cinnamon-control-center-filesystem-4.6.2-1.fc32.x86_64.rpm
Downloading: cinnamon-control-center-devel-4.6.2-1.fc32.x86_64.rpm
Downloading: cinnamon-desktop-devel-4.6.4-1.fc32.x86_64.rpm
Downloading: cinnamon-desktop-4.6.4-1.fc32.x86_64.rpm
Downloading: cinnamon-session-4.6.2-1.fc32.x86_64.rpm
Downloading: copr-cli-1.89-1.fc32.noarch.rpm
Downloading: ddnet-data-14.3.2-1.fc32.noarch.rpm
Downloading: ddnet-14.3.2-1.fc32.x86_64.rpm
Downloading: ddnet-server-14.3.2-1.fc32.x86_64.rpm
Downloading: fedora-obsolete-packages-32-55.noarch.rpm
Downloading: fedora-repos-ostree-32-5.noarch.rpm
Downloading: fedora-repos-eln-32-5.noarch.rpm
Downloading: fedora-gpg-keys-32-5.noarch.rpm
Downloading: fedora-repos-rawhide-32-5.noarch.rpm
Downloading: fedora-repos-32-5.noarch.rpm
Downloading: GAPDoc-doc-1.6.4-4.fc32.noarch.rpm
Downloading: GAPDoc-latex-1.6.4-4.fc32.noarch.rpm
Downloading: GAPDoc-1.6.4-4.fc32.noarch.rpm
Downloading: gap-pkg-autodoc-doc-2020.08.11-1.fc32.noarch.rpm
Downloading: gap-pkg-autodoc-2020.08.11-1.fc32.noarch.rpm
Downloading: golang-github-matrix-org-gomatrix-devel-0-0.1.20200810gite5578b1.fc32.noarch.rpm
Downloading: growlight-1.2.8-4.fc32.x86_64.rpm
Downloading: jd-0.4.0-1.fc32.x86_64.rpm
Downloading: kexec-tools-2.0.20-17.fc32.x86_64.rpm
Downloading: python3-bytesize-2.4-1.fc32.x86_64.rpm
Downloading: libbytesize-2.4-1.fc32.x86_64.rpm
Downloading: libbytesize-devel-2.4-1.fc32.x86_64.rpm
Downloading: libbytesize-tools-2.4-1.fc32.x86_64.rpm
Downloading: libdkimpp-devel-2.0.0-6.fc32.x86_64.rpm
Downloading: libdkimpp-2.0.0-6.fc32.x86_64.rpm
Downloading: linux-firmware-20200817-111.fc32.noarch.rpm
Downloading: iwl6000-firmware-9.221.4.1-111.fc32.noarch.rpm
Downloading: iwl135-firmware-18.168.6.1-111.fc32.noarch.rpm
Downloading: libertas-usb8388-olpc-firmware-20200817-111.fc32.noarch.rpm
Downloading: libertas-sd8686-firmware-20200817-111.fc32.noarch.rpm
Downloading: iwl5150-firmware-8.24.2.2-111.fc32.noarch.rpm
Downloading: iwl6000g2b-firmware-18.168.6.1-111.fc32.noarch.rpm
Downloading: iwl7260-firmware-25.30.13.0-111.fc32.noarch.rpm
Downloading: liquidio-firmware-20200817-111.fc32.noarch.rpm
Downloading: iwl2000-firmware-18.168.6.1-111.fc32.noarch.rpm
Downloading: iwl5000-firmware-8.83.5.1_1-111.fc32.noarch.rpm
Downloading: netronome-firmware-20200817-111.fc32.noarch.rpm
Downloading: iwl3945-firmware-15.32.2.9-111.fc32.noarch.rpm
Downloading: libertas-usb8388-firmware-20200817-111.fc32.noarch.rpm
Downloading: iwl100-firmware-39.31.5.1-111.fc32.noarch.rpm
Downloading: iwl4965-firmware-228.61.2.24-111.fc32.noarch.rpm
Downloading: iwl3160-firmware-25.30.13.0-111.fc32.noarch.rpm
Downloading: iwl1000-firmware-39.31.5.1-111.fc32.noarch.rpm
Downloading: iwl2030-firmware-18.168.6.1-111.fc32.noarch.rpm
Downloading: libertas-sd8787-firmware-20200817-111.fc32.noarch.rpm
Downloading: linux-firmware-whence-20200817-111.fc32.noarch.rpm
Downloading: iwl6000g2a-firmware-18.168.6.1-111.fc32.noarch.rpm
Downloading: iwl105-firmware-18.168.6.1-111.fc32.noarch.rpm
Downloading: iwl6050-firmware-41.28.5.1-111.fc32.noarch.rpm
Downloading: lm_sensors-libs-3.6.0-5.fc32.x86_64.rpm
Downloading: lm_sensors-sensord-3.6.0-5.fc32.x86_64.rpm
Downloading: lm_sensors-3.6.0-5.fc32.x86_64.rpm
Downloading: lm_sensors-devel-3.6.0-5.fc32.x86_64.rpm
Downloading: mangohud-0.5.1-1.fc32.x86_64.rpm
Downloading: marco-devel-1.24.1-1.fc32.x86_64.rpm
Downloading: marco-libs-1.24.1-1.fc32.x86_64.rpm
Downloading: marco-1.24.1-1.fc32.x86_64.rpm
Downloading: mate-applets-1.24.1-1.fc32.x86_64.rpm
Downloading: mate-panel-libs-1.24.1-1.fc32.x86_64.rpm
Downloading: mate-panel-devel-1.24.1-1.fc32.x86_64.rpm
Downloading: mate-panel-1.24.1-1.fc32.x86_64.rpm
Downloading: muffin-4.6.3-1.fc32.x86_64.rpm
Downloading: muffin-devel-4.6.3-1.fc32.x86_64.rpm
Downloading: nemo-extensions-4.6.5-1.fc32.x86_64.rpm
Downloading: nemo-devel-4.6.5-1.fc32.x86_64.rpm
Downloading: nemo-4.6.5-1.fc32.x86_64.rpm
Downloading: neofetch-7.1.0-1.fc32.noarch.rpm
Downloading: nfdump-libs-1.6.21-1.fc32.x86_64.rpm
Downloading: nfdump-1.6.21-1.fc32.x86_64.rpm
Downloading: notcurses-static-1.6.11-1.fc32.x86_64.rpm
Downloading: notcurses-1.6.11-1.fc32.x86_64.rpm
Downloading: python3-notcurses-1.6.11-1.fc32.x86_64.rpm
Downloading: notcurses-utils-1.6.11-1.fc32.x86_64.rpm
Downloading: notcurses-devel-1.6.11-1.fc32.x86_64.rpm
Downloading: oidn-devel-1.2.2-1.fc32.x86_64.rpm
Downloading: oidn-1.2.2-1.fc32.x86_64.rpm
Downloading: oidn-libs-1.2.2-1.fc32.x86_64.rpm
Downloading: oidn-docs-1.2.2-1.fc32.noarch.rpm
Downloading: openshadinglanguage-doc-1.11.7.0-0.1.fc32.noarch.rpm
Downloading: openshadinglanguage-example-shaders-source-1.11.7.0-0.1.fc32.noarch.rpm
Downloading: openshadinglanguage-common-headers-1.11.7.0-0.1.fc32.noarch.rpm
Downloading: openshadinglanguage-1.11.7.0-0.1.fc32.x86_64.rpm
Downloading: OpenImageIO-plugin-osl-1.11.7.0-0.1.fc32.x86_64.rpm
Downloading: openshadinglanguage-devel-1.11.7.0-0.1.fc32.x86_64.rpm
Downloading: python3-openshadinglanguage-1.11.7.0-0.1.fc32.x86_64.rpm
Downloading: openshadinglanguage-libs-1.11.7.0-0.1.fc32.x86_64.rpm
Downloading: openshadinglanguage-MaterialX-shaders-source-1.11.7.0-0.1.fc32.noarch.rpm
Downloading: osbuild-composer-19-1.fc32.x86_64.rpm
Downloading: osbuild-composer-worker-19-1.fc32.x86_64.rpm
Downloading: osinfo-db-20200813-1.fc32.noarch.rpm
Downloading: partio-doc-1.10.1-3.fc32.noarch.rpm
Downloading: partio-devel-1.10.1-3.fc32.x86_64.rpm
Downloading: partio-1.10.1-3.fc32.x86_64.rpm
Downloading: partio-libs-1.10.1-3.fc32.x86_64.rpm
Downloading: python3-partio-1.10.1-3.fc32.x86_64.rpm
Downloading: phan-3.1.1-1.fc32.noarch.rpm
Downloading: php-bacon-qr-code2-2.0.2-1.fc32.noarch.rpm
Downloading: php-dasprid-enum-1.0.2-1.fc32.noarch.rpm
Downloading: php-doctrine-collections-1.6.7-1.fc32.noarch.rpm
Downloading: php-doctrine-event-manager-1.1.1-1.fc32.noarch.rpm
Downloading: php-doctrine-persistence-1.3.8-1.fc32.noarch.rpm
Downloading: php-doctrine-sql-formatter-1.1.1-1.fc32.noarch.rpm
Downloading: php-egulias-email-validator2-2.1.19-1.fc32.noarch.rpm
Downloading: php-pear-PHP-CodeSniffer-3.5.6-1.fc32.noarch.rpm
Downloading: php-symfony-polyfill-1.18.1-1.fc32.noarch.rpm
Downloading: php-twig-1.43.1-1.fc32.noarch.rpm
Downloading: php-twig2-2.13.1-1.fc32.noarch.rpm
Downloading: php-twig3-3.0.5-1.fc32.noarch.rpm
Downloading: pipewire-gstreamer-0.3.10-1.fc32.x86_64.rpm
Downloading: pipewire-utils-0.3.10-1.fc32.x86_64.rpm
Downloading: pipewire-libs-0.3.10-1.fc32.x86_64.rpm
Downloading: pipewire-doc-0.3.10-1.fc32.x86_64.rpm
Downloading: pipewire-devel-0.3.10-1.fc32.x86_64.rpm
Downloading: pipewire-0.3.10-1.fc32.x86_64.rpm
Downloading: pipewire-alsa-0.3.10-1.fc32.x86_64.rpm
Downloading: pipewire-libjack-0.3.10-1.fc32.x86_64.rpm
Downloading: pipewire-jack-audio-connection-kit-0.3.10-1.fc32.x86_64.rpm
Downloading: pipewire-plugin-jack-0.3.10-1.fc32.x86_64.rpm
Downloading: pipewire-libpulse-0.3.10-1.fc32.x86_64.rpm
Downloading: pipewire-pulseaudio-0.3.10-1.fc32.x86_64.rpm
Downloading: python3-pychm-0.8.6-1.fc32.x86_64.rpm
Downloading: python3-copr-1.105-1.fc32.noarch.rpm
Downloading: python-copr-doc-1.105-1.fc32.noarch.rpm
Downloading: python-pikepdf-doc-1.18.0-1.fc32.x86_64.rpm
Downloading: python3-pikepdf-1.18.0-1.fc32.x86_64.rpm
Downloading: python3-pydantic-1.6.1-1.fc32.noarch.rpm
Downloading: python3-xlsxwriter-1.3.2-1.fc32.noarch.rpm
Downloading: python3-ZEO-5.2.2-1.fc32.noarch.rpm
Downloading: Rex-1.12.1-1.fc32.noarch.rpm
Downloading: roundcubemail-1.4.8-1.fc32.noarch.rpm
Downloading: R-prettydoc-0.4.0-1.fc32.noarch.rpm
Downloading: rubygem-kramdown-doc-2.1.0-3.fc32.noarch.rpm
Downloading: rubygem-kramdown-2.1.0-3.fc32.noarch.rpm
Downloading: rubygem-pkg-config-doc-1.4.2-1.fc32.noarch.rpm
Downloading: rubygem-pkg-config-1.4.2-1.fc32.noarch.rpm
Downloading: s3fs-fuse-1.87-1.fc32.x86_64.rpm
Downloading: slingshot-0.9-12.fc32.noarch.rpm
Downloading: smtpping-1.1.3-12.fc32.x86_64.rpm
Downloading: sqlite-doc-3.33.0-1.fc32.noarch.rpm
Downloading: sqlite-devel-3.33.0-1.fc32.x86_64.rpm
Downloading: sqlite-3.33.0-1.fc32.x86_64.rpm
Downloading: lemon-3.33.0-1.fc32.x86_64.rpm
Downloading: sqlite-tools-3.33.0-1.fc32.x86_64.rpm
Downloading: sqlite-libs-3.33.0-1.fc32.x86_64.rpm
Downloading: sqlite-tcl-3.33.0-1.fc32.x86_64.rpm
Downloading: sqlite-analyzer-3.33.0-1.fc32.x86_64.rpm
Downloading: swtpm-0.3.4-1.20200811git80f0418.fc32.x86_64.rpm
Downloading: swtpm-devel-0.3.4-1.20200811git80f0418.fc32.x86_64.rpm
Downloading: swtpm-libs-0.3.4-1.20200811git80f0418.fc32.x86_64.rpm
Downloading: swtpm-tools-0.3.4-1.20200811git80f0418.fc32.x86_64.rpm
Downloading: tetrinetx-1.13.16-30.fc32.x86_64.rpm
Downloading: urlbuster-0.5.0-2.fc32.noarch.rpm
Downloading: python3-urlbuster-0.5.0-2.fc32.noarch.rpm
Downloading: zswap-cli-0.4.1-1.fc32.x86_64.rpm
File zswap-cli-0.4.1-1.fc32.x86_64.rpm already downloaded, skipping
Usage: koji download-build [options] <n-v-r | build_id | package>
(Specify the --help global option for a list of other help options)

koji: error: Only a single package N-V-R or build ID may be specified
find: cannot delete ‘./repodata’: Directory not empty
Directory walk started
Directory walk done - 12227 packages
Loaded information about 12052 packages
Temporary output repo path: ./.repodata/
Preparing sqlite DBs
Pool started (with 5 workers)
Pool finished

real    2m18.439s
user    1m2.305s
sys     0m3.692s

real    4m52.868s
user    1m47.147s
sys     0m10.093s