How To Do A Release
See here.
Current Release Schedule
The list of release executors is maintained here.
2025-04-15 - Yasmin
2025-04-29 - Michael
2025-05-13 - Adam
2025-05-27 - Ash
Michael Armijo changed 3 days agoView mode Like Bookmark
Who's in the hot seat this week? :seat: :studio_microphone:
Apr 28: marmijo
May 5: aaradhak
Action Items
[ ] marmijo: follow up about selinux workaround in >= 41BugZilla
Action: Michael Remove the workarounds and get it tested.
https://github.com/coreos/fedora-coreos-tracker/issues/1926
Michael Armijo changed 4 days agoView mode Like Bookmark
Proposal:
The new updates strategy will not include an update server but it will include
a client that can parse update guidance from a configured location. The update
guidance can be disabled in order to instruct the client to not seek any update
guidance and just use the latest from the current container image that is being
followed.
The update guidance will consist of a single yaml file hosted locally or on
the internet somewhere (file://, https://, docker://). The client knows how
to pull the update guidance and parse it. In the update guidance we define
Scope/Goals?
Who are "we"?"We" are the Fedora CoreOS team
We need to be ready to embrace "bootable containers" when Fedora makes this technology available to us.
What is bootable containers?
we have a "core" thing in Fedora that is built and tested with each package additionnew packages aren't added to the "core" without passing tests
if a package gets in that causes instability it gets ejected?
current editions of Fedora layer on top of this core thing
at the beginninggithub.com/coreos/fedora-coreos-pipeline repo exists with main branch
git clone --depth=1 --branch main https://github.com/coreos/fedora-coreos-pipeline.git
dustymabe changed a year agoView mode Like Bookmark
basic level of information
MCO
How do RHCOS and FCOS relate?
Whats new in RHCOS
CoreOS Layering stories and examples
On Cluster Builds (MCO)
Pre-RHEL-release and major RHEL updates testing with C9S/C10S
dustymabe changed a year agoView mode Like Bookmark
What are the differences between regular s390x VM images vs Secure Execution s390x VM images?
add sdboot partition, and verity partitions (one for boot and one for root)sdboot ext4
verity -> verity thing
generate verity hashes for boot/root so we can verify on first boot that they haven't been tampered with
make a filesystem on the sdboot partition and place an encrypted kernel and initrd in it
encrypt the kernel and initrd and place them here
Jan Schintag changed a year agoView mode Like Bookmark
Common failures
cosa build sometimes tries to fetch packages that were already fetched by cosa fetch
14:59:41 Will download: 1 package (733.6?kB)
14:59:41 Downloading from 'fedora-coreos-pool'...done
14:59:41 [0m[31merror: [0mCannot download Packages/t/tzdata-2022g-1.fc37.noarch.rpm: All mirrors were tried; Last error: Curl error (6): Couldn't resolve host name for https://kojipkgs.fedoraproject.org/repos-dist/coreos-pool/latest/x86_64/Packages/t/tzdata-2022g-1.fc37.noarch.rpm [Could not resolve host: kojipkgs.fedoraproject.org]
14:59:41 error: failed to execute cmd-build: exit status 1
aarch64: network infra flakes for quay.io cdn DNShttps://github.com/coreos/fedora-coreos-pipeline/issues/852
dial tcp: lookup cdn03.quay.io: no such host
Adam Piasecki changed a year agoView mode Like Bookmark
Podman machine OS requirements
Vision
Light-weight, minimal with curated package additions (podman, crun, gvisor-tap-vsock, netavark, aardvark-dns, etc)
Must work for AppleHV, QEMU on Linux, Windows HyperV, and Windows WSL*
WSL is currently based on a Fedora (non-FCOS build).
ManagementManageable via git repo (i.e. we introduce a new dependency, we can add it there)
Automated
Available at Podman release (or nearly thereafter)
dustymabe changed a year agoView mode Like Bookmark
Proposal:
The new updates strategy will not include an update server but it will include a client that can parse update guidance from a configured location. The update guidance can be disabled in order to instruct the client to not seek any update guidance and just use the latest from the current container image that is being followed.
The update guidance will consist of a single yaml file hosted locally or on the internet somewhere (file://, https://, docker://). The client knows how to pull the update guidance and parse it. In the update guidance we define rollouts, barriers, and deadends for each supported stream.
For Fedora CoreOS we'll store the update guidance as a single yaml file in s3 and store it as a single file in a scratch container for mirroring conveniences. These will be updated simultaneously and should always be in sync.
The file format:
streams:
Jonathan Lebon changed a year agoView mode Like Bookmark
Future Work to Scope:
Effort 1 (customer facing, now customers get a better UX)
Complexity HIGH
Integration with osbuild/imagesevaluate current image generation using osbuild/imagesosbuild/images is where IB and other RH tools generate images that are used for production
osbuild-mpp is a dev tool, not really used for production
add new code to osbuild/images to support creating CoreOS images definitions
as part of this understand the architecture of osbuild/images and re-factor things as necessary
dustymabe changed a year agoView mode Like Bookmark
ext.config.butane.grub-userstracked in https://issues.redhat.com/browse/COS-2580
ext.config.boot.bootupd
Should be fixed by https://github.com/coreos/fedora-coreos-config/pull/2786
ext.config.files.root-immutable-bit
tracked in https://issues.redhat.com/browse/COS-2579
Output generated by and stored alongside (with modifications) this script in a fork of the pgm_scripts repo.
Fedora 40 Accepted System-Wide Changes (wiki source)
✔️DNF/RPM Copy on Write enablement for all variantsRPM Copy on Write provides a better experience for Fedora Users as it reduces the amount of I/O and offsets CPU cost of package decompression. RPM Copy on Write uses reflinking capabilities in btrfs, which is the default filesystem starting from Fedora 33 for most variants. Note that this behavior is not being turned on by default for this Change.
Tracking bug: #1915976
NOTES (copied forward): JL: This path of librpm is not used by rpm-ostree. The whole download and unpack path is ostree native and has different tradeoffs. Good to keep track of this conceptually, but nothing for FCOS to do here.
✔️KTLS implementation for GnuTLS
Jonathan Lebon changed a year agoView mode Like Bookmark
we changed the update server URLupdates.stg
updatesmaintained a redirect URL for some time
key rotation
had to change rpm-ostree to not try to fetch the latest
https://github.com/coreos/fedora-coreos-tracker/issues/480#issuecomment-631724629
aarch64 old grub couldn't boot 6.2 kernel
dustymabe changed a year agoView mode Like Bookmark
Subject: Fedora CoreOS testing 38.20231027.2.0 and next 39.20231022.1.0 may not receive updates
Body:
Some recent releases of Fedora CoreOS on testing and next introduced an issue [[1]] that could prevent them from updating further. The issue was introduced in release 38.20231027.2.0 on the testing stream, and release 39.20231022.1.0 on the next stream.
The issue is fixed in the latest testing and next releases rolling out over the next day (39.20231101.2.1 and 39.20231106.1.1 respectively), but systems may not have been able to update to them. To verify if a system is affected, run systemctl status zincati.service and look for error messages like "EMFILE: Too many open files".
Affected systems can be fixed by using the following commands:
Jonathan Lebon changed a year agoView mode Like Bookmark
Suggestions for ways to avoid https://github.com/coreos/fedora-coreos-tracker/issues/1608 in the future:
restart zincati periodicallyallows the process to get out of any stuck state it may be inI think there have been at least two issues where this would have helped
Should have almost no risk / no cost
Switch Zincati to a periodic systemd timer
Instead of having a permanently running background daemon, use a systemd timer to trigger zincati checks at a regular interval
DWM: one problem with this approach may be the periodic timer stuff for finalizing and rebooting the update.TR: The timer would still be triggered every 5 minutes by default which should cover this case
Dusty has grown a wide range of technical skills within the CoreOS team,
where he plays a critical role in translating the upstream work of Fedora
CoreOS (FCOS) into product value RHEL CoreOS (RHCOS). His work has a
direct and indirect impact on Red Hat’s product portfolio including Red
Hat Enterprise Linux, OpenShift, Podman and Podman Desktop, and Edge.
Dusty has grown a wide range of technical skills within the CoreOS team,
where he plays a critical role in translating the upstream work of Fedora
CoreOS (FCOS) into product value for RHEL and RHEL CoreOS (RHCOS). His
work has a direct and indirect impact on Red Hat’s product portfolio
dustymabe changed 2 years agoView mode Like Bookmark
Short presentation about what is Fedora CoreOS
What build tools do we use to build?
Brief overview of the tools that we use to build FCOS
Users get assigned a number for their lab user
Each user gets assigned a number 1..19
Their username to log in to lab instances will then be labuserXi.e. user 1 is labuser1, user 19 is labuser19
For the CoreOS Assembler Tutorial
dustymabe changed 2 years agoView mode Like Bookmark