# 2021.8 kfan braindump (CoreOS Intern 2020-2021) ## Zincati ### New marker-file strategy #### Status The implementation of the new strategy is fully complete, including tests. There has been [some concerns](https://github.com/coreos/zincati/pull/540#issuecomment-852217849) regarding the name, particularly around the lack of any indication that the presence of the marker file indicates allow finalization or disallow finalization. I slightly disagree, as outlined [here](https://github.com/coreos/zincati/pull/540#issuecomment-852311323), but definitely open to changing my opinion! #### What's next - Regarding future improvements to the marker-file strategy, we could consider adding an inotify watch on the `allowfinalize.json` file to tick now so reboots happen immediately if an update is staged. Also consider extending the marker file with an additional `persistAfterReboot` key where Zincati will remove the file before rebooting if false. - The marker-file strategy crucially still requires a central controller that is fully in charge of finalization decisions. The marker-file strategy merely provides an alternative mechanism for such a central controller to communicate its wishes. This means that the marker-file strategy is _not_ suitable for use by multiple uncoordinated agents. An alternative strategy/approach will be required for the uncoordinated use case. #### Links Rationale: https://github.com/coreos/zincati/issues/245#issuecomment-823705074 Main PR: https://github.com/coreos/zincati/pull/540 ### systemd-inhibitor locks #### Status rpm-ostree now respects systemd-inhibitor locks. Related to the new marker-file strategy, users can use `systemd-inhibit` today to prevent Zincati from finalizing (and rebooting into) updates. #### What's next We don't want users to (possibly unintentionally) be able to prevent reboots indefinitely. Possibly add an option in rpm-ostree to `--ignore-inhibitors` and teach Zincati to use this option after a certain amount of attempts to finalize. #### Links https://github.com/coreos/rpm-ostree/pull/2862 ### Manual update checks and reboots #### Status Summary: most of the infrastructure for Zincati to be controlled and ushered between states should be complete. However, no actual useful commands exist at the moment. A D-Bus server interface has been set up (as an additional actor in the Zincati actor model). It has a couple experimental proof-of-concept D-Bus APIs that are currently exposed, including a `LastRefreshTime` that demonstrates proper communication with the main `update_agent` actor. Extra care has been taken to allow Zincati's `update_agent` actor to handle requests concurrently in general. A good amount of refactoring was done to put the update agent's state behind a more granular RwLock and splitting out atomic fields where possible. This work can be found in the links under "Groundwork" below. #### What's next Hash out the details of the desired behaviour for when a user issues commands to Zincati. Some concerns are raised [here](https://github.com/coreos/zincati/issues/554#issuecomment-842962400). Implement useful methods like `check-update` and `finalize-update`, as detailed in the RFE. #### Links RFE: https://github.com/coreos/zincati/issues/498 Groundwork: - https://github.com/coreos/zincati/pull/514 - https://github.com/coreos/zincati/pull/577 - https://github.com/coreos/zincati/pull/613 - https://github.com/coreos/zincati/pull/614 - https://github.com/coreos/zincati/pull/619 - https://github.com/coreos/zincati/pull/624 Concerns: - https://github.com/coreos/zincati/issues/554#issuecomment-842962400 ### Verify fedora-coreos.stream commit metadata before unlocking #### Status Implementation complete, waiting for review. #### Links PR: https://github.com/coreos/zincati/pull/622 ### External kola tests #### Status Kolet can now easily start a HTTP server with the `httpd` command. This is especially useful for Zincati tests where we would like to have a mock Cincinnati server and mock OSTree repo server. An Igntion file has been added in Zincati's `tests/` directory with [instructions](https://coreos.github.io/zincati/development/testing/#server-tests) on how to make use of it to conveniently create new integration tests for Zincati. Example usage can be found at e.g. https://github.com/coreos/zincati/blob/0e7aff5a26a7198dcd7bdf86a83b8fdd89c9488e/tests/kola/server/test-marker-file-strategy.sh #### What's next Actual test files under `tests/server/` are quite long because they still need to create a mock Cincinnati graph to serve. Possibly create a test library to house functions to help create mock Cincinnati graphs to make the code in these test files less repetitive. #### Links Documentation: https://coreos.github.io/zincati/development/testing/#server-tests Example: https://github.com/coreos/zincati/blob/0e7aff5a26a7198dcd7bdf86a83b8fdd89c9488e/tests/kola/server/test-marker-file-strategy.sh Issue: https://github.com/coreos/zincati/issues/626 ### CI flake #### Status Cause unknown. Some guesstimates [here](https://github.com/coreos/fedora-coreos-tracker/issues/756#issuecomment-843700122). Some [additional tracing](https://github.com/coreos/zincati/pull/559) has been added in Zincati to possibly help provide more information when this flake happens again. #### Links Tracker: https://github.com/coreos/fedora-coreos-tracker/issues/756 ## console-login-helper-messages ### Switch to `/run` #### Status Following https://github.com/coreos/console-login-helper-messages/issues/60, console-login-helper-messages tries to make use of upstream tools' to handle the concatenating and displaying of snippets, thus vastly reducing the logic in console-login-helper-messages. These changes landed in v0.21.1, with quite some differences in behaviour from console-login-helper-messages v0.20.3 and under. Notably, console-login-helper-messages [no longer supports](https://github.com/coreos/console-login-helper-messages/blob/main/doc/manual.md#appending-messages) appending messages to the set of snippets that it already displays by dropping snippets into `/run/console-login-helper-messages`. However, this caused console-login-helper-messages to depend on `util-linux 2.36+`. Because `util-linux 2.36` is not yet in RHEL, the console-login-helper-messages packaged in RHCOS is different from the one packaged in FCOS. Because of this, if we need to make any changes that we would like to land in RHCOS (e.g. bug fixes or features), we would need to split off a `legacy` branch based off of v0.20.3 that still does not depend on `util-linux 2.36`. Luckily, there hasn't yet been such a requirement. The [manual](https://github.com/coreos/console-login-helper-messages/blob/main/doc/manual.md#upstream-tools-requirements) details which dependencies are required and why. #### What's next Once `util-linux 2.36` or higher lands in RHEL, package the latest version of console-login-helper-messages for RHCOS to avoid having to branch off a separate branch for RHCOS. #### Links Issue:https://github.com/coreos/console-login-helper-messages/issues/60 Upstream tools' requirements for v0.21.1+: https://github.com/coreos/console-login-helper-messages/blob/main/doc/manual.md#upstream-tools-requirements ### `/run/issue.d` #### Status console-login-helper-messages currently drops issue file snippets into `/etc/issue.d` instead of `/run/issue.d` for the following reason: https://github.com/coreos/console-login-helper-messages/blob/e06fc88ae8fbcc3a422bc8c686f70c15aebb9d9a/usr/lib/console-login-helper-messages/issue.defs#L7-L17 #### What's next The current approach is not so elegant, but no problems have been encountered so far. However, ideally, if we can find a way to remove `/etc/issue`, then we can directly drop snippets into `/run/issue.d`. ## kdump Note: travier should be just as familiar, if not more familiar, with the current status of `kdump`-related work. ### Get kdump to place its initramfs in `/var/lib/kdump` JIRA: https://issues.redhat.com/browse/GRPA-3209 #### Status [Patch to `kexec-tools`](https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org/thread/G5LMDXOA46A7AXGQIXPZ3XU65H52D4I4/) is merged. This is dependent on a modification to the SELinux policies around `kdump` as without it, `kdump` is not able to access the `/var/lib/kdump` directory. On the FCOS side, everything should be in place for using `/var/lib/kdump`. On the RHCOS side, `kexec-tools` [patch](https://bugzilla.redhat.com/show_bug.cgi?id=1918499) should have been backported to 8.4 (https://bugzilla.redhat.com/show_bug.cgi?id=1976252), and same for the SELinux policy update. #### BZ's - F33 `kexec-tools` BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1918493 - RHEL 8.6 `kexec-tools` BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1918499 - `kexec-tools` patch backport to 8.4: https://bugzilla.redhat.com/show_bug.cgi?id=1976252 - F33 SELinux BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1951323 - RHEL 8.5 SELinux BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1965985 - SELinux patch backport to 8.4: https://bugzilla.redhat.com/show_bug.cgi?id=1976260 ### `/boot` RW workaround JIRA: https://issues.redhat.com/browse/GRPA-3862 #### Status We currently have a workaround in RHCOS while "Get kdump to place its initramfs in `/var/lib/kdump`" is being backported. #### What's next Once everything is in place for kdump to place its initramfs in `/var/lib/kdump` automatically, we should revert https://github.com/coreos/fedora-coreos-config/pull/936. ### `dracut-squash` dependency #### Status In Fedora, `dracut-squash` was made a weak dep in https://src.fedoraproject.org/rpms/kexec-tools/pull-request/4 (in f34) and then made a hard dependency again in https://lists.fedoraproject.org/archives/list/kexec@lists.fedoraproject.org/message/MX7INNHDHXKS4JXARMSYYHVEV2HOFHID/ (will be in f35). Not having `dracut-squash` caused some minor issues where a very large portion of memory needed to be reserved for kdump https://github.com/coreos/fedora-coreos-tracker/issues/798, leading to an update in our documentation to instruct users to reserve more memory. #### What's next Once FCOS is based on f35, I believe we can remove the notice to reserve so much memory for kdump in the documentation. ## rpm-ostree ### Better integration with update drivers #### Status Commands like `rpm-ostree upgrade/rebase/deploy` will not work without the `--bypass-driver` flag when an update driver (such as Zincati) is registered. `rpm-ostree` will otherwise error out and point the user to the update driver's documentation. But for Zincati, there are actually no such mechanisms to do actually do manual upgrades. #### What's next Once Zincati gains the ability to do [manual upgrades](https://github.com/coreos/zincati/issues/498), perhaps we can have rpm-ostree directly call Zincati's D-Bus methods. This may be better than having Zincati have it's own CLI since users already know and are familiar with `rpm-ostree`. ## Contact Information Email: kelvinfan001@gmail.com Github: https://github.com/kelvinfan001 LinkedIn: https://www.linkedin.com/in/kelvin-yao-fan/