Enabling kdump on CoreOS

Overview of kdumpctl (executed by kdump.service in the real root)

1. check dump feasibility
   • checks if kdump is supported on this kernel and whether memory is reserved
2. check config
   • check if /etc/kdump.conf has valid configuration
3. check rebuild
   • setup initrd
     • sets appropriate variables for building the initrd, notably, checks /etc/sysconfig/kdump to see if a kernel is specified to be used as the crash kernel, otherwise, set currently running kernel as the kernel for the initrd.
     • sets the value for TARGET_INITRD, which is always ${KDUMP_BOOTDIR}/initramfs-${kdump_kver}kdump.img
   • check system modified
     • check whether TARGET_INITRD exists, if not, return MODIFIED
     • check whether config files (/etc/kdump.conf) were modified
   • rebuild initrd if forced to or system is modified via mkdumprd
4. load the crash kernel
   • call kexec to load the kernel specified in /etc/sysconfig/kdump. If none is specified, use the currently running kernel as the crash kernel.
   • additional kexec args and kernel cmdline arguments for the crash kernel can be supplied by configuring /etc/sysconfig/kdump.
   • without configuration, /proc/cmdline (plus some additional arguments added by kdump) will be used as cmdline arguments for the crash kernel

Integrating into *COS

1. My understanding is that we want all the configuration of kdump to be done in a declarative manifest, and just have it "work" without further manual configuration after booting up.
2. Kernel arguments need to be supplied to the main kernel to reserve memory for the crash kernel and kdump initrd
3. We would like to avoid lettingkdump generate initrds behind rpm-ostree's back so ostree can own /boot modifications.
4. Regarding https://github.com/coreos/fedora-coreos-tracker/issues/622#issuecomment-692362905, we need to make sure the KDUMP_COMMANDLINE variable is configured in /etc/sysconfig/kdump

Proposed Flow

1. Include kexec-tools in the FCOS image.
2. Ignition edits necessary configuration for kdump.
3. Ignition adds the karg crashkernel=256M for the real root. This would require changing the dracut module to reboot at the switch-root point if kargs have been modified.
4. Ignition enables kdump.service.
5. Reboot at switch-root.
6. kdump.service generates its kdump initrd (based off of current running kernel) and places it alongside the main initramfs and kernel (/boot/ostree/fedora-coreos-$bootchecksum).
7. kdump.service loads the kdump initrd and kernel into memory via kexec.
8. kdump is now armed.
9. Update involving a new kernel happens. "BOOT_IMAGE" in /proc/cmdline changes to e.g. /boot/ostree/fedora-coreos-$newbootchecksum.
10. kdump.service detects no kdump initrd exists in /boot/ostree/fedora-coreos-$newbootchecksum and repeats Step 7 and Step 8.
11. kdump is now armed.

Notes

In the above flow, it seems like OSTree/rpm-ostree is not too involved, and most of the work to enable kdump, particularly Step 3, actually involves the First Boot team.
However, Luca also mentioned that something similar to Step 3 can be done through OSTree, as well, thanks to a not-yet-merged PR by Robert.

Another approach would be to just do everything after the switch-root through a systemd unit. I have verified that this works. However, Benjamin mentioned that it is unsafe to reboot from a systemd unit.

To Discuss

• Should we add kexec-tools to the base image or keep it as an extension?
• Support in Ignition to add karg support and reboot as needed before the switch root?
• Should we add sugar to fcct for kdump? How much?
• How much does OSTree want to own the /boot directory? (kdump generates a separate independent initrd in the /boot/ostree/$bootcsum directory)