Subject: Fedora CoreOS `testing` 38.20231027.2.0 and `next` 39.20231022.1.0 may not receive updates Body: Some recent releases of Fedora CoreOS on `testing` and `next` introduced an issue [[1]] that could prevent them from updating further. The issue was introduced in release 38.20231027.2.0 on the `testing` stream, and release 39.20231022.1.0 on the `next` stream. The issue is fixed in the latest `testing` and `next` releases rolling out over the next day (39.20231101.2.1 and 39.20231106.1.1 respectively), but systems may not have been able to update to them. To verify if a system is affected, run `systemctl status zincati.service` and look for error messages like "EMFILE: Too many open files". Affected systems can be fixed by using the following commands: ``` mkdir -p /run/zincati/config.d cat > /run/zincati/config.d/99-immediate.toml << EOF [updates] strategy = "immediate" [identity] rollout_wariness = 0 EOF systemctl restart zincati ``` This will make Zincati immediately update the system. Luckily, we believe the `stable` stream should not be affected. While the issue did make it to `stable`, a fixed release was made quickly enough to avoid any systems getting stuck. However, if using an update strategy other than `immediate`, you should confirm that the host system has been updated to release 38.20231027.3.2. We apologize for any inconvenience this causes. If you have any questions or concerns, please contact us in the Fedora CoreOS issue [1] or in #coreos:fedoraproject.org on Matrix. $name, for the Fedora CoreOS team [1] https://github.com/coreos/fedora-coreos-tracker/issues/1608 --- We recently released a version of Zincati (our update client) that leaked file descriptors and eventually would hit the limit for a process. This caused systems to get into a state where the update client would no longer be able to update the machine. This issue is documented in #1608 [1] on our issue tracker. We have now produced and released new `testing` and `next` stream builds that do not have this problem and those are rolling out over the next day. To unstick your systems all that is needed is to restart Zincati with `sudo systemctl restart zincati.service`. Luckily, we believe this should not affect the `stable` stream. While the problematic software did make it to `stable` we found and fixed the problem quickly enough that we don't think any systems should have got into this stuck state. [1] https://github.com/coreos/fedora-coreos-tracker/issues/1608 --- stable first broken: 38.20231027.3.1 (Nov 06, 2023) stable fixed: 38.20231027.3.2 (Nov 07, 2023) next first broken: 39.20231022.1.0 (Oct 23, 2023) next fixed: 39.20231106.1.1 (Nov 09, 2023) testing first broken: 38.20231027.2.0 (Oct 30, 2023) testing fixed: 39.20231101.2.1 (Nov 08, 2023)