General Technical Review questions Day 0

- CHECK IF THIS IS THE NEWEST VERSION # General Technical Review questions v1.0 ## Introduction The General Technical Review questions can be completed by a project in lieu of a presentation to a Technical Advisory Group (TAG) as well as to satisfy the Engineering Principle requirements of a Sandbox application or Due Diligence for moving levels. The questions are designed to prompt **_design thinking_** for projects that would like to one day be a graduated project. The intent is to understand how the project has adopted and aligned with the CNCF maturation levels as well as encourage good design & security best practices.  ### General Technical Review questions The questions follow the cloud native software lifecycle day schemas: **Day 0 - Planning Phase. (Sandbox)** - This phase covers design and architecture of the cloud native project. **Day 1 - Installation and Deployment Phase (Incubation)** - This phase covers initial installation and deployment of the design developed during Day 0 - Planning Phase. **Day 2 - Day-to-Day Operations Phase (Graduated)** - This phase covers post-deployment operations in production-ready environments to include monitoring, maintenance, auditing and troubleshooting. ### How to use this template Make a copy of the template below and answer questions related to your project level to the best of your ability. _**Not every question will be addressable or relevant to every project.**_ If this is the case for your project, please mark it as not-applicable (N/A) and provide a brief explanation. **NOTE:** The questions are cumulative e.g. if you are applying for incubation or graduation, you should answer both day 0 and day 1 questions etc. #### Tips * Treat the GTR questionnaire as a living document and keep a copy of it in your project's own repo. The GTR questions are helpful to both contributors and users and will make updating it in the future less work when you want to apply to move levels. * Answer more questions than the requirement for your level if it _makes sense for your project_. e.g. if you have documentation covering the different forms of observability in the Day-2 requirements. * You **CAN** link out to your own project's documentation, but be sure to link to it in a _versioned_ form. e.g. link to it at a specific commit instead of the `main` branch, or versioned website. * A recorded demo or diagram(s) may be easier to convey some of the concepts in the questions below. You may provide a link to a recorded demo or add architectural diagrams along with your GTR questionnaire. * If you are unsure or have a question about any section below, **please ask**. Chances are you're not the only one with a question and the template should be updated with additional guidance. --- # General Technical Review - Flatcar Container Linux / Graduation - **Project:** Flatcar Container Linux - **Project Version:** LTS 4081.3.7 (https://www.flatcar.org/releases#lts-release) as of April 27, () - **Website:** https://www.flatcar.org - **Date Updated:** 2026-04-30 - **Template Version:** v1.0 - **Description:** Flatcar Container Linux is a fully open source, minimal footprint Linux distribution purpose-built for running containerized workloads at scale. It provides an immutable, secure operating system with automated atomic updates, reducing operational overhead and minimizing attack surface. Flatcar is widely deployed across public clouds, on-premises environments, and bare metal, and is deeply integrated with Kubernetes and Cluster API tooling. ## Day 0 - Planning Phase ### Scope * Describe the roadmap process, how scope is determined for mid to long term features, as well as how the roadmap maps back to current contributions and maintainer ladder? * The Flatcar roadmap lives publicly on the [project board](https://github.com/flatcar/Flatcar/blob/main/governance.md), which includes a [long-term roadmap view](https://github.com/orgs/flatcar/projects/7/views/9), a [releases tracker](https://github.com/orgs/flatcar/projects/7/views/24), and [special-interest area views ](https://github.com/orgs/flatcar/projects/7/views/14)covering work like ClusterAPI integration and systemd sysext. Scope decisions happen in the open: community requests feed into the roadmap through monthly Office Hours, Developer Syncs, GitHub Discussions, Slack, and Matrix. Before a new package or feature is accepted, it has to demonstrably serve the project's mission, contributing to security, freshness, the container experience, or automation at scale. Maintainers make the final call on a case by case basis, but the discussion is public and users are actively encouraged to champion requests. The maintainer ladder is merit-based: contributors who show sustained engagement across development, releases, community work, or documentation are nominated by existing maintainers and confirmed by consensus, as documented in the [governance file](https://github.com/flatcar/Flatcar/blob/main/governance.md). * Describe the target persona or user(s) for the project? * The people who use Flatcar are typically platform engineers, SREs, and infrastructure teams responsible for running container workloads at scale. Most of them are operating Kubernetes clusters across public clouds, private infrastructure, or bare metal and need an OS layer that stays out of the way. A significant portion came from CoreOS Container Linux and migrated to Flatcar when CoreOS was discontinued. Secondary personas include teams building automated node provisioning pipelines and maintainers of downstream Kubernetes distributions who need a reliable, low-maintenance base. * Explain the primary use case for the project. What additional use cases are supported by the project? * The primary use case is serving as the OS for Kubernetes nodes, control plane and workers, in environments where security, automation, and consistency matter. Flatcar is not a general-purpose Linux distribution and has never tried to be; it does one thing and focuses on doing it well. Beyond Kubernetes, it also works for containerized applications running outside of an orchestrator, edge nodes, single-instance deployments, and as the base for Cluster API-driven provisioning across cloud and on-premises environments. * Explain which use cases have been identified as unsupported by the project. * Flatcar is not suitable for workloads that require a traditional Linux environment. There is no package manager, the OS filesystem is read-only and immutable, and runtime modification of the OS is not supported by design. Any use case that depends on installing software at runtime, persistent OS-level changes, or an interactive desktop environment is outside the scope of the project. * Describe the intended types of organizations who would benefit from adopting this project. (i.e. financial services, any software manufacturer, organizations providing platform engineering services)? * Flatcar fits any organization running containerized workloads at scale, particularly those with Kubernetes in production. This covers cloud-native software manufacturers, platform engineering teams serving internal or external customers, financial services organizations with strong security and compliance requirements, and large-scale infrastructure operators. Current production adopters include Adobe, running Flatcar across over 18,000 nodes spanning 22 regions worldwide, STACKIT, where Flatcar is the foundation of their managed Kubernetes offering powering over 20,000 nodes and is their customers' most popular OS choice, and Wipro, which uses Flatcar to power their hybrid and multi-cloud PostgreSQL containerized DBaaS platform. AT&T, DeepL, Qualys, and others are also listed in the [adopters file](https://github.com/flatcar/Flatcar/blob/main/ADOPTERS.md). * Please describe any completed end user research and link to any reports. * **TODO:** *Fresh adopter interviews are in progress for graduation. Determine what portion of the ecosystem and end user research can be summarised and made publicly available once interviews are complete.* ### Usability * How should the target personas interact with your project? * Platform engineers and SREs running container workloads at scale should interact with Flatcar as little as possible, and that is by design. The configuration happens once, before the node boots. An operator writes a declarative Butane YAML, transpiles it to Ignition JSON, and passes it to the node at first boot. After that, Flatcar takes care of itself. For the vast majority of teams running Flatcar in production, the OS fades into the background and stays there. * For those who want visibility into what is happening across their fleet, Nebraska provides a web UI for managing update channels and rollout policies. When something on a node genuinely needs attention, SSH and a standard systemd environment are available. Teams coming from a traditional Linux background will find nothing surprising. But the measure of success for Flatcar is not how well it supports interaction. It is how rarely interaction is needed at all. * Describe the user experience (UX) and user interface (UI) of the project. * There is no graphical interface, and there does not need to be. Flatcar's user experience is a YAML file written before the node boots. For teams already working with Kubernetes the pattern is immediately familiar: declare what you want, apply it, and move on.The one place Flatcar surfaces a UI is Nebraska, the update server, where fleet operators manage channels, rollout groups, and version policies across their nodes. The quick-start guide is at https://www.flatcar.org/docs/latest/installing/. * Describe how this project integrates with other projects in a production environment. * Flatcar sits at the OS layer and integrates upward into the full cloud native stack. In production it serves as the host OS for Kubernetes nodes, with containerd as the default runtime, CNI plugins for pod networking, and Prometheus and Grafana for monitoring. Locksmith and the Flatcar Linux Update Operator handle safe rolling reboots across live clusters. * For provisioning at scale, Flatcar works with Cluster API providers across AWS, Azure, GCP, vSphere, and bare metal, and supports managed Kubernetes services including GKE, EKS, and AKS in bring-your-own-image configurations. Official images are published for all major cloud providers with full Terraform support. * A full overview is documented at [Project Integrations](https://docs.google.com/document/d/1f7MgJ_vcq6l1mWYs0uoWHNMxV8eLDSaDbX-lQ3egQK4/edit?tab=t.0). * **TODO:** Make the google document public? ### Design * Explain the design principles and best practices the project is following. * Flatcar's design comes down to a few principles that every decision maps back to. The OS ships only what is needed to run containers. No package manager, no unnecessary services. A smaller OS is harder to attack and easier to reason about. * The filesystem is immutable. The OS partition is read-only and dm-verity protected at runtime, meaning every block is cryptographically verified on access. The OS cannot be changed at runtime, eliminating the configuration drift that affects general-purpose Linux distributions in production. * Configuration is declarative and applied exactly once at first boot. Nothing changes after that point without a new provisioning event, so there are no surprises at scale. * Updates are atomic using an A/B partition model. New releases are written to the inactive partition and activated on reboot. If something goes wrong, the system rolls back automatically to the previous known-good state with no partial update risk. * Every release is reproducible and any past release can be rebuilt from its git tag. SLSA build provenance is included in the image and a signed SBOM is published for each release, giving operators cryptographic confidence in what is running on their nodes. Full supply chain documentation is at https://www.flatcar.org/docs/latest/reference/supply-chain/. * Outline or link to the project’s architecture requirements? Describe how they differ for Proof of Concept, Development, Test and Production environments, as applicable. * Flatcar's disk layout and architecture are documented at https://www.flatcar.org/docs/latest/reference/developer-guides/sdk-disk-partitions/ and cluster architecture patterns at https://www.flatcar.org/docs/latest/setup/clusters/architectures/. **(CHECK IF INFORMATION IN LINK IS UP TO DATE)** * The core architecture is consistent across all environments: a fixed partition layout with A and B OS partitions, an EFI System Partition, an OEM partition for vendor configuration, and a stateful read/write root. What varies by environment is the channel and tooling, not the architecture itself. * For a proof of concept, the QEmu image with a minimal Ignition config is enough to get started locally. For development, the entire build environment ships as a container, the Flatcar SDK, requiring only bash, git, and Docker on the host. Any release can be rebuilt from its tag in the scripts repository. For testing, a self-contained test container runs the full scenario suite locally or against any supported cloud. In production, Nebraska manages channel assignment and fleet rollout policies, and the Flatcar Linux Update Operator handles node draining and reboots in Kubernetes environments. * Define any specific service dependencies the project relies on in the cluster. * Flatcar has very few hard dependencies, which is intentional. At the OS level, systemd is the init system and service manager, containerd is the default container runtime shipped as a sysext, and updateengine manages OS update downloads and the A/B partition swap. Ignition handles first-boot provisioning and has no runtime dependency after that initial run. * For updates, nodes contact a Nebraska instance using the Omaha protocol. By default this is the public Flatcar update server, but Nebraska is fully open source (github.com/flatcar/nebraska) and can be self-hosted for air-gapped or sovereign deployments. Nebraska itself requires only a PostgreSQL database to run. * In Kubernetes environments, coordinated reboots are handled by the Flatcar Linux Update Operator (github.com/flatcar/flatcar-linux-update-operator), which runs as an update-operator Deployment and an update-agent DaemonSet. It requires update-engine.service to be running on each node and replaces locksmith, which is used in non-Kubernetes environments to prevent too many nodes rebooting simultaneously. The two are mutually exclusive and should not run together. * Outside of Kubernetes, etcd can be used for distributed reboot coordination via locksmith, but this is optional. * Describe how the project implements Identity and Access Management. * Flatcar uses standard Linux user and group management, defined declaratively in the Ignition configuration at provisioning time. SSH key-based authentication is the default and recommended access mechanism, there is no password authentication by default. The OS is designed to minimize the number of active accounts at runtime. Cloud-provider identity integration is handled by Afterburn, a one-shot boot agent that fetches provider metadata including SSH keys, hostnames, and instance attributes across 21+ platforms. There is no built-in IAM framework beyond what the underlying OS provides; organizations that require IAM-based SSH access control manage this through their cloud provider or through SSH certificate authorities configured at provisioning time. * Describe how the project has addressed sovereignty. * Flatcar can be operated with zero dependency on any project-hosted infrastructure. The full OS is buildable from source by checking out a release tag in the scripts repository and running the SDK tooling, which requires only bash, git, and Docker. Nebraska is fully open source (github.com/flatcar/nebraska) and self-hostable, including the option to host update payloads locally for fully air-gapped environments. Images can be stored and served from private infrastructure. Organizations with strict data sovereignty, air-gap, or regulatory requirements can run a completely independent Flatcar deployment with no external calls. * Describe any compliance requirements addressed by the project. * Flatcar supports kernel FIPS-200 mode via the fips=1 kernel parameter and ships the OpenSSL FIPS provider, which has been FIPS 140-2 validated since August 2022. It is important to note that Flatcar itself is not officially FIPS certified. It provides the tools to operate in a FIPS-compliant manner, which is a meaningful distinction for regulated environments. CIS benchmark tooling and example reports are maintained at https://github.com/flatcar/Flatcar/tree/main/CIS. The project exceeds SLSA Level 3, with SLSA provenance included in every image and a signed SBOM published for each release. The Linux Auditing System is supported and documented at https://www.flatcar.org/docs/latest/setup/security/. Full supply chain documentation is at https://www.flatcar.org/docs/latest/reference/supply-chain/. * Describe the project’s High Availability requirements. * At the OS layer, HA is built into the A/B partition model. If an update causes a failure, the system automatically reverts to the previously working partition on the next reboot, no operator intervention required. Flatcar nodes are designed to be stateless and replaceable, meaning node loss is handled at the Kubernetes or orchestration layer rather than requiring OS-level coordination. Nebraska supports configurable rollout groups so updates can be staged across subsets of a fleet, limiting blast radius. In Kubernetes environments, FLUO drains nodes before rebooting them to preserve workload availability during updates. * Describe the project’s resource requirements, including CPU, Network and Memory. * Flatcar is intentionally minimal. The OS itself adds negligible overhead compared to a general-purpose Linux distribution. Resource consumption on a node is driven almost entirely by the workloads running on it. The CI infrastructure runs the full scenario test suite on instances as small as 2 vCPUs and 2 GB RAM for smoke tests on DigitalOcean, scaling to larger instance types for multi-cloud full test passes. Network access is required at first boot for Ignition to fetch configuration (if using a remote URL) and for nodes to contact the Nebraska update server for ongoing updates. Air-gapped deployments can disable or redirect both. * Describe the project’s storage requirements, including its use of ephemeral and/or persistent storage. * Flatcar uses a fixed GPT partition layout documented at https://www.flatcar.org/docs/latest/reference/developer-guides/sdk-disk-partitions/. The OS occupies two interchangeable partitions, USR-A and USR-B, used alternately during updates. One is mounted read-only at /usr at runtime. Additional partitions include an EFI System Partition, an OEM partition for vendor-specific configuration and sysext images, and a stateful read/write root for runtime state including /etc and /var. Container workload data uses the standard container storage path. Persistent storage for stateful workloads is managed at the Kubernetes layer through Persistent Volumes and is not a concern of the OS. An installation requires a minimum of 8 GB of disk space. * Please outline the project’s API Design: * Describe the project’s API topology and conventions * Describe the project defaults * Outline any additional configurations from default to make reasonable use of the project * Describe any new or changed API types and calls \- including to cloud providers \- that will result from this project being enabled and used * Describe compatibility of any new or changed APIs with API servers, including the Kubernetes API server * Describe versioning of any new or changed APIs, including how breaking changes are handled * Flatcar does not expose a conventional service API. The primary interface operators interact with is the Ignition configuration schema, which is versioned and co-maintained with Fedora CoreOS. Butane YAML is the human-facing format; it compiles down to Ignition JSON, which is the actual applied configuration. Ignition supports both the 2.x Container Linux Config spec and the current 3.x spec, giving operators flexibility during migration. Breaking changes to the Ignition schema are handled through versioned schema migrations with documented migration paths. Operators pin to a schema version in their Butane files. * The update protocol between Flatcar nodes and Nebraska uses the Omaha protocol. Nebraska exposes a REST API for managing update groups, channels, and rollout policies. Container images for Nebraska are published at ghcr.io/flatcar/nebraska. * Flatcar introduces no Kubernetes CRDs and makes no changes to the Kubernetes API server. The Flatcar Linux Update Operator uses only standard Kubernetes API primitives, a Deployment for the update-operator and a DaemonSet for the update-agent. It communicates with update_engine on each node via D-Bus, not through any custom API surface. * There are no new or changed calls to cloud provider APIs introduced by Flatcar itself. Cloud provider interactions for provisioning (instance creation, image references) are handled by the operator's tooling (Terraform, Cluster API, az/gcloud/aws CLI) using standard provider APIs. * Describe the project’s release processes, including major, minor and patch releases. * Releases move through four channels. Alpha tracks main and receives new major releases frequently. This is where new features land first. Beta stabilizes the release and serves as the canary gate; the project encourages including Beta nodes in production fleets to catch issues early. Stable is the production channel, with new major releases promoted from Beta after passing the full test suite. LTS releases are cut from Stable roughly once every 12 to 15 months, use an LTS upstream kernel, and receive security and critical patch updates for a guaranteed minimum of 18 months with a 6-month overlap before the previous stream reaches end of life. * Bug fixes are released directly to the affected channel. Within each channel, updates are planned on a 14-day cadence. Each release is coordinated by a rotating release engineer who tracks progress in a GitHub issue using a structured template. The process requires a Go/No-Go meeting, all tests passing in the HackMD tracking document, a diff review of image file lists and package changes, and a final build via Jenkins before publishing. Releases are announced in the CNCF and Kubernates Slack #flatcar channels, and Discord and Matrix. The full release guide is at https://www.flatcar.org/docs/latest/reference/developer-guides/release-guide/. ### Installation * Describe how the project is installed and initialized, e.g. a minimal install with a few lines of code or does it require more complex integration and configuration? * Installation follows three steps regardless of environment: write a Butane YAML configuration, transpile it to Ignition JSON, and pass that JSON to the node at first boot. On public clouds this means setting it as user-data at instance creation time. On bare metal it is passed via the flatcar-install script. The minimal Butane config to get a node running is a few lines of: ```YAML: variant: flatcar version: 1.0.0 passwd: users: - name: core ssh_authorized_keys: - ssh-rsa AAAA... ``` Transpile it with the Butane container: ```BASH: cat config.yaml | docker run --rm -i quay.io/coreos/butane:latest > ignition.json ``` Then pass it to the provider. For example on AWS: ```BASH: aws ec2 run-instances --image-id <flatcar-ami> --instance-type t3.medium \ --user-data file://ignition.json ``` For local testing with QEMU: ```BASH: wget https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu.sh wget https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu_image.img chmod +x flatcar_production_qemu.sh ./flatcar_production_qemu.sh -i ignition.json ``` * How does an adopter test and validate the installation? * The most reliable way to validate a configuration before deploying it to production is to test it locally using the QEMU image. Flatcar publishes a QEMU image and helper script that let adopters provision a local VM with their exact Ignition config in minutes, verify services start correctly, and iterate on the configuration before it touches real infrastructure. Using ```butane --strict``` during transpilation catches configuration errors at compile time before any node boots. Once a node is running, standard Linux tooling, ```systemctl status```, ```journalctl```, SSH, is available to verify the provisioned state. The quick-start guide walks through this process at https://www.flatcar.org/docs/latest/installing/. ### Security * Please provide a link to the project’s cloud native [security self assessment](https://tag-security.cncf.io/community/assessments/). * The TAG Security self-assessment and joint assessment are published at https://tag-security.cncf.io/community/assessments/projects/flatcar/joint-assessment/ and https://tag-security.cncf.io/community/assessments/projects/flatcar/self-assessment/. * **TODO:** The existing self-assessment was completed for incubation. A graduation-specific update and third-party security review are currently in progress and will be linked here once complete. * Please review the [Cloud Native Security Tenets](https://github.com/cncf/tag-security/blob/main/community/resources/security-whitepaper/secure-defaults-cloud-native-8.md) from TAG Security. * How are you satisfying the tenets of cloud native security projects? * Flatcar was designed with security as a first principle, not a retrofit. The immutable read-only OS partition, dm-verity verification, no package manager, and automatic updates are not optional features, they are the architecture. Every node that boots Flatcar inherits these controls without any configuration required from the operator. Where operators genuinely need to loosen defaults, such as deferring updates in a staged rollout, the project provides documented and explicit mechanisms to do so. Security limitations, such as the remaining gaps to SLSA Level 4, are publicly tracked and explained. The project aims to satisfy all 8 tenets of the Cloud Native 8 framework. * Describe how each of the cloud native principles apply to your project. 1. **Make security a design requirement.** Security is Flatcar's founding principle. dm-verity, immutability, and automatic patching were design decisions from day one, not additions. The project uses threat modeling and a dedicated security team with a daily runbook to keep security embedded in every release. 2. **Applying secure configuration has the best user experience.** A minimal Butane YAML is all that is needed to provision a secure, fully updated node. There is nothing additional an operator needs to do to be secure out of the box. 3. **Selecting insecure configuration is a conscious decision.** Disabling automatic updates requires explicitly setting ```SERVER=disabled``` in the update configuration. There is no accidental path to an insecure state; operators have to deliberately choose it. 4. **Transition from insecure to secure state is possible.** The four-channel model (Alpha, Beta, Stable, LTS) lets operators move fleets gradually toward newer, more secure releases. Nebraska's rollout groups make staged migrations across a fleet straightforward. 5. **Secure defaults are inherited.** Every Flatcar node inherits dm-verity, a read-only OS partition, SSH-only access, and automatic updates by default. No operator action is required. 6. **Exception lists have first-class support.** Nebraska supports custom rollout groups and version pinning, giving operators explicit and tracked control over which nodes receive which updates and when. 7. **Secure defaults protect against pervasive vulnerability exploits.** The minimal package set and immutable OS eliminate whole categories of exploit. The XZ/CVE-2024-3094 supply chain attack is a real example: Flatcar was unaffected because it does not patch OpenSSH to link against libsystemd, a design decision that paid off directly when it mattered. 8. **Security limitations of a system are explainable.** The project publicly documents where it falls short of SLSA Level 4, what the remaining gaps are, and what is being done about them. The supply chain documentation at https://www.flatcar.org/docs/latest/reference/supply-chain/ explains both what is in place and what is not yet. * How do you recommend users alter security defaults in order to "loosen" the security of the project? Please link to any documentation the project has written concerning these use cases. * The honest recommendation from the project is not to loosen the defaults in production. That said, there are documented scenarios where operators need to adjust things and the project makes those paths explicit. * Automatic updates can be deferred or disabled by setting ```SERVER=disabled``` in ```/etc/flatcar/update.conf```. This is the recommended approach over masking the update-engine service, which makes manual updates harder later. For temporarily pausing updates, stopping update-engine and locksmithd is the right method. Full update strategy options including maintenance windows and reboot policies are at https://www.flatcar.org/docs/latest/setup/releases/update-strategies/. * Additional security configuration including the Linux Auditing System and FIPS mode is documented at https://www.flatcar.org/docs/latest/setup/security/. **Security Hygiene** * Please describe the frameworks, practices and procedures the project uses to maintain the basic health and security of the project. * Security hygiene in Flatcar operates across three layers: organizational structure, process cadence, and architectural design choices that function as security controls in their own right. * **Organizational structure** Security ownership sits with a dedicated [Flatcar Security Team](https://github.com/orgs/flatcar/teams/flatcar-security-team), a named subset of the maintainers elected by the maintainers group. The team meets fortnightly *(check cadence)* in a private video call and maintains a weekly rotation of security Primaries and Secondaries, who are expected to actively engage in security work each day, including but not limited to executing the runbook and working on ongoing security issues. Undisclosed or embargoed issues are tracked in a private repository accessible only to security team members; public issues are tracked in the main GitHub issue tracker using ```security``` and ```advisory``` labels. * **Daily security runbook** Every day, Primaries work through upstream security trackers including Gentoo GLSAs, the oss-security mailing list, the Golang announce mailing list, and Rust security announcements. Newly discovered CVEs are added to an internal database and automation creates a corresponding issue in the Flatcar GitHub tracker. Team members are also subscribed to private NDA security lists for pre-disclosure access to embargoed vulnerabilities. Each finding receives an independent severity assessment, separate from the upstream CVSS score, to decide whether an urgent out-of-cycle release is warranted. * **Vulnerability reporting and embargo handling** Security issues can be reported privately via security@flatcar-linux.org, documented on flatcar.org/security. For embargoed issues, the team backports patches into private builds and publishes them at embargo lift. Rather than waiting for the upstream release, the team applies patches downstream, a common LTS practice, allowing patched releases to ship precisely at embargo lift without the risk of unrelated upstream changes. Every release includes a concise CVE summary in the announcement and at flatcar.org/releases, plus a per-CVE detailed report. This is a structured part of the release process, not an optional practice. * **Development process controls** All changes require peer review by at least one maintainer who did not author the change. Only maintainers can push branches or merge PRs; external contributors work from forks. CI builds and tests run on dedicated, separate infrastructure. 2FA is enforced across the Flatcar GitHub organisation. * **Supply chain security** Flatcar complies with SLSA Level 3 and is working to address the remaining requirements for Level 4. The build is fully reproducible from any release tag. All source inputs and binary outputs are attested using in-toto/SLSA 0.1 JSON format. A signed SBOM is published with every release. Release artifacts are signed via an offline hardware-token signing procedure before publication, keeping signing keys out of the build pipeline. * **Architecture as hygiene** The base image ships fewer than 300 packages, a deliberate ceiling that keeps the CVE surface small and the daily runbook tractable for a small team. The immutable read-only ```/usr``` partition means a vulnerable binary cannot be replaced at runtime. Automatic updates ensure security patches reach nodes without operator action. The practical value of these choices was demonstrated during XZ/CVE-2024-3094: Flatcar was unaffected because it deliberately does not patch OpenSSH to link against libsystemd, a design decision that paid off directly. CIS benchmark tooling and example reports are maintained at [github.com/flatcar/Flatcar/tree/main/CIS](https://github.com/flatcar/Flatcar/tree/main/CIS) for operator self-service; the project does not currently run these automatically as part of its own release process, and acknowledges this as a gap. * Describe how the project has evaluated which features will be a security risk to users if they are not maintained by the project? * The project's primary control is **architectural minimalism enforced by an explicit inclusion policy**. Before any package or feature is accepted into the base image, it must demonstrably serve the project's core mission. The bias is toward exclusion: features that don't clear that bar don't ship and can't become unmaintained liabilities. For features that do ship, maintenance risk is evaluated through several lenses: * **Package count as a risk budget.** The base OS carries fewer than 300 packages, with under 500 upstream dependencies total across the OS and SDK. This is a deliberate ceiling, every package added is one that requires ongoing CVE tracking and release validation. The small count is what makes comprehensive daily security review feasible for the team. * **Sysexts as risk containment.** Features that are useful but not universally needed, Docker, Podman, CRI-O, Incus, are delivered as system extensions rather than baked in. A sysext that falls behind on security updates affects only users who opted into it, and can be updated or deprecated independently of the base OS release cycle. * **Explicit tracking of gaps.** The project publicly documents where security coverage is incomplete. SLSA Level 4 remaining gaps are documented at flatcar.org/docs/latest/reference/supply-chain/. The TAG Security self-assessment documents residual risk in project infrastructure components such as the image servers and Nebraska update server. The competitive gaps analysis tracks security capabilities that are not yet shipped (SELinux enforcing by default, Secure Boot, dm-verity on ```/usr```) and notes active workstreams where applicable. * **First-class treatment of high-consequence components.** The update engine is the clearest example: if ```update-engine``` became unmaintained, every Flatcar node would stop receiving security patches. The project treats this as a primary maintenance commitment, not a peripheral one. Nebraska's open-source, self-hostable design means operators can continue delivering updates to their own fleets independently of project-operated infrastructure. The Ignition schema is co-maintained with Fedora CoreOS, distributing that maintenance responsibility across a broader upstream community. * **Elevated bar for features that open network-accessible surfaces.** The only service listening by default on a Flatcar node is sshd on port 22. Any new feature that would add a persistent listening service faces a high inclusion bar precisely because of the ongoing maintenance commitment and attack surface it would represent. **TO DO/INVESTIGATE:** The project does not currently have a formal, periodic process for re-evaluating already-shipped features against a security sustainability framework. The weekly runbook implicitly surfaces packages that are difficult to keep patched, but a structured feature sunset process would strengthen this, particularly as the sysext ecosystem grows. **Cloud Native Threat Modeling** * Explain the least minimal privileges required by the project and reasons for additional privileges. * Flatcar's privilege surface is fixed at build time by the OS architecture. Because the filesystem is immutable and there is no package manager, no new privileged processes can be added at runtime without a full OS update. * **Default user** Flatcar has a single default user account called ```core```, which has access to the ```wheel``` group and therefore ```sudo```. The hardening guide documents how to restrict this: either require a sudo password without setting one, or disable login for the core user entirely. * **Services requiring root** Two OS-level services run as root, both for unavoidable reasons: * ```update-engine``` must write a full OS partition image to the inactive block device and signal GRUB to switch partitions. There is no lower-privilege path for a service that manages bootloader state and raw disk writes. * ```afterburn``` is a one-shot boot agent that writes SSH keys and provider metadata to the system. It runs once at provisioning and has no persistent daemon footprint. * **Flatcar Linux Update Operator (FLUO)** In Kubernetes environments, FLUO's ```update-agent``` DaemonSet communicates with ```update-engine``` on each node via D-Bus and requires permission to annotate nodes and coordinate drain operations. It uses standard Kubernetes RBAC with no cluster-admin rights beyond what node draining requires. * **Network exposure** The only service listening by default on a Flatcar node is sshd on port 22. All other services are absent or must be explicitly enabled via Ignition. The hardening guide at flatcar.org/docs/latest/setup/security/hardening-guide/ documents further restrictions including disabling the Docker socket's group access, disabling SMT, and blacklisting unused kernel modules. * Describe how the project is handling certificate rotation and mitigates any issues with certificates. * Flatcar does not ship a certificate lifecycle daemon. This is an intentional scope boundary: the OS provides the plumbing, operators own the rotation. * **What Flatcar provides** Certificates are provisioned at first boot via Ignition, which supports injecting TLS material and custom CA bundles declaratively before any service starts. The Butane/Ignition spec supports a ```tls``` stanza for additional certificate authorities, each referenced by URL and optionally hash-verified. Documentation for generating self-signed certificates using ```cfssl``` for services like etcd is at flatcar.org/docs/latest/setup/security/generate-self-signed-certificates/, and adding custom CAs to the system trust store is documented at flatcar.org/docs/latest/setup/security/adding-certificate-authorities/. * **What operators own** Runtime certificate rotation, for etcd, for workloads, for Kubernetes control plane components, is managed by the operator's tooling. In Kubernetes environments this is typically ```cert-manager```. Because ```/etc``` is writable, certificates placed there can be updated without an OS update; the relevant service simply needs restarting to pick up new material. * **Project-internal signing key rotation** The Flatcar image signing key has a one-year lifetime. Renewal requires split secrets from multiple maintainers, ensuring no single person can rotate the key unilaterally. * Describe how the project is following and implementing [secure software supply chain best practices](https://project.linuxfoundation.org/hubfs/CNCF\_SSCP\_v1.pdf) * The full technical documentation is at flatcar.org/docs/latest/reference/supply-chain/. The summary: * **SLSA Level 3, working toward Level 4.** Flatcar meets all SLSA requirements through Level 3: version-controlled source with verified history retained indefinitely and two-person review; scripted builds on a build service in an ephemeral isolated environment with build-as-code; authenticated, service-generated, non-falsifiable provenance with complete dependency tracking. The two remaining Level 4 gaps, hermetic builds and two-administrator approval for build infrastructure changes, are publicly tracked. * **Build from source.** All artifacts are built from source with no pre-generated binaries. The SDK used to build is itself the validated output of a previous build. Upstream source tarball integrity is validated against cryptographic checksums stored in the package definition repos before any build begins. * **Reproducible builds.** Any past release can be rebuilt from its git tag. Output is bit-for-bit identical except where upstream software inserts volatile values like timestamps at compile time. * **Per-package SLSA provenance and signed SBOM.** SLSA provenance is generated for each package during the build and shipped within the image at ```/usr/share/SLSA/```. A signed SBOM is published alongside every release. * **Offline HSM signing for update payloads.** Update images are signed with a key stored on a hardware security module in an air-gapped environment, ensuring the signing key is never exposed to the internet or the build pipeline. ```update_engine``` validates every update payload against a baked-in public key before installation. * **Runtime integrity via dm-verity.** All OS binaries live on a read-only partition mounted at ```/usr```. The partition is validated on every boot using dm-verity, with the verity hash baked into the initrd at build time. * **Build infrastructure access controls.** Builds run on a dedicated bare-metal machine in an access-controlled data center, reachable only via VPN with SSH key authentication, limited to a small number of core maintainers. Accounts with access to the update server use 2FA.