# Flatcar Infra 2.0 & Equinix Metal migration

*Sync every thursday 10AM CET / 6PM KST / 2:30pm IST https://meet.flatcar.org/OfficeHours*

Board: https://github.com/orgs/flatcar/projects/12

## Thursday 11th Dec. 2025

Attendees:
- Jan
- Thilo
- Mathieu

Notes:
- Equinix metal: blogpost
- akamai / linode: to put Lexi in contact with them
- safe-settings: PR is opened, to have a look next weeks

## Thursday 4th Dec. 2025

Attendees:
- Jan
- Thilo
- Mathieu

Notes:
- remaining stuffs:
- vsphere: Meeting w/ Broadcom to talk about Flatcar sponsoring on vSphere
- jenkins: move on azure for the workers (reuse the IaaC)
- garm workers
- bincache quotas approved
- mirror nigthlies:
- source tarballs are stored on origins (via: https://github.com/flatcar/flatcar-linux-infra/blob/master/ansible/roles/origin-local-storage/files/source-mirror.sh)

Todos:
- [ ] version tfstate

## Thursday 27 Nov. 2025

Attendees:
- Jan
- Kai
- Sayan
- Ervin
- Thilo
- Mathieu

Notes:
- LTS promoted (with a side effect on our automation)
- asked: bincache bucket quotas increase (it's likely to be approved)
- bucket instance on Equinix Metal: was used by Mantle testing

Todos:
- [ ] version tfstate

## Thursday 20 Nov. 2025

Attendees:
- Jan
- Mathieu
- Thilo

Notes:
- investigate arm64 virtualization
- focus for the next days:
- finish mirror / updates DNS promotion (LTS to come on Monday)
- jenkins workers

Todos:
- [ ] version tfstate

## Thursday 13 Nov. 2025

Attendees:
- Jan
- Mathieu
- Thilo
- Sayan
- Kai

Notes:
- Equinix Metal: meeting. questions regarding control plane
- stable promotion: monday morning
- cleanup check for bincache?
- cleanup old releases (> 2 months ago)
- cleanup 1 week stuff (nightlies)
- hetzner to sponsor arm64 colocation?
- vsphere -> we run tests as long as possible.
- should we investigate on running on GCP? how much it costs?
- should we ask to vmware directly?

Todos:
- [ ] version tfstate
- [x] Cloudbase for ARM64
- [x] cleanup check for bincache? (https://github.com/flatcar/scripts/blob/main/ci-automation/garbage_collect.sh#L156)
- [x] hetzner to sponsor arm64 colocation? (https://www.hetzner.com/colocation/)

## Thursday 6 Nov. 2025

Attendees:
- Jan
- Mathieu
- Thilo
- Kai

Notes:
- bincache is up for review
- origin servers:
- LTS, Stable -> to be done (stable: this week, lts: next week)
- mirror (and origin migration has to be done)
- t-lo:
- update payloads + sysexts updates -> in the same place as the release?
- could be in the same directory
- kai: has to be channel agnostic
- conclusion: let's keep this in the current state for now

- jan:
- sharing the CNCF status regarding the CF account
- misc has been merged
- CT provider:
- CT is missing a release: https://github.com/poseidon/terraform-provider-ct/pull/227
- https://github.com/poseidon/terraform-provider-ct/releases/tag/v0.14.0

- kai:
- update payload via SSH or bucket?
- -> let's do bucket

discussion about:
- vsphere sponsoring
- aws access for domain names
-

Todos:
- [x] Cloudbase for ARM64

Todos:
- [x] we can ask to Hetzner for ARM64 instances :arrow_down:

## Thursday 30 Oct. 2025

Attendees:
- Jan
- Mathieu
- Ervin
- Sayan

Notes:
- misc module foundation has been merged + documentation
- CF/OpenTofu/CNCF thing:
- meeting with Shah from CNCF about Flatcar CF bucket
- TL; DR: we need our own CF account instead of the shared one
- We asked:
- own account with admin privileges (including DNS setup!)
- Action:
- small PoC of how CNCF could manage as code their own stuffs
- what about doing this as a PoC: https://github.com/flatcar/flatcar-linux-infra/issues/316 ?
- beta.release.flatcar-linux.net is live
- bincache / misc folder
- call w/ CNCF about cloudflare. TL; DR: we asked to promote the current Flatcar CF account to a sponsored one
- Nebraska
- AWS pg 13 is deprecated in Feb.
- move to full kubernetes setup w/o bitnami setup?
- storage is still growing on the AWS database but that's slow but needs to be monitored

Todos:
- [x] we can ask to Hetzner for ARM64 instances :arrow_down:

### Hetzner sponsoring request

Hello,

My name is Mathieu and I work as a Software Engineer inside Microsoft on a CNCF[^1] project called "Flatcar" [^2]. Flatcar is an open-source Linux based operating system (OS) designed to run container workloads (Kubernetes or not).

Following Equinix Metal sunsetting announcement [^3], the Flatcar infrastructure used for serving releases and running tests is currently under migration and most of the services are migrated to a CNCF sponsored provider: Akamai/Linode.
To guarantee the stability of Flatcar users workloads, we run integration tests every night, for some PRs, and before each release on the supported architectures and supported providers (including Hetzner!). While most of the resources are covered by the CNCF provider, we still miss some resources for ARM64 testing hence this request.

Providing access to ARM64 resources for Flatcar testing would be another great opportunity for Hetzner to contribute to the open-source ecosystem and to add a new collaboration with Flatcar to the list of existing ones. In the past, the Flatcar team and Hetzner have successfully collaborated on various topics:
* an engineering implementation of Flatcar support for Hetzner
* a talk in Germany to present the Flatcar / Hetzner collaboration [^4]
* a CI sponsoring to run Flatcar Hetzner tests on short-lived Hetzner VM instances

Now for the technical details, to run ARM64 QEMU VM tests we currently have two ARM64 instances (`c3.large.arm64`) running on Equinix Metal with the following spec:
* github-runner-arm64
* CPU: Ampere Altra Q80-30 Processor 80-Core @ 3.00GHz
* Memory: 256GB RAM
* Storage: 2x 960GB NVME
* Rx: 620 GB/month
* Tx: 3 GB/month
* bld-armstrong
* CPU: Ampere Altra Q80-30 Processor 80-Core @ 3.00GHz
* Memory: 256GB RAM
* Storage: 2x 960GB NVME
* Rx: 2.48 **TB**/month
* Tx: 29 GB/month

So a single RX220 offering[^5] (merging both servers to save cost) seems to be the best match to replace those two instances because we need virtualization support to run QEMU VMs.

The Flatcar maintainers and the Flatcar community are already grateful for all the Hetzner collaboration provided until now and we are now hoping and looking forward to start a new chapter with you.

Thanks and have a great day,

Mathieu Tortuyaux (@tormath1) from the Flatcar maintainers team.

[^1]: https://www.cncf.io/
[^2]: https://flatcar.org
[^3]: https://docs.equinix.com/metal/#sunsetting-equinix-metal
[^4]: https://media.ccc.de/v/froscon2024-3038-flatcar_linux_what_s_new_in_this_container_os
[^5]: https://www.hetzner.com/dedicated-rootserver/rx220/

## Thursday 23 Oct. 2025

Attendees:
- Jan
- Sayan
- Mathieu

Notes:
- bincache:
- should we use a regular Bucket (CF / Object Storage from Akamai/Linode) ?
- should we use a block storage?
- arm64 workers for Jenkins: dedicated root servers on hetzner
- we can ask to Hetzner (200$/month)
- https://www.hetzner.com/dedicated-rootserver/#cpu_type=ARM

Todos:
- [x] we can ask to Hetzner for ARM64 instances

## Thursday 16 Oct. 2025 (offline)

Attendees:
- offline session!

Notes:
- Mathieu:
- Continued to work on Origin migration (https://github.com/flatcar/flatcar-linux-infra/pull/309)
- Now distributed on 3 regions:
- Europe: http://139-144-160-76.ip.linodeusercontent.com/
- North-America: http://172-237-140-171.ip.linodeusercontent.com/
- Asia: http://172-233-69-106.ip.linodeusercontent.com/
- Alpha, Beta, Stable and LTS backed-up on the CloudFlare Bucket
- Next steps / Currently on for the end of the week:
- TLS termination for the 3 endpoint via Caddy / Route53
- Update `alpha.release.flatcar-linux.net` to point to the new setup
- (Add node-exporter / metrics endpoint)
- With Kai we had a call with CNCF to setup public bucket domain name
- We opened a ticket to provide more details (CNCFSD-3035)
- Worst case scenario, we use the following domain already setup: `flatcar.cdn.cncf.io`
- Question for Thilo:
- Do you know where are the sources of the Caddy image used by current Origin servers? The one on quay.io: quay.io/flatcar/caddy:2.3.0
- Thilo: There's nothing special with that image afaict, we should just use upstream caddy from dockerhub?
- Mathieu: I think we built a special image to add the Route53 DNS plugin for cert renewal.

- Jan:
- [The Safe Settings App Deployment](https://github.com/flatcar/flatcar-linux-infra/pull/312#issuecomment-3398528659):
- The question here is what domain should we associate with our apps? And also do we have a preffered way of setting up a `Load Balancer` in the `misc/` module? As all redeployments on Linode end up changing the original `IPv4 address` the `Load Balancer` is needed, as even if we add `SSL` certs, they will keep changing the address, and thus the `IP` to which the domain should point to
- [Jenkins workers migration](https://github.com/flatcar/flatcar-linux-infra/pull/317), its still work in progress, but this is the problem so far:
- As far as I know Linode does not offer `ARM64` Nodes/VMs. What would we like to do in this case? Emulate...? Or would we dploy our `ARM64` workers elsewhere?
- [Add Secure VM Access and Linode service account/token](https://github.com/flatcar/flatcar-linux-infra/pull/318):
- Here I just wanted to ask for some more feedback, if you guys have any more ideas or question :)
- Thilo:
- Worked on Jitsi IAC: https://github.com/flatcar/flatcar-linux-infra/pull/319
- Most things work, Jitsi provisions successfully
- Includes a workaround for disk resize as Jitsi fails to provision on non-resized disks. Basically the automation will reboot the node once after it came up, then wait for it to become ready before sinishing.
- Includes an "automation ssh key" we can use to later modify the instance via IAC, e.g. by using remote file objects to manage passwords, ssh keys, etc.
- Worked with Jan to get a new ct provider release out (mostly Jan, I didn't do much). https://github.com/poseidon/terraform-provider-ct/pull/227
- That release would include support for `local:` file sources in Butane, which would simplify my Jitsi IAC.
- Currently investigating DNS updates on CloudFlate via IAC. infra@ account does not have sufficient privileges for API access to change DNS; working on it. Last piece missing for full Jitsi automation. Will also un-block https://github.com/flatcar/flatcar-linux-infra/issues/316.

Todos:
- [x] Check cloudflare DNS settings to see what's covered
- [ ] DNS cloudfare import to opentofu: https://github.com/flatcar/flatcar-linux-infra/issues/316

## Thursday 9 Oct. 2025

Attendees:
- Gabriel
- Jan
- Mathieu
- Thilo (was late)
- Kai
- Ervin

Notes:
- Jan: safe-settings PR (misc directory). CI does not work.
- Thilo: jitsi setup presentation + one more run before shutting down the instance + goal is to have on-demand
- IAM: how to control who can do what:
- For K8s -> github groups
- For the infra itself -> best effort through folder split (e.g origin, linode, misc, etc.)

Todos:
- [ ] Check cloudflare DNS settings to see what's covered
- [ ] DNS cloudfare import to opentofu
- [x] Lexi to have a look to escalate the CF bucket ticket
- [x] Have a look to the Jenkins access for Jan

## Thursday 2 Oct. 2025

Attendees:
- Mathieu
- Jan

Notes:
- Discussed about safe-settings PR
- Discussed about origins PR
- Discussed about CNCF ticket to get full control on the CF bucket
- Discussed about Jenkins worker migration
- Linode images might not be visible for user without full ownership

Todos:
- [ ] Lexi to have a look to escalate the CF bucket ticket
- [ ] Have a look to the Jenkins access for Jan