2026-01-27 - HackMD

--- tags: meeting, infrastructure project: infrastructure ---  # 2026-01-27 [:movie_camera: Zoom Meeting Room](https://zoom.us/j/92454301214?pwd=aEVoUi9EanpaakN3L1ZxRlpDQk5Ddz09) [:calendar: Shared Calendar](https://jenkins.io/event-calendar/) [:speech_balloon: Chat Room #jenkins-infra](https://matrix.to/#/#jenkins-infra:matrix.org) [:email: Google Group (mailing list) jenkins-infra](https://groups.google.com/g/jenkins-infra) [🧠 Previous Meeting Notes](https://github.com/jenkins-infra/documentation/blob/main/meetings/2026-01-22.md) ## Attendees 👥  * @dduportal (Damien Duportal) * @MarkEWaite (Mark Waite) * @hervelemeur (Hervé Le Meur) ## Announcements :loudspeaker: 1. Jenkins Weekly Releases * This Week: 2.548 - https://matrix.to/#/!JlkqzpdEnsUUuVtjgE:matrix.org/$SMa_-J1dSqOD1LmSZUx0p6etmPlHEc_p_6IjA1yxWvc?via=matrix.org&via=gitter.im * Windows 2022 images failing due to Microsoft registry timeout, replayed job on only 2022 images 2. Announcements: * Team capacity: * FOSDEM incoming: expect lower availability * Jay is off starting tomorrow * Herve and Mark will be travelling to/from FOSDEM * Damien off for ~6 weeks on the first February, maybe earlier * Priorities: * Infra Roadmap: https://www.jenkins.io/project/roadmap/ * Current topics: * (not in roadmap, no need to) ci.jenkins.io => cloud billing and stability * Windows 2025/2022/2019 * JDK25 as runtime for our controllers * Get rid of Puppet (in favor of Ansible) * (add to roadmap + open helpdesk issue): Maven 4 RC/GA * [Maven 4 pre-release version preview in ci.jenkins.io](https://groups.google.com/g/jenkinsci-dev/c/0JEzgNEIg6U) mailing list thread * (add to roadmap): weekly.ci.jenkins.io as public live demo * [add real-world job on weekly.ci.jenkins.io](https://github.com/jenkins-infra/helpdesk/issues/4562) * Team meeting: cancelling 2026-02-03. Next one will be on 2026-02-10. ## Upcoming Calendar 📆 * Next infra meeting in two weeks * Next Weekly: 2026-02-03 - 2.549 * Next LTS: * 2026-02-04 - 2.541.2 - RC * 2026-02-18 - 2.541.2 * Next Security Release as per [jenkinsci-advisories](https://groups.google.com/g/jenkinsci-advisories): N.A. * Upcoming credentials expirations (~3 weeks): * 202-02-16: * [Issue to create] Netlify token on infra.ci.jenkins.io expires (Previous renewal: https://github.com/jenkins-infra/helpdesk/issues/4860) => Check if doable by Hervé * [Issue to create] NPM token on infra.ci.jenkins.io expires (Previous renewal: https://github.com/jenkins-infra/helpdesk/issues/4859) => Check if doable by Hervé * Next major event: * Contributor Summit + FOSDEM in Brussels, 30/31 Jan and 1st Feb * We have a booth for Jenkins at FOSDEM * Agenda for Summit is still WiP. Bring ideas (in community.jenkins.io) * https://community.jenkins.io/t/jenkins-contributor-summit-on-jan-30-2026-call-for-topics-and-ideas/35773 * Mark is confirming with people based on the prioritized list from the Jenkins Board ## Cloud Budgets * Azure (CDF paid) * October: $5.5k (invoice) * November: $5.7k (invoice) * December: $5.4k (invoice) * January: $4.8k (forecast at $5.8k) * Azure Sponsorship (Microsoft Credits): still stuck. * We see the $100k credits in the account but can't use them * Alex Brandes (NotMyFault) is on it (dealing with Microsoft Support and Partner program). Send him hugs. * DigitalOcean - Remaining $10,411.25 until January 02, 2027 (20 months left at current rate) * October: $788.81 (invoice) * November: $831.38 (invoice) * December: $454.60 (invoice, previously forecasted at $585) * January: $354.80 (forecast at $407, previously at $345) * AWS: * Sponsored account - $54,613.14 left (31 December 2025) until 31 May 2027=> :warning: 5 months remaining at this rate * October: $6.9k * November: $5.9k * December: $10.7 (previously forecasted at $9.5k way higher than usual) * Huge increase due to EC2 plugin bug * January: $13.4k (forecast at $15.3k, previously at $15.7k) * Still really high, ci.jenkins.io is once again at risk if we don't decrease the consumption or if we don't move it somewhere else before end of April 2026 * Need an issue ASAP for the EC2 agents GC (Damien) * Need an issue for packer image various resources cleanup * Mark opened an issue on the EC2 plugin: https://github.com/jenkinsci/ec2-plugin/issues/1990 * Request for sponsoring renewal sent * Jfrog Artifactory Usage * Storage: 1.54 Tb - (was 1.53TB last time) * Bandwidth: * October: 16.67 Tb * November: 15.9 Tb * December: 30.34Tb * A single IP address consumed 12Tb in December * JFrog didn't block this address until this month (unless this consumer changed its IP?) * January: 13.83 Tb (forecast at 15.9 Tb, previously at 16.5 Tb) * Sounds like the abuser stopped abusing ## Notes :book: * Done (Milestone [193](https://github.com/jenkins-infra/helpdesk/milestone/[193]?closed=1)) * Support * ci.jenkins.io: * [ci.jenkins.io is responding slowly](https://github.com/jenkins-infra/helpdesk/issues/4894) * [ci.jenkins.io is not allocating Kubernetes agents](https://github.com/jenkins-infra/helpdesk/issues/4970) * [ci.jenkins.io reports HTTP 502 instead of showing its pages](https://github.com/jenkins-infra/helpdesk/issues/4969) * [LTS 2.541.1 release](https://github.com/jenkins-infra/helpdesk/issues/4945) * [docs.jenkins.io nginx configuration leaks the nginx version](https://github.com/jenkins-infra/helpdesk/issues/4938) * [Make Kris Stern an admin for all docs-related Jenkins repos](https://github.com/jenkins-infra/helpdesk/issues/4936) * Keep infrastructure sane and maintainable * [Set up resource root URL on all Jenkins instances](https://github.com/jenkins-infra/helpdesk/issues/4959) * [Set up 2.539+ CSP protection on LTS controllers, remove `csp` plugin and associated configuration](https://github.com/jenkins-infra/helpdesk/issues/4962) * [Set up 2.539+ CSP protection on Weekly controllers, remove `csp` plugin and associated configuration](https://github.com/jenkins-infra/helpdesk/issues/4881) * [Grant SSH access for `hlemeur` to the (Puppet managed) VMs](https://github.com/jenkins-infra/helpdesk/issues/4966) * Keep infrastructure up to date * [[infra.ci.jenkins.io] Track Digital Ocean PAT expiration (2026-01-28) used by terraform with updatecli](https://github.com/jenkins-infra/helpdesk/issues/4943) * [[private.vpn.jenkins.io] VPN CRL expires on 2026-06-01](https://github.com/jenkins-infra/helpdesk/issues/4951) * We learnt that revoking an user, it extends CRL expiration date * Was initially expiring on 2026-01-28 * An updatecli manifest will now open an informative PR before expiration * [Closed as not planned](https://github.com/jenkins-infra/helpdesk/milestone/193?closed=1): * support * [There has been issue while creating the jenkins account](https://github.com/jenkins-infra/helpdesk/issues/4964) * Work in Progress (Milestone [ID](https://github.com/jenkins-infra/helpdesk/milestone/[ID])) * Support * [[ci.jenkins.io] Windows build with JDK25 fails with unexpected errors (out of memory, etc.)](https://github.com/jenkins-infra/helpdesk/issues/4947) * Closeable (not seen again at first sight) => yes. * Is the Windows 2025 working better? * Maybe the new Oracle patch fixes it (not delivered yet, waiting for Temurin) * [Creating new key set fails on Windows agents - Error 5: Access is denied](https://github.com/jenkins-infra/helpdesk/issues/4939) * Main issue solved, fix in progress in the Docker test harness (due to no more administrator) * [decommission (or rename) docs.jenkins.io?](https://github.com/jenkins-infra/helpdesk/issues/4935) * Wip on the new hostname /service "alpha.docs.jenkins.io" * Next step: migrate production to it and decommision the old Fastly docs.jenkins.io * Note: "docs.jenkins.io" will still exist and redirect to alpha.docs after the migration * [Spot reclamation caused build failures in core build](https://github.com/jenkins-infra/helpdesk/issues/4906) * Delayed to stabilize ci.jenkins.io and its spending * [`qualys-cs` plugin bundles closed-source library](https://github.com/jenkins-infra/helpdesk/issues/4917) * Still waiting for Daniel. No infra action requested. * Moving out of milestone (+ ping Daniel privately as reminder) * [Bunch of bad links on https://updates.jenkins.io/ and https://get.jenkins.io](https://github.com/jenkins-infra/helpdesk/issues/4882) * Delayed, we can resume work on it. * WiP: understanding the root cause in the HTML/JS in the way jenkins components are setup. * Maybe require manual update, and if so, a corresponding runbook entry * [[incrementals.jenkins.io/ci.jenkins.io] Outage of incrementals due to `releasebot` user's API token reset](https://github.com/jenkins-infra/helpdesk/issues/4852) * Damien still need to reproduce * Operation to perform when deploying enforced auth. on ci.jenkins.io: remove all users one time (no more failed migration on startup) but require re-creating API tokens * Operation to describe (potential incremental service interruption, could put ci.jenkins.io in shutdown) * [`github-jenkinsci-permissions-report.json` hasn't been updated since Sep 12](https://github.com/jenkins-infra/helpdesk/issues/4826) * Delayed: no major issue,we want to use (in infra.ci) a VM agent for this job (non spot) instead of container agent. * Keep infrastructure sane and maintainable * [[(status|contributors|stats|docs|stories|cn|jenkins-io-components).jenkins.io, Websites PR previews] Netlify billing plan changed: evaluate and act](https://github.com/jenkins-infra/helpdesk/issues/4958) * No more emergency: their support said we don't need credit card but they need info from us (Damien to send it from Jenkins board answer he receieved earlier today) * Next step: move status.jenkins.io production out of Netlify to GH pages. * [[docker-openvpn] Track OpenVPN User Certificate Expiration with `updatecli`](https://github.com/jenkins-infra/helpdesk/issues/4932) * Jay is off, Damien and herve takes over the last step. * [[staging.pkg.origin.jenkins.io/staging.get.jenkins.io] Garbage collect the inactive branches](https://github.com/jenkins-infra/helpdesk/issues/4886) * On pause * [[get.jenkins.io,mirrors.updates.jenkins.io] Resume GeoIP database weekly update](https://github.com/jenkins-infra/helpdesk/issues/4878) * On pause * [Ensure the BOM cache filler runs successfully and is easier to use and monitor](https://github.com/jenkins-infra/helpdesk/issues/4877) * On pause * [Setup a job/set of jobs to allow performing maintenance operations (cron, GCs of resources, etc.)](https://github.com/jenkins-infra/helpdesk/issues/4876) * On pause * [[stats.jenkins.io/infra-statistics] Move "data for the usage stats site" generation (from anonymized data) out from Andrew machine](https://github.com/jenkins-infra/helpdesk/issues/4666) * On pause * Keep infrastructure up to date * [Switch controllers to use Windows 2025 agents by default](https://github.com/jenkins-infra/helpdesk/issues/4955) * One last step: infra acceptance tests change to check the Windows version, PR open, wip. * [[pkg.jenkins.io/release.jenkins.io] Certificate signing the MSI Jenkins package expires on 16 May 2026](https://github.com/jenkins-infra/helpdesk/issues/4923) * On pause * TODO (next milestone: https://github.com/jenkins-infra/helpdesk/milestone/194) * [[ci.jenkins.io] Require authentication for read access](https://github.com/jenkins-infra/helpdesk/issues/4971) * [Renovate on `jenkins-infra/plugin-modernizer-tool`](https://github.com/jenkins-infra/helpdesk/issues/4968) * [Optimise cost and maintainance by merging Windows 2022 and Windows 2025 templates](https://github.com/jenkins-infra/helpdesk/issues/4956)  ## Action Points :muscle:  * [x] Publish this meeting record on Youtube: https://youtu.be/4A8OvrUIvBg * [x] Publish a topic in community.jenkins.io with the tag `meeting`: https://community.jenkins.io/t/infrastructure-team-meeting-january-27-2026/36108 * [x] Publish this meeting note on [jenkins-infra/documentation](https://github.com/jenkins-infra/documentation) Next meeting note - [YYYY-MM-DD](https://github.com/jenkins-infra/documentation/blob/main/meetings/YYYY-MM-DD.md)