OSMF OPS meeting

# 2025-11-27 OSMF OPS meeting 27 November 2025, 19:00 London time, unless rescheduled [Time in your timezone](https://www.timeanddate.com/worldclock/fixedtime.html?msg=OSM+Foundation+OPS+meeting+-++Thursday+27+November+2025&iso=20251127T19&p1=136&ah=1) [Online calendar](https://framagenda.org/apps/calendar/p/fce4xrpFGx7fMxz8) [Subscription to future events](https://framagenda.org/remote.php/dav/public-calendars/fce4xrpFGx7fMxz8?export) Frequency of meetings: every two weeks, on Thursday at 19:00 London time, unless rescheduled. [Video room](https://osmvideo.cloud68.co/user/dor-x99-y3m) ## Participants * Paul * Grant * Tom * Héctor (board) * Alex (guest) * Minh (OSMF core facilitator) ----- ## New action items from this meeting * **Paul to notify the board that without their response to the OWG budget questions before mid-December the OWG can't provide their 2026 budget before the end of the year.** *[Topic: On OWG 2026 budget]* * **Grant to get back to all of the persons interested to help with new OSM Spyglass service by next Wednesday.** *[Topic: SotM Europe 2025 follow up - Looking for sysadmin help with new OSM Spyglass service]* ----- ### Alex - self-introduction Alex (guest) is an occasional contributor to OSM and has set up a Panoramax instance for Wales. Server: Hertzner Cloud S3 provider: Backblaze Might move from Backblaze to Mega, as when it does the blurring and makes the derivatives, they both count as downloads. Mega has more favourable download allowances. Comment: * API access seems the constraint, rather than speed. Otherwise, it might have made sense to add a proxy in front of any S3 providers, as there are performance gains. He was interested to see how the more professional management of servers happened. ----- ### On looking for sysadmin help with new OSM Spyglass service See: https://community.openstreetmap.org/t/looking-for-sysadmin-help-with-new-openstreetmap-spyglass-service/138487 * The OWG will help the person get on board. * There are several volunteers. ----- ### On hardware prices * 64GB DDR5 cost approximately GBP 1,000 * RAM and hard drive prices have increased a lot * Some refurbished disks might be reasonably priced, but their prices are not dropping ----- ### On OWG 2026 budget > Question from Paul to Grant whether the board has been responding, as he did not receive a reply on his emails about the OWG 2026 budget. Héctor (board) * Roland (Treasurer) has been busy. * The board will discuss it in their chat or meeting in two weeks or three weeks. **Action item: Paul to notify the board that without their response to the OWG budget questions before mid-December the OWG can't provide their 2026 budget before the end of the year.** * Concern expressed by Grant on creating a budget while the RAM prices are wildly changing. ** Most of the OWG budget is not going to be capital. ----- ## Reportage ### AWS Identity and Access Management Roles Anywhere * *Related to action item: [2025-10-16](https://hackmd.io/Pv21I7zsRnuZM595BD2Mzg/edit) Grant and Paul to set up a meeting about AWS Identity and Access Management Roles Anywhere https://docs.aws.amazon.com/rolesanywhere/latest/userguide/introduction.html. [Topic: AWS CA cert]* There was a discussion but the next steps were not defined. Grant would like to speak to someone about AWS security but hasn't found someone. He had posted on Mastodon about discussing this during the State of the Map Europe 2025 conference, but this constrained it. Action item: Grant to post another message on Mastodon and potentially on community.osm.orh about discussing AWS security. He is most interested in Roles. ----- ### Test Kitchen related refactoring * *Related to action item: [2025-10-16](https://hackmd.io/Pv21I7zsRnuZM595BD2Mzg/edit) Grant to create a PR regarding refactoring some stuff. [Topic: Reworking of Test Kitchen methods for defining which jobs run on Test Kitchen Github actions]* In PR. ----- ### Nominatim: purchase of second-hand server * *Related to action item: [2025-10-02](https://hackmd.io/frQWtzX8SQCaa0hvP7tW6A/edit) Grant to go ahead with the purchase of the Gen10 (second-hand) server for Nominatim in the US. [Topic: Gen10 Nominatim purchase (USA)]* Invoice request sent to the provider just before this meeting, to find out the approximate price. There is an invoice for Treasurer to pay, which might arrive tomorrow. The purchase is within OWG's budget. ----- ### Fastly frontend * *Related to action item: [2025-07-24](https://hackmd.io/LZwDRuy7QHeYWNvEQkeeYw/edit) Grant to set-up a test for OWG's review [Topic: Switching www.osm.org to Fastly frontend]* Grant wants to try faking the IP address header we send to the Fastly frontend, to be in the same format as Cloudflare currently sends. Suggestion: Change the Apache configuration to recognise the Fastly header, rather than the Cloudflare one. Grant tried to get every domain currently supported for Nominatim to be supported on Fastly and two domains give errors (see below). These domains have not been put on the config of the main site yet. Grant is waiting for a reply from Fastly. "Error: Domain 'openstreetmap.com' is already taken by another customer" "Error: Domain 'openstreetmaps.org' is already taken by another customer" ----- ### Mailman 2 to 3 conversion. * *Related to action item: [2025-07-24](https://hackmd.io/LZwDRuy7QHeYWNvEQkeeYw/edit) Grant to do the Mailman 2 to 3 conversion [Topic: Mailing lists] - https://github.com/openstreetmap/operations/issues/1264* In a ticket. Action item to be removed. ----- ### USA University offers * *Related to action item: [2025-05-01](https://hackmd.io/XM71u-YHS6WYugFA02lotw?edit) Grant to see if other USA University offers are still available and what hardware would be required. [Topic: OSUOSL funding / issues] #2025-09-18 parked* Action item to be removed. ----- ### Potential South African hardware donation * *Related to action item: [2025-03-20](https://hackmd.io/G2QZRHpkRyOQG_DC-Je2Tw/edit) Grant to follow-up with the South African contact about the potential hardware donation from a mobile network. [Topic: New offers of Servers Australia and South Africa] #2025-09-18 parked* Low priority on the South African contact's to-do list and our to-do list. Grant needs to keep the communication. Action item will be removed from this list. ----- ### Identifying more email providers used by spammers * *Related to action item: [2025-03-20](https://hackmd.io/G2QZRHpkRyOQG_DC-Je2Tw/edit) Grant to run an SQL query to identify more email providers used by spammers. [Topic: Spam] #2025-05-01 Grant has created a small list now disposable email providers. #2025-09-18 parked* Grant has an SQL query which is reasonable good at finding the actual spammers from OSM deleted accounts. ---- # Agenda ## Wiki capacity planning Server: Konqi Links shared: * https://prometheus.openstreetmap.org/d/Ea3IUVtMz/host-overview?orgId=1&from=now-1y&to=now&timezone=utc&var-instance=konqi&refresh=1m * https://prometheus.openstreetmap.org/d/lMxPYkt7k/filesystems?orgId=1&from=now-1y&to=now&timezone=utc&var-instance=konqi&var-mountpoint=$__all&refresh=1m CPU pressure: 0.09 % Memory pressure: 0 I/O pressure: 0.0002 The machine is not at capacity. Suggestions * Put Varnish or Fastly in front of the wiki, to improve performance of some pages, such as Map Features (which has many nested templates), or other pages with massive templated logic. ** Varnish makes it faster for logged out users - for logged-in users, it's uncached. Space * Growth looks mostly flat, with some bumps, probably due to failures of backup script (there's a ticket for it). * Space is probably not a constraint. No capacity issues for the next ~ 3 years. Any issues will be due to feature changes, rather than growth. ### Map features Issues with MapFeatures page * **Wiki Commons images**: The main issue causing the page to not load was the linked images from Wiki Commons, as we get rate-limited by Wiki Commons.The PHP worker would hang, while waiting for WIki Commons to respond, and would hit time-out. Whenever the cache was clear (a common occurance), it would wait idling for the Wiki Commons to never respond. * **Caching**: We create a dump of the wiki - it would go through every item of the site, including the history, and would create cache entries for those. This would cause the cache to expire.. Grant changed the cache purging from daily to weekly. ** The language bar might be interfering with the caching, as it is using a hack to localise itself. Currently, the page performance is reasonable, unless you hit it when a cache purge happens on a Sunday. Suggestions * Make the Map features page shorter. * Reach out to people at Wikimedia, to fix the limits on our account. Other points mentioned during discussion * Minh is talking with Mateusz, who is interested in creating a website for dedicated iD tagging schema, which might take some pressure off that page. *Hector left 28' after start.* ----- ## SotM Europe 2025 follow up ### Looking for sysadmin help with new OSM Spyglass service 6-8 interested people. Some junior but eager, some senior in the OSM space. Suggestion * Grant to create an outline of what is required and end goals. Plan * Grant to get back to all of the interested persons by next Wednesday. * Select two people and suggest to them to work together for Spyglass. Suggest other tasks for the rest of the people - Grant to offer mentoring for PRs. Other points mentioned during discussion * Desperately need more people, but we can't handle 8 doing one task. * Working on Mailman 2 to 3 conversion was rejected as a potential task, as it contains private data. Selection * Paul and Grant to select the people. Tom and Minh are also welcome to get involved. **Action item: Grant to get back to all of the persons interested to help with new OSM Spyglass service by next Wednesday.** ### Postgresql consultant During SotM Europe 2025, Jochen Topf asked Grant if the OWG had any questions for the Postgresql consultant. Grant lacked the background, as he had not been included in the related email chain, however he came up with a list of questions, which he will share with the OPS for review, before sending them to the consultant (it's imited hours of consultancy time). ---- ## Any other business ### Tom and Pablo Brasero are working on spam Tom and Pablo Brasero are working on spam. Other points mentioned during discussion * Pablo also talked with DWG members, who have some ideas for things he could work on. Minh to work with Pablo, to make sure the discussions stay on track. ### MapLibre discussion Grant wants to change some websites with his imagery (e.g. https://aerial.openstreetmap.org.za/) from using Leaflet to Maplibre. He wants to show the sheets that make up these sites, as an overlay. * Hard part is computing the sheet boundaries. ** Tom is already doing it for his site. #### Maplibre in the OSM Wiki Suggestions * Converst to Maplibre for maps served by the OSM wiki. ** There is currently a seperate template for Maplibre, for compatibility reasons, on the OSM wiki. We'll gradually move things over. * The issue of fully resolved urls in the style must be fixed. Other points mentioned during discussion * It would help to have a shortbread style to point at, without worrying about relative urls. * Would we allow anyone to point at the style, or restrict it to osm.org sites? ** Suggestion: Start from a restrictive core, add contact information and then expand. ### Vector changes to Fastly config Grant did some small Vector-related changes to the Fastly config. ----- ## Open Ops Tickets Review open, what needs policy and what needs someone to help with... https://github.com/openstreetmap/operations/issues https://github.com/orgs/openstreetmap/projects/1 https://github.com/orgs/openstreetmap/projects/1/views/2?filterQuery=-is%3Aclosed ## Action items * [2025-10-16](https://hackmd.io/Pv21I7zsRnuZM595BD2Mzg/edit) Grant and Paul to set up a meeting about AWS Identity and Access Management Roles Anywhere https://docs.aws.amazon.com/rolesanywhere/latest/userguide/introduction.html. [Topic: AWS CA cert] * [2025-10-16](https://hackmd.io/Pv21I7zsRnuZM595BD2Mzg/edit) Grant to create a PR regarding refactoring some stuff. [Topic: Reworking of Test Kitchen methods for defining which jobs run on Test Kitchen Github actions] * [2025-10-16](https://hackmd.io/Pv21I7zsRnuZM595BD2Mzg/edit) Grant to create a PR about adding logic to Chef for retrying failed initial creation of Let's Encrypt certificates [Topic: Add logic to Chef for retrying failed initial creation of Let's Encrypt certificates] * [2025-10-02](https://hackmd.io/frQWtzX8SQCaa0hvP7tW6A/edit) Grant to discuss with Paul Norman and flesh out his suggestion and determine the practicalities (e.g. key revocation). [Topic: AWS CA cert] * [2025-10-02](https://hackmd.io/frQWtzX8SQCaa0hvP7tW6A/edit) Grant to follow up with Paul. [Topic: Serving vector tile styles] * [2025-10-02](https://hackmd.io/frQWtzX8SQCaa0hvP7tW6A/edit) Grant to go ahead with the purchase of the Gen10 (second-hand) server for Nominatim in the US. [Topic: Gen10 Nominatim purchase (USA)] * [2025-09-18](https://hackmd.io/yDbLczVeSAWrLQbBFpxTzQ/edit) Paul to look at potential issues related to the collation of indexes - Debian Postgres upgrade. [Topic: OSM DB upgrade to Postgres 17] * Not public: [2025-09-18](https://hackmd.io/yDbLczVeSAWrLQbBFpxTzQ/edit) Grant to produce list of keys for Ops to review and rotate. [Topic: AWS Security] * [2025-07-24](https://hackmd.io/LZwDRuy7QHeYWNvEQkeeYw/edit) Grant to set-up a test for OWG's review [Topic: Switching www.osm.org to Fastly frontend] * [2025-03-20](https://hackmd.io/G2QZRHpkRyOQG_DC-Je2Tw/edit) Grant to run an SQL query to identify more email providers used by spammers. [Topic: Spam] #2025-05-01 Grant has created a small list now disposable email providers. 2025-09-18 parked ## OPS pads for 2025 meetings [2025-01-09](https://hackmd.io/LQexyX9iSSu8GG6JsYKyxA/edit) [2025-01-23](https://hackmd.io/sXaKftrrRNOQsOgNPF1grg/edit) [2025-02-06](https://hackmd.io/mpF7kUW3SBaiFV64KIxEoQ/edit) [2025-02-20](https://hackmd.io/1tldw0TbT8-fMcHrK3FZbQ/edit) [2025-03-06](https://hackmd.io/YbsJCyKMRji8xuSPqp5l4Q/edit) [2025-03-20](https://hackmd.io/G2QZRHpkRyOQG_DC-Je2Tw/edit) [2025-04-03](https://hackmd.io/_puAtHLWTtC0QY85S3f5Lw/edit) [2025-04-17](https://hackmd.io/WMJtb4XKRD6KwqiaJBleaw/edit) [2025-05-01](https://hackmd.io/XM71u-YHS6WYugFA02lotw/edit) [2025-05-15](https://hackmd.io/6Y1ERM5YTsK3W_npgEfCwA/edit) [2025-05-29](https://hackmd.io/W-vfoOMXT4GJHUWCgSEalg/edit) [2025-06-12](https://hackmd.io/auqIBdUBTmufr0J28ILUFA/edit) [2025-06-26](https://hackmd.io/RuUmi--vQyC9r8Dw9SV7QQ/edit) [2025-07-10](https://hackmd.io/FOnb4lVDTouSKquJ6N_Cug/edit) [2025-07-24](https://hackmd.io/LZwDRuy7QHeYWNvEQkeeYw/edit) [2025-08-07](https://hackmd.io/et_yASB0QsCp7VIQXA6e6w/edit) [2025-08-21](https://hackmd.io/NU41pU-hRxmCg_38_4VECQ/edit) [2025-09-04](https://hackmd.io/VBUSEJwzQPyInb7zXiCvYQ/edit) [2025-09-18](https://hackmd.io/yDbLczVeSAWrLQbBFpxTzQ/edit) [2025-10-02](https://hackmd.io/frQWtzX8SQCaa0hvP7tW6A/edit) [2025-10-16](https://hackmd.io/Pv21I7zsRnuZM595BD2Mzg/edit) [2025-10-30](https://hackmd.io/j2yqXiX4SReufR30qO8wMQ/edit) [2025-11-13](https://hackmd.io/YByQ6Lj3SaWzSa-JQQcbDQ/edit) [2025-11-27](https://hackmd.io/ux4TpF3_S6axy9x1PKwuew/edit) [2025-12-11](https://hackmd.io/7WV8uEE0T9y7iOVU7-rmkw/edit) [2025-12-25](https://hackmd.io/RkiREgvpQ-aJqPwiQ5gLLw/edit)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.