2026-03-19 OSMF OPS meeting

# 2026-03-19 OSMF OPS meeting 19 March 2026, 19:00 London time, unless rescheduled [Time in your timezone](https://www.timeanddate.com/worldclock/fixedtime.html?msg=OSM+Foundation+OPS+meeting+-++Thursday+19+March+2026&iso=20260319T19&p1=136&ah=1) [Countdown](https://www.timeanddate.com/countdown/generic?p0=136&iso=&msg=OSM%20Foundation%20OPS%20meeting%20-%20%20Thursday%2020260319T19) [Online calendar](https://framagenda.org/apps/calendar/p/fce4xrpFGx7fMxz8) Subscription to future events: See instructions further below Frequency of meetings: every two weeks, on Thursday at 19:00 London time, unless rescheduled. [Video room](https://osmvideo.cloud68.co/user/dor-x99-y3m) ## Participants * Craig * Paul * Grant * Tom * Héctor ## New action items from this meeting * Paul to create a breakdown of QGIS tile traffic statistics for different zoom levels. [Topic: QGIS Tiles usage] * Grant to research what triggers a large download from QGIS. [Topic: QGIS Tiles usage] * Paul to overhaul how we're doing the 404 tiles. [Topic: QGIS Tiles usage] * Paul to look into the typo on tile block message 403r [Topic: AOB - Typo on tile block message 403r?] * Paul and Grant will run some time limited experiments during non peak hours to test catching anonymous/fake-ua scrapers. Genuine Google Bot etc will continue to be permitted. [Topic: Fastly Client Challenges] ----- ## Reportage ### Mailman conversion * [2026-03-05](https://hackmd.io/XHac49e5RTSmWc0sPn6S_g) Grant to do a dry run for the Mailman conversion, probably on Rhaegal in Croatia. [Topic: Upgrades: Machines on Ubuntu 22.04] in progress ### 2026 OWG Budget * [2026-02-05](https://hackmd.io/AOrSFcCkSwmhk9HU-zT5-g) [2026 OWG Budget] OWG to work out what is needed in 2026, and see if budget adjustments are required. Will come back to the board. Ongoing planning. No need to track seperately. ### MediaWiki * [2026-02-05](https://hackmd.io/AOrSFcCkSwmhk9HU-zT5-g) Grant to test some MediaWiki settings to improve size selection. [Topic: WikiCommons image resize] Done and issue seems resolved. The current LTS MediaWiki version does not have the features we want for image sizing. We have turned on the options that allow us to do some resizing, but there are some edge cases. The ultimate fix is moving to a new long-term supported version of MediaWiki. We still get occasionally rate-limited by MediaWiki Commons. Mediawiki-related new errors * One related to .pdf handling * A wikibase one - needs fix by Yuri. ----- # Agenda ## QGIS Tiles usage We will produce some stats (tiles per zoom, peak rate, and tile usage heat-map) We are likely to turn on TOTP validation for osm.org requests, might need extra set-cookie on osm.org ------ Issue: High use of OSMF tiles by QGIS. * It is unclear why the traffic is so high and there are limited things we can do. * We would like to cut traffic to 1/4-1/2. ### Background QGIS * uses the OSMF tileservers. * has removed OSMF tiles from the browse tile layers, but on startup it asks the user whether they want to start from a template and the template is an OSM base layer, with OSMF tiles. * supports arbitrary zooms ** if someone zooms out just before they switch to another tile layer zoom, they can have 128 pixel tiles, which can fill a 4K screen. While this is not a load issue, it means we can't put any very effective low rate limiting. * does some oversampling by downloading a higher zoom level than it actually needs. ** the set-up is good for imagery, and bad for raster maps. ** could be only for high-definition monitors. On traffic * Issue: The QGIS traffic peak during a European day is 2 to 4 times more than osm.org traffic, looking at zoom levels 13 and above. * Daily average: QGIS is using more tiles than osm.org. Traffic could be caused by * Export function * Plugins No header difference between the two cases, so we can't rate limit on the CDN. We might not be able to figure out the reason for the high traffic. On export functionThe export to print tries to download tiles at 300 DPI. ### Suggestions * One QGIS tile server: Send all the QGIS traffic to one tile server, and let the server become overloaded. ** Sarah Hoffmann does something similar with Nominatim. ** Have to send to 2 servers. ** Present a reasonable image tile error message to the users. * Rate-limit the tiles - we already doing that - we can't do it for anything over 4 kilobytes. * Rate-limit on the backend. * Create a breakdown of zoom-level statistics and 1) present the case to them and options they have or 2) remove a few zoom layers for QGIS. * Peak hours: do not allow download of tiles from high zoom levels and display a tile message "we don't have capacity" to the users. On suggesting to QGIS to switch to vector tiles * Vector tiles would help because they're bigger area tiles, particularly when you get into over zoom. * They could download the tiles from Geofabrik. * The stylesheet support was not adequate, but this must be fixed by now. * Issues: ** Time: If they release a new version, it would take 2 years for upgrade for most users. ** Hosting the style: It would be nice if QGIS hosts it, as they could change it as needed. On using Fail2Ban * Fail2Ban might be able to pick up multiple requests for tiles, if we scoped it very carefully. * We would have to take into account capacity weighted distribution among the five European tile servers. I.e. if someone gets two meta tiles on the same server, their load is going to be significantly higher than someone who gets meta tiles on different servers. We would have to set different limits for each server, in proportion to how much of the traffic they're serving. On asking QGIS to buy a server * We don't give people that privilege - but they might be the exception to the rule. * Price: ** ~ EUR 7K for a general purpose machine, similar to what we have. ** The cost is about 70% more than the machines that we bought. ** Prices are very high right now. * Would need 2 general purpose machines to support QGIS. Better to get something newer. ### Other points mentioned during discussion * QGIS financially supports OSM as a Silver OSMF Corporate Member. * It would help if OSMF tiles weren't the only option by default. On urgency * They probably don't realise yet how urgent the situation is. * They can't do anything urgent, due to people upgrading very slowly. On tarpiting: * Applies to objects under 4 kilobytes (so, mostly empty tiles). * Delays transmission of a set of bytes by an integer number of seconds. Nominatim: * One queue for preferred people. * One for everyone else. ### Decision * Create a breakdown of zoom-level statistics, a heatmap of what is accessed and present the case to QGIS and the options they have. * If the QGIS high tile traffic starts to cause significant harm to other people, we can change the directors to send the QGIS traffic to the Polish tile server. ### Action items * Paul to create a breakdown of QGIS tile traffic statistics for different zoom levels. * Grant to research what triggers a large download from QGIS. * Paul to overhaul how we're doing the 404 tiles ## TOTP cookies for access control Paul started work towards being able to use the TOTP cookie. It's easy to change the website to set that cookie, which we send anyway. * Fastly gave us a code block, which should work. * Paul tried to do a validation by checking the TOTP cookie's presence, but 1) not all website pages set the cookie and 2) unclear if MapLibre requests send the cookie to tile.osm.org. On several people getting tile access blocked notifications, after the recent referrer-related changes Potential reasons * using a privacy-related browser extension, like uMatrix ** The default in umatrix is spoof-referrer, which will send tile.openstreetmap.org as the referrer and will lead to blocks. * having the website security header turned on * other browser extensions or using privacy mode. * overridding your default browser accept headers Suggestions * Put a parameter on the request tile URL. * Change the website to set the cookie to any pages which need to access Overpass, as Overpass checks the cookie. * OWG to set an acceptable level of false positives. * OWG to document what people can do to fix the blocked tile access. * OWG can set TOTP so that when people visit osm.org in the last TOTP duration, this will carry over to other OSM-related sites. Discussion on the following case: Someone visits example.com, which had blocked access to OSMF tiles. They then visit osm.org, see the tiles and get a TOTP cookie, and go back to example.com * If they view the same area, it will always work because successful tiles are cacheable and they will be in the browser's cache. * Depends on our configuration: ** If the OWG explicitly blocks example.com, it would not work. ** If we are serving stale tiles to example.com, we might not want to serve stale tiles to viewers from example.com who have also visited osm.org. Other points mentioned during discussion * Some of the third-party layers (e.g. Tracestrack Topo) that we have don't work, because they insist on having OSM.org as the referrer. * The OWG needs to know who is using the tiles and can't cater for every single case. * The TOTP currently rolls over once an hour - there is a hard setting in the code. Action item: Paul to make a PR on the website to send the TOTP to any page which has a map. ''Paul was thanked for his changes to the error tiles, which now have codes.'' ## Fastly Client Challenges We can add a challenge to osm.org to prove those who access it are humans. Might help with scraping. Not to be used for the API. On robots.txt * it was recently updated by the OWG. * it is mostly ignored by AI and modern scrapers - unless the scraper is explicitly named. On scrapers * the "openAI scraper" traffic we see does not come from openAI's published IPs - so it could be fake. * OWG sees also a lot of chatGPT traffic which is probably not from chatGPT. Suggestions * Leverage Fastly's tags, such as "suspected bots" and "official bots". * Add trackpoint rate-limiting on 408s. * Add a restriction so that chatGPT can browse only from official chatGPT IPs. * Put a trackpoint rate-limiting on 408s. On spam * 150 accounts/day created from Pakistan, related to spam. * 200 signups within a few minutes with different email addresses. * The OWG thinks that the sign-ups are being created by people trying to hack people's mailboxes. * Some spammers report the emails from us as spam. Other points mentioned during discussion * Fastly identifies "suspected bots", likely to be faking its user agents, and "official bots". ** The "suspected bots" tag is not useful for the API. * It won't help with the trackpoints scrapers. **Action item: Paul and Grant will run some time limited experiments during non peak hours to test catching anonymous/fake-ua scrapers. Genuine Google Bot etc will continue to be permitted.** ## AOB - Typo on tile block message 403r? [by Dorothea] https://cdn.masto.host/enosmtown/cache/media_attachments/files/116/256/790/787/629/574/original/a9cfec7f5f01878e.png "of to" * The tile message is manually line broken. * It is difficult to fit in the right number of words into a size-minimised tile, which then has to be small enough to get base64 encoded into our configuration. **Action item: Paul to look into the typo on tile block message 403r.** ## AOB - funding a second sysadmin [by Craig] The board during the February board meeting decided to try and fund a second systems operator and get that person in place as soon as possible. Process * undefined * long. It will probably take 9 months to hire someone. Might see someone in Nov/Dec. * a lot of consultations with OWG expected to take place to figure out what will be needed. Budget * we have funds which could probably fund 1/4 of a year. * we have to raise EUR 200,000 to make the budget balance this year- so pushing for fundraising. Position Unclear if this is going to be a contractor or employee. OPS comments * Paul is looking into contracting opportunities. * Immediate temporary contract: Paul could be contracted, and OSMF in parallel have the formal bureaucratic process of long-term hiring. ## AOB - BTC Background at https://osmfoundation.org/wiki/Board/Minutes/2026-02#Creation_of_OSMF_account_on_BTC_exchange_service_for_BTC_donations Topic raised by Grant, who asked Héctor (board) whether there was an update on his research for an BTC exchange service where OSMF could create a business account. Héctor has contacted some companies regarding their account rates, but non have answered so far. * OSMF has to cash out a 2 [[Donate/Bitcoin|BTC]] donation ()[https://community.openstreetmap.org/t/thank-you-for-2-bitcoin-donation-to-openstreetmap/139836/1 1], [https://www.blockchain.com/explorer/transactions/btc/ddd0a9a868e2846efaa9667107a48f6a6c1a921e935355d5f867e7d5a8155c9c 2]) * OSMF needs to get its own account on a BTC exchange service (note: see the [[Board/Minutes/2026-02#Creation_of_OSMF_account_on_BTC_exchange_service_for_BTC_donations|2026-02-26 board disccusion]]). Otherwise, Grant would have a tax issue. BTC down 40% over 6 months. * 75000 EUR in Jan * 60000 EUR now. On converting BTC to fiat * We have been typically converting BTC donations immediately. * Most NGOs cash out immediately. * For small BTC donations, we wait. Other points mentioned during discussion * Risk of handling BTC is more. * Coinbase does not do business accounts and have 40% tax. ## Open Ops Tickets Review open, what needs policy and what needs someone to help with https://github.com/openstreetmap/operations/issues https://github.com/orgs/openstreetmap/projects/1 https://github.com/orgs/openstreetmap/projects/1/views/2?filterQuery=-is%3Aclosed ## Action items * [2026-03-05](https://hackmd.io/XHac49e5RTSmWc0sPn6S_g) Grant to do a dry run for the Mailman conversion, probably on Rhaegal in Croatia. [Topic: Upgrades: Machines on Ubuntu 22.04] * ~~[2026-02-05](https://hackmd.io/AOrSFcCkSwmhk9HU-zT5-g) [2026 OWG Budget] OWG to work out what is needed in 2026, and see if budget adjustments are required. Will come back to the board.~~ * ~~[2026-02-05](https://hackmd.io/AOrSFcCkSwmhk9HU-zT5-g) Grant to test some mediawiki settings to improve size selection. [Topic: WikiCommons image resize]~~ * [2026-01-22](https://hackmd.io/3f65lQYMRvmK2poYRDdlRA) Grant to get AWS S3 bucket credentials for the dev server. [Topic: Credativ consultancy on OSM.org Postgres database update] * [2026-01-22](https://hackmd.io/3f65lQYMRvmK2poYRDdlRA) Tom to draft follow up question on pgbackrest local backup required or can /JUST/ S3 be used. [Topic: Credativ consultancy on OSM.org Postgres database update] * [2025-10-16](https://hackmd.io/Pv21I7zsRnuZM595BD2Mzg/edit) Grant and Paul to set up a meeting about AWS Identity and Access Management Roles Anywhere https://docs.aws.amazon.com/rolesanywhere/latest/userguide/introduction.html. [Topic: AWS CA cert] * [2025-10-16](https://hackmd.io/Pv21I7zsRnuZM595BD2Mzg/edit) Grant to create a PR regarding refactoring some stuff. [Topic: Reworking of Test Kitchen methods for defining which jobs run on Test Kitchen Github actions] * [2025-10-16](https://hackmd.io/Pv21I7zsRnuZM595BD2Mzg/edit) Grant to create a PR about adding logic to Chef for retrying failed initial creation of Let's Encrypt certificates [Topic: Add logic to Chef for retrying failed initial creation of Let's Encrypt certificates] ## Automatic addition of OPS meetings to your calendar If you want future OPS meetings to be automatically added to your calendar, you can subscribe to the following iCalendar link. Instructions on how to do that depends on which calendar software/service you use. The link is provided for your convenience. Subscription link (do not download and import the file): https://framagenda.org/remote.php/dav/public-calendars/fce4xrpFGx7fMxz8?export Instructions for two services: Mailbox.org calendar * Open the Mailbox Calendar * Left panel: go to "Add new Calendar" * Select "Subscribe via URL (iCal) * Paste the URL https://framagenda.org/remote.php/dav/public-calendars/fce4xrpFGx7fMxz8?export Google calendar * Open your Google calendar * Left panel: go to "other calendars". * Press the "+" sign. * Select "from URL". * Paste the URL: https://framagenda.org/remote.php/dav/public-calendars/fce4xrpFGx7fMxz8?export * Press "Add calendar". Please note that: * The link might not work when Framasoft servers experience problems. * You might get delayed updates, depending on how often the calendar service from your side syncs/checks for updates. ## OPS pads for 2026 meetings [2026-01-08](https://hackmd.io/3KyLac85RT68-jV-_18ubQ) [2026-01-22](https://hackmd.io/3f65lQYMRvmK2poYRDdlRA) [2026-02-05](https://hackmd.io/AOrSFcCkSwmhk9HU-zT5-g) [2026-02-19](https://hackmd.io/Vl-K9p_iRduqxjtBtsYojQ) [2026-03-05](https://hackmd.io/XHac49e5RTSmWc0sPn6S_g) [2026-03-19](https://hackmd.io/J7vD7BUYScmhBFuN7RTOfA) [2026-04-02](https://hackmd.io/3NBzjklSS4yLj7ZmSHNw1A) [2026-04-16](https://hackmd.io/13g-1QlDQTKsWTtPa3BXSQ) [2026-04-30](https://hackmd.io/ruGMkFaaTeCUu-6WDxrEHg) [2026-05-14](https://hackmd.io/rlwfBOQoS9uNuTnQMRAj5Q) [2026-05-28](https://hackmd.io/RGLmQ3LyQASsgF2zhNoxHQ) [2026-06-11](https://hackmd.io/5zOKJ1YqTU6zihkfIswB2A) [2026-06-25](https://hackmd.io/aCi9MajCQRCLykT7nkvaGA) [2026-07-09](https://hackmd.io/In2XmJBVTkuu4vLR0I-_kg) [2026-07-23](https://hackmd.io/dPFH_ablQrGFjskZlGrnwA) [2026-08-06](https://hackmd.io/PhhE8m1-QLCcibWF1igwSw) [2026-08-20](https://hackmd.io/EoQnvXgwRoGmo2Bmz7HIPg) [2026-09-03](https://hackmd.io/vtR_FKAxSpuPebkbZdQxrw) [2026-09-17](https://hackmd.io/WBZ1kh2dSiaMVBi3f28lcQ) [2026-10-01](https://hackmd.io/mJAYPt6uTHei23pPQo199g) [2026-10-15](https://hackmd.io/NcmUFvCoSWeL6QGNj4BzRg) [2026-10-29](https://hackmd.io/pl4Ek4JHQ0aDLB-pm3uhWQ) [2026-11-12](https://hackmd.io/3PTYFNMDQ9S9WbNKpcdmqg) [2026-11-26](https://hackmd.io/6zea95cxTa-Ex6NToeIGLA) [2026-12-10](https://hackmd.io/WUCLnWZwS4CtAN7Z0j5Mlg) [2026-12-24](https://hackmd.io/qoMD_Ig0Rde-dWtH_0fIzQ)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.