# OSMF OPS meeting

Thursday 30 May 2024, 19:00 London time, unless rescheduled
[Time in your timezone](https://www.timeanddate.com/worldclock/fixedtime.html?msg=OSM+Foundation+OPS+meeting+-++Thursday+30+May+2024&iso=20240530T19&p1=136&ah=1)
[Countdown](https://www.timeanddate.com/countdown/generic?p0=136&iso=20240530T19&msg=OSM%20Foundation%20OPS%20meeting%20-%20%20Thursday%2030%20May%202024)
[Online calendar](https://framagenda.org/apps/calendar/p/fce4xrpFGx7fMxz8)
[Subscription to future events](https://framagenda.org/remote.php/dav/public-calendars/fce4xrpFGx7fMxz8?export)
Frequency of meetings: every two weeks, on Thursday at 19:00 London time, unless rescheduled.
[Video room](https://osmvideo.cloud68.co/user/dor-x99-y3m)

## Participants
* Tom
* Grant
* Dorothea
* Guillaume (14' after start)

## New action items from this meeting
* **Grant to revert the request timeouts. [Topic: Discussion of false positives hitting DDOS protections due to request timeouts hitting max allowed and DDOS protection.]**
* **Tom to amend the draft editor policy document and move it from Gdoc to hack.md [Topic: Editor Policy]**
* **OPS to add the SDRP requirement to the Editor Policy draft and see what feedback we receive. [Topic: Editor Policy]**
* **Grant to download a back-up of Fastly's VCL, every month. [Topic: Fastly future projects]**

## Reportage

### UCL visit
Official process for taking goods out is via the goods in/goods out. A form has to be filled and the requirements are:
* there needs to be a ticket for it.
* only allowed to do it during business hours.

Grant has access to the ticketing system and can create a support ticket, which is probably different from the type of ticket that UCL needs. Grant will follow up with a person from UCL.

## Agenda
### Discussion of false positives hitting DDOS protections due to request timeouts hitting max allowed and DDOS protection.
* https://github.com/openstreetmap/chef/commit/ab4b82d3ca1eb1be198b331203f3505c4a043b59

Issue affecting few different people, including people in Bloomberg.

Options
* Be more lenient on negotation time.
* Require more of them before we trigger the block,
* Both.

Changes by Tom
* December: fail2ban block on the timeouts. This was implemented because all the Apache slots were tied up by connections that weren't doing anything.
* January: reduced timeouts.

Tom's changes of negotation time resulted in peaks during the day to be significantly higher. There are also more peaks during weekdays than at weekends. The body was not changed, as people uploading GPS traces might take some time.

Other points mentioned
* Apache provides only absolute timeouts, while what we need is the timeout since it last sent any data.
* We can extend the maximum time but keep the minimum time the same.
* Reduced time-outs in January.

Suggestions
* Revert the changes.
* Also change the number of incidents at the same time.
** It might be better to do one change at a time.

#### Decisions:
* We will revert the timeouts
* Will not modify fail2ban until we have feedback from the change.

#### Action item: Grant to revert the request timeouts.

### Editor Policy
* We need to finish the draft amends.
* Tom will amend the document later and move to HackMD.
* Should editors be required to join the https://wiki.openstreetmap.org/wiki/Software_dispute_resolution_panel
- We will add a SDRP requirement to the Editor Policy draft and see what feedback we receive.

Related to action item: [2024-05-16](https://hackmd.io/XFtol8teRoinL34n7h2bcA) Paul to pull the proposed changes for the editor policy from the OPS minutes into the ticket. [Reportage: Editor policy]

* OPS were contacted by some people, including from Rapid editor (Meta), about the editor policy.
* Mikel offered to help.
* Document in Gdoc and has to be moved, to be easily modified.

The software dispute resolution panel was designed to arbitrate on disputes, and was never used.

Suggestions
* Add in the editor policy the requirement that any new editors must join the [https://wiki.openstreetmap.org/wiki/Software_dispute_resolution_panel software dispute resolution panel]. This would apply both to editors which are part of he website code and external editors.
* The board to decide if the want to make it a requirement of new editors joining SDRP.
* Add the requirement of new editors joining SDRP, in the "community feedback" section of the policy.

On the policy
* There has been a discussion about adding mobile editors.
* OPS are concerned with the technical requirements.
* OPS can suggest new editors joining SDRP. Making it a requirement has to be a board decision.
* The policy can be revised later.

Other points mentioned
* GEO:: is complicated because it's also used by map viewers, as well as editors, and it doesn't allow you to pass anything other than coordinates.
* Establishing SDRP and iD joining was a board decision.
* JOSM is probably grandfathered-in.
* Rapid is a distant iD fork.

#### '''Action items:'''
* Tom to amend the draft editor policy document and move it from Gdoc to hack.md
* OPS to add the SDRP requirement to the Editor Policy draft and see what feedback we receive.

### Fastly future projects
* Separate infrastructure for tests?
* Attribution block error tile image
* Tofu more? Currently partial: https://github.com/openstreetmap/opentofu-fastly
* Automatic config backups?

=== Attribution block error tile image ===
* Possible. Requires a lot of testing and careful header handling.
* We could get away with having our error response be an image, because some clients e.g. will display the image despite it being a 403.
** That was a problem with Leaflet.

Suggestion: Get the test infrastructure working and do iterative tests.

=== Automatic config backups? ===
* We get some special flags turned-on on our accounts, which requires someone in Fastly to change something on our accounts.
* Fastly have raised the amount of backends we can have - the default is 5.
* We have main and a backup distribution and changes were made to our second distribution.

=== Manual back-ups ===
* There is a function in Fastly to do exports.
* Can export global config: it combines all your templates and data objects into a VCL. Once you have a saved copy, you can create a service using it.

VCL:
* current version 329.
* 1432 lines

'''Action item: Grant to download a back-up of Fastly's VCL, every month.'''

### Containerisation of Ridley services, future of containerisation
* Load balancing static services
* High availability [with keepalived](https://j3ffyang.medium.com/nginx-high-availability-and-load-balancing-with-keepalived-521d44798bff)?

More time needed for discussion.

### AOB: Brief discussion on nginx + passenger as replacement apache + mod_evasive.
* Tom has some test work
* brotli compression is unsupported officially in the free version - only supported in the commercial one.

Tom has been making multiple changes to the website. Andy merged last week about three or four months of work.

Suggestion: Potentially get rid of mod-evasive, as Nginx has much better rate limiting support.

### AOB: osm2pgsql expiry (Huge Area issue)
* Tom's patch upstream to limit expiry tiles was not accepted.
* Suggestion we should move to flex backend of osm2pgsql
* Jochen is looking at out-of-memory issue.

### Ubuntu 24.04 PR
* Grant suggested if Tom wants to take over the PR.
* Some bits missing.
* Tom has a spreadsheet of things to build.
* Grant to park the PR.

### AOB: Ironbellly
* Grant would like to move it to Debian.
* Need to setup a separate gateway.
* Tom and Grant will work on it tomorrow.

### AOB: Minting our own Debian packages

Guillaume offered to talk to Florian.

### AOB: Vandalism

Minutes redacted after request.

## Open Ops Tickets
Review open, what needs policy and what needs someone to help with...
https://github.com/openstreetmap/operations/issues
https://github.com/orgs/openstreetmap/projects/1
https://github.com/orgs/openstreetmap/projects/1/views/2?filterQuery=-is%3Aclosed

## Action items
* [2024-05-02](https://hackmd.io/6VQhtcbqRYKDSwx1i-fpWg) Paul to draft a policy regarding deploying new WordPress sites. [Topic: 20th birthday WordPress site] [Ticket created](https://github.com/openstreetmap/operations/issues/1071)
* [2024-05-02](https://hackmd.io/6VQhtcbqRYKDSwx1i-fpWg) Grant to open a ticket about the PR to turn off DNSSEC [Topic: PR to turn off DNSSEC]
* [2024-05-02](https://hackmd.io/6VQhtcbqRYKDSwx1i-fpWg) OPS to revisit the OpenMapTiles application.
* [2024-04-18](https://hackmd.io/_FT8I1sSRL2UNVDM94IbQA) Grant to reply to Equinix, to accept their offer. [Topic: Equinix]
* [2024-02-08](https://ops.pads.ccc.de/meeting-202402-A) OWG to review the Editor policy during one of the next calls and possibly vote on it. [Editor Policy adding to OpenStreetMap.org]
* 2023-11-30 Grant to revisit the "policy for purchasing" document, which currently is focused on specs, and add information such as the process for obtaining approval for purchases. [Reportage] Added info: Who Approves / Steps etc -> Grant to create GitHub ticket
* 2023-11-30 OPS to review the issue of spam reports to ISPs in 6 months (May 2024) -> Grant to create GitHub ticket
* 2023-05-18 Paul to start an open document listing goals for longer-term planning. [Topic: Longer-term planning]

## OPS pads for 2024 meetings
* [2024-03-21](https://hackmd.io/pnBMmOtbQHmweS2XqC8E6g)
* [2024-04-04](https://hackmd.io/O2KUiG9RQXSk9vMAgPA7Dw)
* [2024-04-18](https://hackmd.io/_FT8I1sSRL2UNVDM94IbQA)
* [2024-05-02](https://hackmd.io/6VQhtcbqRYKDSwx1i-fpWg)
* [2024-05-16](https://hackmd.io/XFtol8teRoinL34n7h2bcA)
* [2024-05-30](https://hackmd.io/IM0d3VWhTqWcY-FPwgXy-A)
* [2024-06-13](https://hackmd.io/rDX4FuDfRa-0Trc0IKFJ5Q)
* [2024-06-27](https://hackmd.io/Gxa-ukx6SuKKNlu7lokf6A)
* [2024-07-11](https://hackmd.io/oU9CCUivSPyNUWrp7FP2tQ)
* [2024-07-25](https://hackmd.io/iyFjUWl1RY6D_pevem8ciA)
* [2024-08-08](https://hackmd.io/su12wMb9TR2kd1I5lLJ8vw)
* [2024-08-22](https://hackmd.io/ky2mjfvdTnaJFk9K98TL_g)
* [2024-09-05](https://hackmd.io/A15F6GBrTgKfcXQkaTzDiQ)
* [2024-09-19](https://hackmd.io/yqkAQF6FT2OzmLqNR7I7_g)
* [2024-10-03](https://hackmd.io/Bd98MgqETlyqM1YDoUkq6Q)
* [2024-10-17](https://hackmd.io/wIMfTi0jSVi07XoCN3RFrg)
* [2024-10-31](https://hackmd.io/dqE6-AIvSjmPU31IJ2MtCA)
* [2024-11-14](https://hackmd.io/gru50cHsSvetMprTz7gJ7Q)
* [2024-11-28](https://hackmd.io/mei-vI-bRmSxkWAfWX-jlw)
* [2024-12-12](https://hackmd.io/PXNLnaX_R4OvqQWSyXzzHw)
* [2024-12-26](https://hackmd.io/u3nFsnpQSpCitJL2xe5Hzg)

**[Minutes published on the OSMF website](https://osmfoundation.org/wiki/Working_Group_Minutes#2024)**

### Automatic addition of OPS meetings to your calendar
If you want future OPS meetings to be automatically added to your calendar,
you can subscribe to the following [iCalendar link](https://framagenda.org/remote.php/dav/public-calendars/fce4xrpFGx7fMxz8?export). Instructions on how to do
that depends on which calendar software/service you use. The link is provided
for your convenience.

Note that:
* This might not work when Framasoft servers experience problems.
* You might get delayed updates, depending on how often the calendar service
from your side syncs/checks for updates.