# OSMF OPS meeting 2024-03-07

7 March 2024, 19:00 London time, unless rescheduled
[Time in your timezone](https://www.timeanddate.com/worldclock/fixedtime.html?msg=OSM+Foundation+OPS+meeting+-++Thursday+7+March+2024&iso=20240307T19&p1=136&ah=1)
[Countdown](https://www.timeanddate.com/countdown/generic?p0=136&iso=20240307T19&msg=OSM%20Foundation%20OPS%20meeting%20-%20%20Thursday%207%20March%202024)
[Online calendar](https://framagenda.org/apps/calendar/p/fce4xrpFGx7fMxz8)
[Subscription to future events](https://framagenda.org/remote.php/dav/public-calendars/fce4xrpFGx7fMxz8?export)
Frequency of meetings: every two weeks, on Thursday at 19:00 London time, unless rescheduled.
[Video room](https://osmvideo.cloud68.co/user/dor-x99-y3m)

## Participants
Paul, Tom, Grant, Guillaume

## New action items from this meeting
* Grant to open a ticket about running our own IMAP server, to describe the problem and sample cases. [Topic: Running our own IMAP server]
* Grant and Guillaume to discuss further about the redundancy of gateway / IPv6 "private subnet". Benefit / "Cost" [Topic: Redundancy of Gateway / IPv6 "private subnet"]

## Agenda

### Equinix Price Increase
Equinix is requiring a 24% price increase on space, 24% on power, and 5% on crossconnects.

* Space cost increase: The increase in space cost is likely justifiable, as it has remained unchanged since we moved in.
* Crossconnects cost increase: appears reasonable.
* Power cost increase: We were informed of a price increase in October, effective from January, with assurances of no further increases.
** A ticket regarding this matter has been opened with the salesperson. Resolution may require escalation.
** The statement of "No price increase" was communicated via a notification email from a no-reply address in November 2023.

=== Suggestions ===
* Pursue clarification on the validity of the "No price increase" statement.
* Ask for a discount, citing our acknowledgment of Equinix on the thanks page.
* Verify if the "No price increase" commitment extends to future increases in Dublin.
** This query will be directed to a different salesperson.
** Concensus seemed to be to defer this inquiry for now. If a price increase email is received, we'll follow up then.
* Seek an explanation for the rise in electricity costs, mentioning our budgetary commitments and donor scrutiny.

=== On formally asking for a discount as we have Equinix on our thanks page ===
* We inquired about potential discounts during our initial signup in Amsterdam. There is no clarity about any discounts at the time, although we did secure a favorable price for one rack.
* Requesting a discount after several years, may seem weird.
* Should Equinix decline to agree, remove their mention from the thanks page.
* Grant offered to email Equinix.
* Will save ~ EUR 8,000 with greater savings anticipated next year, given its mid-year commencement.

### Attendence at SOTM EU
* Paul
* Tom, Grant, Guillaume maybe.

Suggestion: If all OPS go, they could book an extra night for a meeting.

### Insurance update
* They need some locations.
* They probably have not provided an actual dollar value.
*
### Running IMAP server
* Ticket: Mailbox.org forward has issues with header rewrite... Check SPF (Grant)
* Ticket: "Spam" sent via OSMF forms (Grant)

We forward a lot of spam that we get from the DMCA form or OTRS and sometimes, it doesn't get delivered.
* The DMCA form submits emails to DWG.
* Emails to OTRS generate email notifications: the email goes to the forwarder email address before reaching other recipients.
Those emails are frequently spam messages, which we automatically send to others. The spam that appears to be generated from our site. The destination ISPs think it's spam and bounce it.

<u>Suggestions</u>:
* Operating our own IMAP server (with no SMTP functionality, exclusively for internal use without outgoing email capabilities) could resolve the issue. It would act as an incoming SMTP, preventing forwarded emails from being bounced by other servers.
** An IMAP server without an SMTP service to deliver email into it is irrelevant, except in rare scenarios.
** There are better alternative solutions than running an IMAP server.
* Open a ticket.
* Stop people using their private email addresses and instead have them get an OSMF email address.
** This will have some admin burden.
* The DMCA form should post straight into the OTRS API.
** This approach would move the problem to OTRS, as OTRS will send notifications by email that will get bounced.

<u>Other points mentioned during discussion</u>:
* Forwarding emails through mailbox.org to other addresses is highly unreliable. Mailbox.org does not do the things that Google does.
* An SPF issue may arise.
** It's probably fine because it is coming from an authorized OSMF outgoing server.
* We prefer not to manage email infrastructure unless it becomes an absolute necessity.

**'''Action item: Grant to open a ticket, to describe the problem and sample cases.'''**

### Redundancy of Gateway / IPv6 "private subnet"

Concern: running tasks such as replication on old machines, like Ironbelly and Ridley.

Suggestion: Grant and Guillaume to discuss the issue,ly, documenting objectives, constraints, and the workload for implementing changes.

<u>Other points mentioned during discussion</u>
* We should avoid relying on a single host, as it is a point of failure.
** Over the past twelve years or more, failures have occurred infrequently and were typically resolved within five minutes. Considering the potential workload for changes, maintaining the current setup may be justified.

**Action item: Grant and Guillaume to discuss further. Benefit / "Cost"**

### Vector tile
Performance discussion of current development implementation.

* Paul put a demo up of his vector tiles work.
* Currently runs single-threaded on a Hertzner machine (6 core, consumer drive, several year old machine) and performs quite wall.
* It's less demanding than an image rendering server on disk space and on I/O.

Future:
* Parallel processing still needed for the initial pre-rendering of millions of tiles.
* Paul wants to dockerise it properly, to simplify deployment of development environments.
** Grant offered to help.
** Implementation is expected no sooner than one month from now.

Suggestions:
* Move it to a shared machine, because it doesn't warrant a full machine.
* Remove the libraries it relies on to ensure complete separation.

### Nominatim
* Nominatim is currently on a Debian machine.
* Sarah Hoffmann asked to keep the old machine for some time, for experimentation purposes.

### help.osm.org
help.osm.org is now read-only. The login button has been removed, and the OAuth page returns a 403 error.

<u>Tasks</u>
* Replace memory.
* Derack all the machines. . Currently, we lack storage or disposal options for them.

<u>Options</u>
* Convert it into a container.
* Create a static archive.
Some archive.org people are scraping it, currently running 3 similtaneous gets.

<u>On suggestion to make it read-only, removing user permissions on the tables</u>
* This process is relatively complex with PostgreSQL.
* Django might encounter errors, necessitating that we retain the session tables.
Grant truncated the session table to force all users to log out.
We can change the session database to use Redis and Memcached, but the benefits may not outweigh the effort.

<u>Other points mentioned during discussion</u>
* We want to avoid running unmaintained code.

OPS to look for the minimal solution with minimal ongoing administrative overhead.

## UCL visit by Grant
*Upcoming visit by Grant, discussion of challenges regarding bringing in and removing of hardware. Large number of machines to be decommissioned which may need removing.*

<u>Tasks</u>
* Replace memory.
* Derack all the machines. . Currently, we lack storage or disposal options for them.

<u>Options:</u>
* E-waste.
* Post to inquire if anyone is interested in taking them.

<u>E-waste issues</u>
* Drop-off is nearly impossible without certification documents and evidence that we're not disposing of hazardous materials like nuclear waste. It involves significant physical effort and paperwork.
* Collection is usually from commercial businesses.

<u>Suggestions</u>
* Email UCL to ask them about the possibility of using their storage center for drop-off.
* Email the data centre directly.
** They will probably request the opening of a ticket, but we lack access to their ticketing system.
* Email the small team that assisted with our move from the old site.

<u>Other points mentioned during discussion</u>
* UCL E-waste bin is in locked separate section.
* There are public waste bins.
** Equinix used these.
* UCL are getting the data center via JYSKK, who subcontract the operation to a commercial company.
* It's is unclear if our initial contacts are there anymore.

=== Plan ===
Do maintenance (RAM swaps), take inventory of hardware that needs to be moved, ensure access works. Then put a notice offering the hardware and do a follow-up visit with better timing.

### Ridley
Many services still need to be moved. Containers are primary choice.

* There's a ticket for that.
* Some OSMF stuff has already moved.

## Open Ops Tickets
Review open, what needs policy and what needs someone to help with...
https://github.com/openstreetmap/operations/issues
https://github.com/orgs/openstreetmap/projects/1
https://github.com/orgs/openstreetmap/projects/1/views/2?filterQuery=-is%3Aclosed

## Action items
~~* 2024-02-22 Paul to email Equinix about the price increase [Topic: Equinix AM6 Price Increase]~~
~~* 2024-02-22 Paul to open a ticket for Fastly root access and ask Grant there whether Ian has access. [Topic: Fastly root access]~~
~~* 2024-02-08 Paul to ping Sarah on the ticket regarding the Overpass move. [New general purpose machines for DB4]~~
* 2024-02-08 OWG to review the Editor policy during one of the next calls and possibly vote on it. [Editor Policy adding to OpenStreetMap.org]
~~* 2024-01-25 Tom to investigate what happens to existing OAuth 1 tokens when OAuth 1 gets turned off (they keep working), and whether error messages can be sent back in the html of the response to basic/oauth 1 users (yes, PR in progress). [Topic:OAuth 1.0a]~~
* 2023-11-30 Grant to revisit the "policy for purchasing" document, which currently is focused on specs, and add information such as the process for obtaining approval for purchases. [Reportage] Added info: Who Approves / Steps etc -> Grant to create github ticket
* 2023-11-30 OPS to review the issue of spam reports to ISPs in 6 months (May 2024) -> Grant to create github ticket
* 2023-05-18 Paul to start an open document listing goals for longer-term planning. [Topic: Longer-term planning]

## OPS pads for 2024 meetings
* [2024-03-21](https://hackmd.io/pnBMmOtbQHmweS2XqC8E6g)
* [2024-04-04](https://hackmd.io/O2KUiG9RQXSk9vMAgPA7Dw)
* [2024-04-18](https://hackmd.io/_FT8I1sSRL2UNVDM94IbQA)
* [2024-05-02](https://hackmd.io/6VQhtcbqRYKDSwx1i-fpWg)
* [2024-05-16](https://hackmd.io/XFtol8teRoinL34n7h2bcA)
* [2024-05-30](https://hackmd.io/IM0d3VWhTqWcY-FPwgXy-A)
* [2024-06-13](https://hackmd.io/rDX4FuDfRa-0Trc0IKFJ5Q)
* [2024-06-27](https://hackmd.io/Gxa-ukx6SuKKNlu7lokf6A)
* [2024-07-11](https://hackmd.io/oU9CCUivSPyNUWrp7FP2tQ)
* [2024-07-25](https://hackmd.io/iyFjUWl1RY6D_pevem8ciA)
* [2024-08-08](https://hackmd.io/su12wMb9TR2kd1I5lLJ8vw)
* [2024-08-22](https://hackmd.io/ky2mjfvdTnaJFk9K98TL_g)
* [2024-09-05](https://hackmd.io/A15F6GBrTgKfcXQkaTzDiQ)
* [2024-09-19](https://hackmd.io/yqkAQF6FT2OzmLqNR7I7_g)
* [2024-10-03](https://hackmd.io/Bd98MgqETlyqM1YDoUkq6Q)
* [2024-10-17](https://hackmd.io/wIMfTi0jSVi07XoCN3RFrg)
* [2024-10-31](https://hackmd.io/dqE6-AIvSjmPU31IJ2MtCA)
* [2024-11-14](https://hackmd.io/gru50cHsSvetMprTz7gJ7Q)
* [2024-11-28](https://hackmd.io/mei-vI-bRmSxkWAfWX-jlw)
* [2024-12-12](https://hackmd.io/PXNLnaX_R4OvqQWSyXzzHw)
* [2024-12-26](https://hackmd.io/u3nFsnpQSpCitJL2xe5Hzg)

**[Minutes published on the OSMF website](https://osmfoundation.org/wiki/Working_Group_Minutes#2024)**

### Automatic addition of OPS meetings to your calendar
If you want future OPS meetings to be automatically added to your calendar,
you can subscribe to the following [iCalendar link](https://framagenda.org/remote.php/dav/public-calendars/fce4xrpFGx7fMxz8?export). Instructions on how to do
that depends on which calendar software/service you use. The link is provided
for your convenience.

Note that:
* This might not work when Framasoft servers experience problems.
* You might get delayed updates, depending on how often the calendar service
from your side syncs/checks for updates.