# OSMF Operations working group meeting

Thursday 17 October 2024, 19:00 London time, unless rescheduled
[Time in your timezone](https://www.timeanddate.com/worldclock/fixedtime.html?msg=OSM+Foundation+OPS+meeting+-++Thursday+17+October+2024&iso=20241017T19&p1=136&ah=1)
[Countdown](https://www.timeanddate.com/countdown/generic?p0=136&iso=20241017T19&msg=OSM%20Foundation%20OPS%20meeting%20-%20%20Thursday%2017%20October%202024)
[Online calendar](https://framagenda.org/apps/calendar/p/fce4xrpFGx7fMxz8)
[Subscription to future events](https://framagenda.org/remote.php/dav/public-calendars/fce4xrpFGx7fMxz8?export)
Frequency of meetings: every two weeks, on Thursday at 19:00 London time, unless rescheduled.
[Video room](https://osmvideo.cloud68.co/user/dor-x99-y3m)

## Participants
* Grant
* Tom
* Guillaume
*

# Reportage

### Buying DL380
Grant has discussed some of the options with Paul.

### AArnet servers removal
*Related to action item [2024-09-19](https://hackmd.io/yqkAQF6FT2OzmLqNR7I7_g) Grant to confirm that the AArnet servers will be removed and to ask the Australian community whether there is interest in hosting/providing a render server in Australia or Asia/Pacific [2024-09-19 topic: AArnet Servers going away]*

* A governmental organisation in Australia might be able to offer some servers. Grant to follow up with them.
* They asked us to wait, as they might be able to offer better options later.

### On previous OPS meeting discussion
Grant during one of his recent weekly meetings with Dani Waltersdorfer (board) mentioned that the previous OPS meeting was heated and there were disputes. Dani decided to investigate it further. Guillaume asked Grant to talk to him, as the new board is going to wonder what had happened. Grant made clear that he had not asked Dani to follow-up.

# Agenda

## OpenMapTiles application?

Short discussion, and decision to discuss this with Paul in two weeks.

* We need to assess the application against the criteria.
* The application was previously blocked due to use of Wikipedia data and other issues, which now might be resolved (see email: 29 April).

Other points mentioned
* Policy question: Is it novel enough to qualify. Novelty: Vector Tiles.
* We have removed styles only for tech issues.

### On Paul's Vector Tiles project
* The infrastructure is running and it is in a production suitable state.
* There is a working demo, using the CDN.
* OSMF seems to have scaled down the project and it currently does not have a style.

Suggestions:
* Use a different style.
* Discuss with Paul during the next meeting.

<u>Other point mentioned</u>u>: Paul's tiles are live updated.

## Editor inclusion policy?

There seemed to be a desire on the community discussion https://community.openstreetmap.org/t/updated-proposal-osm-org-editor-inclusion-policy-draft-2/116547 to not require explicit membership to the panel.

<u>Suggestion</u>: The board to decide.
<u>Decision</u>: Examine feedback on community forum and re-discuss in two weeks.

''A part of the discussion was not minuted after request.''

## Imagery Server Proposal

Kessie and Ironbelly are very old.

Kessie has 12x2TB. It's not capable of running modern imagery stuff - it's just too slow. CPU is like a low cost 2012 CPU.

Ironbelly is 10 years old. Disks failing; ongoing replacements with cheap spares. Failures keep happening. Weird unmonitored raid card, weird oob. Management tools hard to get, etc.

Grant is proposing a HP Gen9 DL380, like our other servers. Decent one is £500 excluding storage disks. Grant is suggesting six large disks; Paul thinks we should use SSDs/NVMes, but Grant thinks the cost isn't justified.

Grant has a personal machine we can use. See email.

AWS is possible, but has many unknowns.

Grant will ask the chat for a vote including Paul.

### On imagery storage and access
* Australian reference imagery: 3 TB, doesn't get accessed much anymore.
* UK Ordnance Survey OpenMap and Ordnance Survey Street View imagery: 1 TB processed, we keep all the source files and very small percentage of that gets accessed.
* Namibian Topo: 100 GB, very little of that gets accessed.
* Luxembourg rendered DEM: does not get accessed much.
* US imagery: 1 TB, does not get accessed much.
* Brazil: 200 GB, was given to us and Grant was asked to help with it
* UK small regions: 1 TB

Total storage requirement: At least 50 terabytes.

### On replacing Ironbelly and Kessie

Ironbelly
* Age: ~ 11 years old.
* Disks: failing were replaced with cheap second hand, 1 already failed one expected to fail.
* Weird RAID card, not properly monitored as we have upgraded and management tools are hard to get.
* 7% annual failure rate for the life of the machine. Backblaze failure rates: 1.7% on average.

Kessie:
* Alerting every few mins due to weird RAID performance issue.
* We have replaced all disks in it over time.
* Storage: 20 TB.
* Not capable of running modern imagery.
* Processor old ~ 2012.
* At Exonetric. They are willing to host another server for us, but they wanted something modern and power efficient.

### On not doing the work on AWS
* We'd either have to run on EC2 or Lambda.
* Running on Lambda requires sponsorship, a lot of time to develop it and get it working, unknown how severe the latencies to S3 would be and unknown resources requirements. E.g. how big the machines would need to be and how much data would we push. AWS will need estimates from us before sponsoring. This might take a lot of time to setup.

### On using COGs

Concerns:
* CPU requirements unknown
* Potential latency issues (see below)
* Storage (see below)
* Time invested
* Updates will take long to process, as reprocessing would involve starting from scratch.

**On potential latency issues**
If Cloud Optimized GeoTIFFs (COGs) are not aligned with the intended tile boundaries, compositing multiple layers of COGs becomes necessary, which is resource-intensive. At tile boundary edges, four range requests are often needed, requiring substantial CPU power to composite these layers with masking. High latency in these requests can lead to slow tile loading. For instance, in the case of the Texas imagery that Grant worked on, hundreds of COG files had to be composited, resulting in considerable slow tile loading.

**On storage**
Processing will result in output in a more storage-efficient format, like WebP, where the volume is typically reduced to about half or a quarter of the original. However, even with this reduction, the total storage required is larger than the initial dataset, as both the initial files and the WebP ones have to be stored.

<u>Other points mentioned during discussion</u>
* Community members in New Zealand have developed code that processes COG files, generating new, perfectly tiled COG files.
* Let’s not let perfection be the enemy of completion. We have done this work before, it's not a huge amount of money and can be done relatively quickly.

<u>Suggestion</u>: Post on the forum that we're getting a new imagery server, we're doing it the old way because this is what we know, but if someone wants to play with this project and this project and build a proof of concept.
* Grant has tried on OSM US channel, but there were no takers.

### On options and cost

1. DL380 of the Gen 10, 12 bays - GBP 19K.

2. 6x 18-20TB enterprise new SAS disks for £1,650.
* Advantage: space to grow.
* New disks mean we don't have failure rate issues. Expected: ~ 1% failure rate, advertised: 0.35%.
* The cost is within the budget left for the year.

Grant purchased the machine with his own money, as he was planning to go to Amsterdam and will take it back if OPS decide not to use it. Has seven 6TB disks, which is enough storage to get started and run some imagery. Needs more storage, which would have to be added with remote hands.

## Any Other Business
### Staging of blog.osm.org on Tabaluga
* Grant will host the staging of blog.osm.org on [Tabaluga](https://hardware.openstreetmap.org/servers/tabaluga.openstreetmap.org/) ''(HP ProLiant DL360 Gen9)''. The CWG has a contractor that will work on improving the blog and he's in contact with Mikel Maron (CWG).
* Grant agreed to rework blog.osm.org into Hugo and create a demo, as it might be better to move away from WordPress. There are many templates for Hugo and commercial themes.

## Open Ops Tickets
Review open, what needs policy and what needs someone to help with...
https://github.com/openstreetmap/operations/issues
https://github.com/orgs/openstreetmap/projects/1
https://github.com/orgs/openstreetmap/projects/1/views/2?filterQuery=-is%3Aclosed

## Action items
* [2024-09-19](https://hackmd.io/yqkAQF6FT2OzmLqNR7I7_g) Grant to create a ticket for action item [2024-08-08](https://hackmd.io/su12wMb9TR2kd1I5lLJ8vw) OPS to evaluate Fastly Security (DDOS) Protections we could use. [Topic: Cloudflare / Fastly]
* [2024-09-19](https://hackmd.io/yqkAQF6FT2OzmLqNR7I7_g) Grant to create an IP blocklist. [Topic: Cloudflare keep enabled?][2024-09-19 Reportage] - Discussion during [2024-07-25](https://hackmd.io/iyFjUWl1RY6D_pevem8ciA) OPS to make a reasonable evaluation whether to go with Cloudflare, Fastly or none.
* <s>[2024-09-19](https://hackmd.io/yqkAQF6FT2OzmLqNR7I7_g) Paul to add the OpenMapTiles application to the next agenda, together with the editor inclusion policy. [2024-09-19 Reportage]</s> # On the 2024-10-17 agenda
* <s>[2024-09-19](https://hackmd.io/yqkAQF6FT2OzmLqNR7I7_g) Grant to come up with estimates for space needs in next 5 years and cost for Ironbelly replacement* [2024-09-19 topic: Ironbelly Replacement? ]</s> Done
* [2024-09-19](https://hackmd.io/yqkAQF6FT2OzmLqNR7I7_g) Grant to confirm that the AArnet servers will be removed and to ask the Australian community whether there is interest in hosting/providing a render server in Australia or Asia/Pacific [2024-09-19 topic: AArnet Servers going away]
* [2024-08-22](https://hackmd.io/ky2mjfvdTnaJFk9K98TL_g) Guillaume to make a limited in scale experiment to assess impact and practicality. E.g. look if there are clients that say they support WEBP and don't. [Topic: Fastly image recoding]
* [2024-08-22](https://hackmd.io/ky2mjfvdTnaJFk9K98TL_g) Guillaume to keep OPS in the loop about what Fastly says. [Topic: Fastly image recoding] # 2024-10-17 No news
* [2024-08-22](https://hackmd.io/ky2mjfvdTnaJFk9K98TL_g) Grant to talk to Guillaume on setting up the testing about image recoding and shielding. [Topic: Fastly image recoding]
* [2024-08-08](https://hackmd.io/su12wMb9TR2kd1I5lLJ8vw) OPS to evaluate Fastly Security (DDOS) Protections we could use. [Topic: Cloudflare / Fastly]
* [2024-07-25](https://hackmd.io/iyFjUWl1RY6D_pevem8ciA) OPS to make a reasonable evaluation whether to go with Cloudflare, Fastly or none. [Topic: Cloudflare keep enabled?]
* [2024-07-25](https://hackmd.io/iyFjUWl1RY6D_pevem8ciA) Grant to determine the Cloudflare API call to block IPs, in order to deal with scrappers [Topic: Cloudflare keep enabled?] 2024-08-08 update: Briefly discussed
* [2024-06-27](https://hackmd.io/Gxa-ukx6SuKKNlu7lokf6A) OPS to do capacity planning for tile.openstreetmap.org [Topic: rhaegel usage?]
~~* [2024-05-02](https://hackmd.io/6VQhtcbqRYKDSwx1i-fpWg) OPS to revisit the OpenMapTiles application. # 2024-06-13 They haven't responded to the questions. Paul to email them again. # 2024-07-25 They have replied, OPS haven't had a chance to look at the answers. ~~

## OPS pads for 2024 meetings
* [2024-03-21](https://hackmd.io/pnBMmOtbQHmweS2XqC8E6g)
* [2024-04-04](https://hackmd.io/O2KUiG9RQXSk9vMAgPA7Dw)
* [2024-04-18](https://hackmd.io/_FT8I1sSRL2UNVDM94IbQA)
* [2024-05-02](https://hackmd.io/6VQhtcbqRYKDSwx1i-fpWg)
* [2024-05-16](https://hackmd.io/XFtol8teRoinL34n7h2bcA)
* [2024-05-30](https://hackmd.io/IM0d3VWhTqWcY-FPwgXy-A)
* [2024-06-13](https://hackmd.io/rDX4FuDfRa-0Trc0IKFJ5Q)
* [2024-06-27](https://hackmd.io/Gxa-ukx6SuKKNlu7lokf6A)
* [2024-07-11](https://hackmd.io/oU9CCUivSPyNUWrp7FP2tQ)
* [2024-07-25](https://hackmd.io/iyFjUWl1RY6D_pevem8ciA)
* [2024-08-08](https://hackmd.io/su12wMb9TR2kd1I5lLJ8vw)
* [2024-08-22](https://hackmd.io/ky2mjfvdTnaJFk9K98TL_g)
* [2024-09-05](https://hackmd.io/A15F6GBrTgKfcXQkaTzDiQ)
* [2024-09-19](https://hackmd.io/yqkAQF6FT2OzmLqNR7I7_g)
* [2024-10-03](https://hackmd.io/Bd98MgqETlyqM1YDoUkq6Q)
* [2024-10-17](https://hackmd.io/wIMfTi0jSVi07XoCN3RFrg)
* [2024-10-31](https://hackmd.io/dqE6-AIvSjmPU31IJ2MtCA)
* [2024-11-14](https://hackmd.io/gru50cHsSvetMprTz7gJ7Q)
* [2024-11-28](https://hackmd.io/mei-vI-bRmSxkWAfWX-jlw)
* [2024-12-12](https://hackmd.io/PXNLnaX_R4OvqQWSyXzzHw)
* [2024-12-26](https://hackmd.io/u3nFsnpQSpCitJL2xe5Hzg)

**[Minutes published on the OSMF website](https://osmfoundation.org/wiki/Working_Group_Minutes#2024)**

### Automatic addition of OPS meetings to your calendar
If you want future OPS meetings to be automatically added to your calendar,
you can subscribe to the following [iCalendar link](https://framagenda.org/remote.php/dav/public-calendars/fce4xrpFGx7fMxz8?export). Instructions on how to do
that depends on which calendar software/service you use. The link is provided
for your convenience.

Note that:
* This might not work when Framasoft servers experience problems.
* You might get delayed updates, depending on how often the calendar service
from your side syncs/checks for updates.