hachyderm SLOs

in an effort to be more targeted with our efforts to keep the site online, it seems reasonable to consider SLO definitions. this will allow us to both focus where focus is necessary, and to stop dashboard trawling and rely on alerts instead.

this document will define SLOs per grafana dashboard, for some level of structure.

nginx SLOs

dashboard

• 90th %ile response times (GET 200) [global]: 10s for 1m
  • under normal circumstances we are around 1s - 3s.
  • issues are reported during periods of much higher response times (10s+)
• HTTP 5xx Errors (global): 10k
  • we have a background level of ~5k 500s (boo)
  • outages happen during periods of much higher rates (100k+)

mastodon SLOs

dashboard
while it would be great to claim all queues are under some threshold, different queues affect the service differently. in priority order:

​​​​* default: the local experience
​​​​* push: delivery to other instances
​​​​* ingress: incoming activity pub processing
​​​​* mailers: sending email
​​​​* scheduler: cron jobs like refreshing hashtags and cleaning logs
​​​​* pull: low priority tasks such as imports, backups, deleting users

• queue latency [default]: 30s
  • anything above 1s feels slow locally
• queue latency [push]: 5m
  • being a good citizen
• queue latency [ingress]: 5m
  • being a good citizen
• queue latency [mailers]: 30m
  • based on expected experience
• queue latency [pull]: 5d
  • if we haven't done these things in 5d we probably should.

node SLOs

dashboard

- [ ] disk space used [/mnt/mastodon-storage]: 80%

• sys load 5m: 200%
  • this might be a bit tight as we do run the machines hot
• RAM: 90%

postgresql SLOs

dashboard

TODO!(idle sessions or active sessions could be options)