# Postgres Migration :warning: Public Document :warning: ### State of The World - pgbouncer is not running anywhere #### freud - no sidekiqs (confirmed) - `postgresql.service` running - `/etc/cron.daily/pg_archive_cleanup.sh` dead #### neitzsche - `postgresql.service` running (replica) #### fritz - `mastodon-web.service` running - `mastodon-streaming.service` running - `mastodon-sidekiq-mailers.service` running - `mastodon-sidekiq-default@{1,2}` running - `mastodon-sidekiq-pull@1` running - `mastodon-sidekiq-push@1` running #### cdn-fremont-1 - no sidekiqs (confirmed) #### cdn-frankfurt-1 - sidekiq server - `mastodon-sidekiq@1.service` running - `mastodon-sidekiq-pushpull@1.service` running #### slappy - no sidekiqs (confirmed) #### esme - no sidekiqs (confirmed) #### franz - `mastodon-sidekiq-scheduler.service` running - `mastodon-sidekiq-default@{1..6}.service` running - `mastodon-sidekiq-ingress@{1..3}.service` running - `mastodon-sidekiq-pull@{1..5}.service` running - `mastodon-sidekiq-push@{1..5}.service` running # Migration Steps ## Fritz: Stop processes that generate jobs (web, streaming) ```shell ssh novix@fritz.hachyderm.io sudo systemctl stop \ mastodon-web.service \ mastodon-streaming.service ``` Wait for queues to be zero: https://hachyderm.io/sidekiq/queues ## All: Stop sidekiqs First, stop the scheduler ```shell ssh novix@franz.hachyderm.io sudo systemctl stop \ mastodon-sidekiq-scheduler.service ``` Then the rest ```shell ssh novix@fritz.hachyderm.io sudo systemctl stop \ mastodon-sidekiq-mailers.service \ mastodon-sidekiq-default@{1,2} \ mastodon-sidekiq-{push,pull}@1 ``` ```shell ssh novix@cdn-frankfurt-1.hachyderm.io sudo systemctl stop \ mastodon-sidekiq{,-pushpull}@1.service ``` ```shell ssh novix@franz.hachyderm.io sudo systemctl stop \ mastodon-sidekiq-default@{1..6}.service \ mastodon-sidekiq-ingress@{1..3}.service \ mastodon-sidekiq-{pull,push}@{1..5}.service ``` ## Nietzsche: Check replication delay ```shell ssh novix@nietzsche.hachyderm.io watch -n1 'sudo -u postgres psql -c "select now()-pg_last_xact_replay_timestamp() as replication_lag;"' ``` ## (Freud, Nietzsche): Stop postgres ```shell ssh novix@freud.hachyderm.io sudo systemctl stop postgresql sleep 4 # randomly chosen by fair dice roll ssh novix@nietzsche.hachyderm.io sudo systemctl stop postgresql ``` ## Nietzsche: Configure as main ```shell ssh novix@nietzsche.hachyderm.io sudo rm /var/lib/postgres/data/standby.signal ssh novix@nietzsche.hachyderm.io sudo sed -i 's/^primary_conninfo/#primary_conninfo/' /var/lib/postgres/data/postgresql.conf ssh novix@nietzsche.hachyderm.io sudo sed -i 's/^hot_standby/#hot_standby/' /var/lib/postgres/data/postgresql.conf ssh novix@nietzsche.hachyderm.io sudo systemctl start postgresql ``` ## Freud: Configure as replica ```shell ssh novix@freud.hachyderm.io sudo touch /var/lib/postgres/data/standby.signal ssh novix@freud.hachyderm.io sudo sed -i -e 's/^#primary_conninfo/primary_conninfo/' -e '/^primary_conninfo/s/alice/nietzsche/' /var/lib/postgres/data/postgresql.conf ssh novix@freud.hachyderm.io sudo sed -i 's/^#hot_standby/hot_standby/' /var/lib/postgres/data/postgresql.conf ssh novix@freud.hachyderm.io sudo systemctl start postgresql ``` ## Freud: Check replication delay ```shell ssh novix@freud.hachyderm.io watch -n1 'sudo -u postgres psql -c "select now()-pg_last_xact_replay_timestamp() as replication_lag;"' ``` ## All: Modify client configs to connect to Nietzsche ```shell for server in fritz cdn-frankfurt-1 franz; do ssh novix@$server.hachyderm.io sudo sed -i 's/DB_HOST=freud/DB_HOST=nietzsche/' /etc/mastodon.conf done ``` ## All: Start all clients First, start the scheduler sidekiq ```shell ssh novix@franz.hachyderm.io sudo systemctl start \ mastodon-sidekiq-scheduler.service ``` Then the rest ```shell ssh novix@fritz.hachyderm.io sudo systemctl start \ mastodon-sidekiq-mailers.service \ mastodon-sidekiq-default@{1,2} \ mastodon-sidekiq-{push,pull}@1 ``` ```shell ssh novix@cdn-frankfurt-1.hachyderm.io sudo systemctl start \ mastodon-sidekiq{,-pushpull}@1.service ``` ```shell ssh novix@franz.hachyderm.io sudo systemctl start \ mastodon-sidekiq-default@{1..6}.service \ mastodon-sidekiq-ingress@{1..3}.service \ mastodon-sidekiq-{pull,push}@{1..5}.service ``` Then start the web and streaming ```shell ssh novix@fritz.hachyderm.io sudo systemctl start \ mastodon-web.service \ mastodon-streaming.service ``` # Follow Up Tasks - [x] Install mastodon on freud - [x] Copy mastodon.conf to freud - [x] Move sidekiq from `cdn-frankfurt-1` -> `freud` - [x] `mastodon-sidekiq{,-pushpull}@1.service` - [ ] Move sidekiq from `fritz` -> `freud` - [X] `mastodon-sidekiq-mailers.service` - [X] `mastodon-sidekiq-default@{1,2}` - [X] `mastodon-sidekiq-pull@1` - [X] `mastodon-sidekiq-push@1` - [X] Start sidekiq extras on `freud` - [ ] 2 more ingress queues - [ ] 3 more default queues - [X] Clean up systemd sidekiqs on cdn-frankfurt-1 and fritz - [ ] Copy media/images/mailer/icon_email.png from fritz to freud --- ``` # cdn-frankfurt-1: # /etc/systemd/system/mastodon-sidekiq-pushpull@.service [Unit] Description=mastodon-sidekiq (%i) After=network.target [Service] Type=simple User=mastodon WorkingDirectory=/var/lib/mastodon Environment="RAILS_ENV=production" Environment="DB_POOL=10" Environment="MALLOC_ARENA_MAX=2" Environment="LD_PRELOAD=libjemalloc.so" ExecStart=/usr/bin/bundle exec sidekiq -c 10 -q pull -q push TimeoutSec=15 Restart=always # Proc filesystem ProcSubset=pid ProtectProc=invisible # Capabilities CapabilityBoundingSet= # Security NoNewPrivileges=true # Sandboxing ProtectSystem=strict PrivateTmp=true PrivateDevices=true PrivateUsers=true ProtectHostname=true ProtectKernelLogs=true ProtectKernelModules=true ProtectKernelTunables=true ProtectControlGroups=true RestrictAddressFamilies=AF_INET RestrictAddressFamilies=AF_INET6 RestrictAddressFamilies=AF_NETLINK RestrictAddressFamilies=AF_UNIX RestrictNamespaces=true LockPersonality=true RestrictRealtime=true RestrictSUIDSGID=true RemoveIPC=true PrivateMounts=true ProtectClock=true # System Call Filtering SystemCallArchitectures=native SystemCallFilter=~@cpu-emulation @debug @keyring @ipc @mount @obsolete @privileged @setuid SystemCallFilter=@chown SystemCallFilter=pipe SystemCallFilter=pipe2 ReadWritePaths=/var/lib/mastodon [Install] WantedBy=multi-user.target ``` ``` # fritz # /etc/systemd/system/mastodon-sidekiq-mailers.service [Unit] Description=mastodon-sidekiq After=network.target [Service] Type=simple User=mastodon WorkingDirectory=/var/lib/mastodon Environment="RAILS_ENV=production" Environment="DB_POOL=1" Environment="MALLOC_ARENA_MAX=2" Environment="LD_PRELOAD=libjemalloc.so" ExecStart=/usr/bin/bundle exec sidekiq -c 1 -q mailers TimeoutSec=15 Restart=always # Proc filesystem ProcSubset=pid ProtectProc=invisible # Capabilities CapabilityBoundingSet= # Security NoNewPrivileges=true # Sandboxing ProtectSystem=strict PrivateTmp=true PrivateDevices=true PrivateUsers=true ProtectHostname=true ProtectKernelLogs=true ProtectKernelModules=true ProtectKernelTunables=true ProtectControlGroups=true RestrictAddressFamilies=AF_INET RestrictAddressFamilies=AF_INET6 RestrictAddressFamilies=AF_NETLINK RestrictAddressFamilies=AF_UNIX RestrictNamespaces=true LockPersonality=true RestrictRealtime=true RestrictSUIDSGID=true RemoveIPC=true PrivateMounts=true ProtectClock=true # System Call Filtering SystemCallArchitectures=native SystemCallFilter=~@cpu-emulation @debug @keyring @ipc @mount @obsolete @privileged @setuid SystemCallFilter=@chown SystemCallFilter=pipe SystemCallFilter=pipe2 ReadWritePaths=/var/lib/mastodon [Install] WantedBy=multi-user.target ``` ``` # fritz # /etc/systemd/system/mastodon-sidekiq-default@.service [Unit] Description=mastodon-sidekiq default (%i) After=network.target [Service] Type=simple User=mastodon WorkingDirectory=/var/lib/mastodon Environment="RAILS_ENV=production" Environment="DB_POOL=10" Environment="MALLOC_ARENA_MAX=2" Environment="LD_PRELOAD=libjemalloc.so" ExecStart=/usr/bin/bundle exec sidekiq -c 10 -q default TimeoutSec=15 Restart=always # Proc filesystem ProcSubset=pid ProtectProc=invisible # Capabilities CapabilityBoundingSet= # Security NoNewPrivileges=true # Sandboxing ProtectSystem=strict PrivateTmp=true PrivateDevices=true PrivateUsers=true ProtectHostname=true ProtectKernelLogs=true ProtectKernelModules=true ProtectKernelTunables=true ProtectControlGroups=true RestrictAddressFamilies=AF_INET RestrictAddressFamilies=AF_INET6 RestrictAddressFamilies=AF_NETLINK RestrictAddressFamilies=AF_UNIX RestrictNamespaces=true LockPersonality=true RestrictRealtime=true RestrictSUIDSGID=true RemoveIPC=true PrivateMounts=true ProtectClock=true # System Call Filtering SystemCallArchitectures=native SystemCallFilter=~@cpu-emulation @debug @keyring @ipc @mount @obsolete @privileged @setuid SystemCallFilter=@chown SystemCallFilter=pipe SystemCallFilter=pipe2 ReadWritePaths=/var/lib/mastodon [Install] WantedBy=multi-user.target ``` ``` # fritz # /etc/systemd/system/mastodon-sidekiq-push@.service [Unit] Description=mastodon-sidekiq push (%i) After=network.target [Service] Type=simple User=mastodon WorkingDirectory=/var/lib/mastodon Environment="RAILS_ENV=production" Environment="DB_POOL=10" Environment="MALLOC_ARENA_MAX=2" Environment="LD_PRELOAD=libjemalloc.so" ExecStart=/usr/bin/bundle exec sidekiq -c 10 -q push TimeoutSec=15 Restart=always # Proc filesystem ProcSubset=pid ProtectProc=invisible # Capabilities CapabilityBoundingSet= # Security NoNewPrivileges=true # Sandboxing ProtectSystem=strict PrivateTmp=true PrivateDevices=true PrivateUsers=true ProtectHostname=true ProtectKernelLogs=true ProtectKernelModules=true ProtectKernelTunables=true ProtectControlGroups=true RestrictAddressFamilies=AF_INET RestrictAddressFamilies=AF_INET6 RestrictAddressFamilies=AF_NETLINK RestrictAddressFamilies=AF_UNIX RestrictNamespaces=true LockPersonality=true RestrictRealtime=true RestrictSUIDSGID=true RemoveIPC=true PrivateMounts=true ProtectClock=true # System Call Filtering SystemCallArchitectures=native SystemCallFilter=~@cpu-emulation @debug @keyring @ipc @mount @obsolete @privileged @setuid SystemCallFilter=@chown SystemCallFilter=pipe SystemCallFilter=pipe2 ReadWritePaths=/var/lib/mastodon [Install] WantedBy=multi-user.target ``` ``` # fritz # /etc/systemd/system/mastodon-sidekiq-pull@.service [Unit] Description=mastodon-sidekiq pull (%i) After=network.target [Service] Type=simple User=mastodon WorkingDirectory=/var/lib/mastodon Environment="RAILS_ENV=production" Environment="DB_POOL=10" Environment="MALLOC_ARENA_MAX=2" Environment="LD_PRELOAD=libjemalloc.so" ExecStart=/usr/bin/bundle exec sidekiq -c 10 -q pull TimeoutSec=15 Restart=always # Proc filesystem ProcSubset=pid ProtectProc=invisible # Capabilities CapabilityBoundingSet= # Security NoNewPrivileges=true # Sandboxing ProtectSystem=strict PrivateTmp=true PrivateDevices=true PrivateUsers=true ProtectHostname=true ProtectKernelLogs=true ProtectKernelModules=true ProtectKernelTunables=true ProtectControlGroups=true RestrictAddressFamilies=AF_INET RestrictAddressFamilies=AF_INET6 RestrictAddressFamilies=AF_NETLINK RestrictAddressFamilies=AF_UNIX RestrictNamespaces=true LockPersonality=true RestrictRealtime=true RestrictSUIDSGID=true RemoveIPC=true PrivateMounts=true ProtectClock=true # System Call Filtering SystemCallArchitectures=native SystemCallFilter=~@cpu-emulation @debug @keyring @ipc @mount @obsolete @privileged @setuid SystemCallFilter=@chown SystemCallFilter=pipe SystemCallFilter=pipe2 ReadWritePaths=/var/lib/mastodon [Install] WantedBy=multi-user.target ``` ``` # franz # /etc/systemd/system/mastodon-sidekiq-ingress@.service [Unit] Description=mastodon-sidekiq After=network.target [Service] Type=simple User=mastodon WorkingDirectory=/var/lib/mastodon Environment="RAILS_ENV=production" Environment="DB_POOL=20" Environment="MALLOC_ARENA_MAX=2" Environment="LD_PRELOAD=libjemalloc.so" ExecStart=/usr/bin/bundle exec sidekiq -c 20 -q ingress TimeoutSec=15 Restart=always # Proc filesystem ProcSubset=pid ProtectProc=invisible # Capabilities CapabilityBoundingSet= # Security NoNewPrivileges=true # Sandboxing ProtectSystem=strict PrivateTmp=true PrivateDevices=true PrivateUsers=true ProtectHostname=true ProtectKernelLogs=true ProtectKernelModules=true ProtectKernelTunables=true ProtectControlGroups=true RestrictAddressFamilies=AF_INET RestrictAddressFamilies=AF_INET6 RestrictAddressFamilies=AF_NETLINK RestrictAddressFamilies=AF_UNIX RestrictNamespaces=true LockPersonality=true RestrictRealtime=true RestrictSUIDSGID=true RemoveIPC=true PrivateMounts=true ProtectClock=true # System Call Filtering SystemCallArchitectures=native SystemCallFilter=~@cpu-emulation @debug @keyring @ipc @mount @obsolete @privileged @setuid SystemCallFilter=@chown SystemCallFilter=pipe SystemCallFilter=pipe2 ReadWritePaths=/var/lib/mastodon ReadWritePaths=/mnt/mastodon-storage [Install] WantedBy=multi-user.target ``` # Resulting State of the World #### Nietzsche - Postgres primary - 4X Ingress queues - 4X Default queues #### Freud - Postgres secondary - 3X Ingress queues - 3X Default queues - 1X Mailers queues - 2X Pull queues - 2X Push queues #### Fritz - 1X Mastodon Web - 1X Mastodon Streaming #### Franz - 1X Schedulers - 6X Default queues - 3X Ingress queues - 5X Pull queues - 5X Push queues
Last changed by  
 
Kris Nova
392

Read more

Artifical Intelligence and Machine learning Resources

Sorted by "genre" Consciousness Video Could a Large Language Model be Conscious?Talk and slides of David Chalmers at NeurIPS2022, written version coming soon Machine Learning Book: Natural Language Processing BookNatural Language Processing with Transformers (Building with Python library: Hugging Face) Artificial Intelligence Book: Artifical Intelligence EnginesTutorial on Mathematics and Deep Learning

3/26/2023
Untitled

From @thisismissem: hey, sorry if I wasn't communicating correctly on twitch, I've experience of the mastodon codebase (former contributor), and I wrote one of the first node websocket servers; trying to help via chat whilst you stream isn't easy, more than happy to help someone debug via voice chat or on discord. I don't think the 5xx's or alerts should be caused by either rack-attack (rate limiting connections from clients because they're spamming requests, likely due to unstable mobile networks) This is the rack-attack configuration, so you can see the different rules it's applying: https://github.com/mastodon/mastodon/blob/main/config/initializers/rack_attack.rb#L57 Essentially, if node streaming talking to directly to the ruby server (rack), then it'd be (hopefully) over localhost, and therefore not rate limited based on those rules, as localhost connections are excluded from the rate limiter. (to clarify: node.js is not talking to ruby/rack, it's talking only with postgresql and redis) This is the logging setup for rack-attack, which is where the rate limit messages are logged:

1/8/2023
TLS Rotation

We need media.hachyderm.io working on CDN nodes We need hachyderm.io working on CDN nodes cdn-fremont-1 sudo certbot delete --cert-name cla.aurae.io sudo certbot delete --cert-name privilegeescalation.org sudo certbot delete --cert-name grafana.hachyderm.io certbot certonly -d media.hachyderm.io --nginx Successfully received certificate.

12/28/2022
Cooperative Corporation

Quick notes and FAQ for forming a "Cooperative Corporation". It is possible to buy *.coop domains from Identity COOP. Profits At the end of the day a cooperative can still be a for profit business entity, however the ownership of the corporation is given to the employees instead of the shareholders. See the NCBA article to understand other differences between a cooperative and a corporation. Any profits the cooperative earns are either re-invested in the company, similar to a nonprofit corporation, or distributed among its member-owners, as with a for-profit corporation.

12/12/2022
Read more from Kris Nova
Published on HackMD