Try   HackMD

ZFS Research on Alice

State of Alice after Shutting down

/dev/sdg ata-JAJS600M2TB_AA202100000031004406
/dev/sdh ata-JAJS600M2TB_AA202100000031004409

[root@alice]: /dev/disk># zpool status
  pool: data
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: resilvered 2.48G in 02:09:28 with 0 errors on Tue Nov 29 21:32:38 2022
config:

        NAME                                      STATE     READ WRITE CKSUM
        data                                      ONLINE       0     0     0
          ata-CT1000MX500SSD1_1937E21EA7CD        ONLINE       0     0     0
          mirror-1                                ONLINE       0     0     0
            ata-CT1000MX500SSD1_1914E1F7B15F      ONLINE       0     0     0
            sdf                                   ONLINE       0     0     0
          mirror-2                                ONLINE       0     0     0
            ata-JAJS600M2TB_AA202100000031004406  ONLINE       0     0     0
            ata-JAJS600M2TB_AA202100000031004409  ONLINE       0     0     0
          ata-CT1000MX500SSD1_1914E1F7DAA6        ONLINE       0     0     0

errors: No known data errors

It looks like /dev/sdg and /dev/sdh (the problematic disks) are in mirror-2.

Hypothesis

We believe that zfs is responsible for the high iowait and io utilization on NFS

Observation

Removing ZFS snapshots on Alice caused a spike in IO utilization

zfs list -t snapshot
zfs list -t all

zfs list -H -o name -t snapshot | xargs -n1 zfs destroy

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Removing ZFS snapshots

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Last changed by 
 
Kris Nova
0
116

Read more

Artifical Intelligence and Machine learning Resources

Sorted by "genre" Consciousness Video Could a Large Language Model be Conscious?Talk and slides of David Chalmers at NeurIPS2022, written version coming soon Machine Learning Book: Natural Language Processing BookNatural Language Processing with Transformers (Building with Python library: Hugging Face) Artificial Intelligence Book: Artifical Intelligence EnginesTutorial on Mathematics and Deep Learning

Mar 26, 2023
Untitled

From @thisismissem: hey, sorry if I wasn't communicating correctly on twitch, I've experience of the mastodon codebase (former contributor), and I wrote one of the first node websocket servers; trying to help via chat whilst you stream isn't easy, more than happy to help someone debug via voice chat or on discord. I don't think the 5xx's or alerts should be caused by either rack-attack (rate limiting connections from clients because they're spamming requests, likely due to unstable mobile networks) This is the rack-attack configuration, so you can see the different rules it's applying: https://github.com/mastodon/mastodon/blob/main/config/initializers/rack_attack.rb#L57 Essentially, if node streaming talking to directly to the ruby server (rack), then it'd be (hopefully) over localhost, and therefore not rate limited based on those rules, as localhost connections are excluded from the rate limiter. (to clarify: node.js is not talking to ruby/rack, it's talking only with postgresql and redis) This is the logging setup for rack-attack, which is where the rate limit messages are logged:

Jan 8, 2023
TLS Rotation

We need media.hachyderm.io working on CDN nodes We need hachyderm.io working on CDN nodes cdn-fremont-1 sudo certbot delete --cert-name cla.aurae.io sudo certbot delete --cert-name privilegeescalation.org sudo certbot delete --cert-name grafana.hachyderm.io certbot certonly -d media.hachyderm.io --nginx Successfully received certificate.

Dec 28, 2022
Postgres Migration

:warning: Public Document :warning: State of The World pgbouncer is not running anywhere freud no sidekiqs (confirmed) postgresql.service running

Dec 19, 2022
Read more from Kris Nova
Published on HackMD