# Teku 25.9.0 upgrade incident

When upgrading to Teku 25.9.0, several users reported that they were no longer able to participate in duties.

## Log evidence
The logs showed duties were not being performed. On one node we were able to see an exception

...
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
	at tech.pegasys.teku.networking.eth2.gossip.encoding.SnappyBlockCompressor.compress(SnappyBlockCompressor.java:54) ~[teku-networking-eth2-25.9.0.jar:25.9.0]
	at tech.pegasys.teku.networking.eth2.gossip.encoding.SszSnappyEncoding.encode(SszSnappyEncoding.java:38) ~[teku-networking-eth2-25.9.0.jar:25.9.0]
	at tech.pegasys.teku.networking.eth2.gossip.subnets.AttestationSubnetSubscriptions.lambda$gossip$0(AttestationSubnetSubscriptions.java:90) ~[teku-networking-eth2-25.9.0.jar:25.9.0]
	at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) ~[?:?]
	... 46 more
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: /tmp/snappy-1.1.10-2ffe9e25-4684-44ae-80eb-bbb84ab6689a-libsnappyjava.so: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory [in thread "GossipRouter-event-thread-0"]
...

## Root cause

This tracked back to a snappy change that was included in release `org.xerial.snappy:snappy-java:1.1.10.8`, https://github.com/xerial/snappy-java/pull/632

This change is determining libraries to load, and detecting musl based on the existence of a file: `/lib/ld-musl-x86_64.so.1`

After detection, it was observed that it would attempt to load a different file: `/lib/libc.musl-x86_64.so.1`.

This file is not present in a node that is based on ubuntu and has grafana installed, which is a reasonable representation of solo stakers potentially.

## Possible solutions

### work around the issue

Investigated keeping this point release. This would ultimately require that users faced with this problem install `musl-dev` and create a softlink manually. This seems like a sub-optimal fix.

### Re-release on the old snappy
Downgrading snappy seems like the 'right' fix.

Raised PR to downgrade: https://github.com/Consensys/teku/pull/9839
Raised Issue on snappy: https://github.com/xerial/snappy-java/issues/692

We will release 25.9.1 with the downgraded snappy to mitigate this issue.

## Advice

If you are looking to upgrade to Teku 25.9.0, check for the files:

ls -l /lib/libc.musl-x86_64.so.1 /usr/lib/ld-musl-x86_64.so.1

If you have both of these files, or neither, you will be fine to use the 25.9.0 release.

If you have only got `/usr/lib/ld-musl-x86_64.so.1`, then you should defer upgrading until Teku 25.9.1 is made available.