# Teku Incident Report - Failed to load proposer config 08 October 2024 ## Description Teku 24.10.0 version had a regression bug that caused an error when loading a user-supplied proposer config file. The error was preventing teku from starting up. Users had to downgrade back to 24.8.0 to be able to start Teku again. The problem was caused by a recent refactoring in our Validator API code. Unfortunately, our testing code used a different code path than production to initialize `ProposerConfigLoader`, preventing our automated tests from catching this issue. Only our testing fleet in the Sepolia network had a node using this option. This was an oversight on our part, as our Sepolia fleet is not used as part of our release testing process. [Link to GitHub issue](https://github.com/Consensys/teku/issues/8686) A minor unrelated issue was that the release notes for 24.10.0 had the wrong hash for the tar.gz file. This has been reported by one user. ## User Impact Any users running Teku with the option `--validators-proposer-config` ([docs](https://docs.teku.consensys.io/development/how-to/configure/use-proposer-config-file)). A few users reported the issue on Discord. ## Timeline (all times in UTC) - `2024-10-08 09:29` Release announced - `2024-10-08 14:14` First user reported issue on Discord - `2024-10-08 14:56` Issue confirmed by the Teku team - `2024-10-08 15:14` Announcement posted on #teku-public and #besu-teku - `2024-10-08 16:06` Raised PR fixing the issue - `2024-10-09 07:00` Teku 24.10.1 released ## Root Cause Analysis Error log reported by users: ``` FATAL - Failed to load proposer config from 'file:/var/lib/teku/proposer_setting.json' - Cannot find a (Map) Key deserializer for type [simple type, class org.apache.tuweni.bytes.Bytes48] at line: 1, column: 1 ``` This is due to missing `ObjectMapper` serialisation/deserialisation modules, which are not registered for the production `ProposerConfigLoader`. ## Resolution The issue was fixed by ensuring that the needed serializers/deserializers were mapped correctly on the production ProposerConfigLoader. [Link to Pull Request fixing issue](https://github.com/Consensys/teku/pull/8687) The fix included a unit test to ensure that mappers are present to prevent this bug from happening again. One of our nodes in Teku’s Holesky fleet has been updated to use –validator-proposer-config. This means the feature will be exercised in our release testing for every new release. A new version of Teku 24.10.1 will be released with the fix. ## Lessons Learned We learned that our unit tests for `ProposerConfigLoader` were not exercising the same path that is used in production, and added unit tests to ensure we catch if we ever don't register the serialisation/deserialisation modules. Another thing we want to do is to extend our acceptance test suite to include a test specific for this feature. This will help us catch any potential startup issue related to the feature (for example, invalid config handling or any other realated issue). One of our nodes in our Holesky fleet has also been updates to use the `--validators-proposer-config` flag, so now we will find out any other issues with this feature during our testing for a release. Last but not least, we identified a few issues with our hotfix release process and we are discussing a few ways we can improve it in the future to ensure better communication and less time in-between fixing the bug and pushing a new version of Teku.