Nethermind client Consensus Issue Jan 2024: Post-mortem

Issue Date: Sun 21/01/2024 # Issue description On January 21st, 2024, at 06:06:23 PM UTC on Ethereum Mainnet, the Nethermind client versions 1.23.0, 1.24.0, 1.25.0, and 1.25.1 processed a valid block as invalid. As a result, all validators using these versions stopped attesting to blocks, leading to a loss of attestation rewards and the incurrence of missed attestation penalties. # Cause of the issue A revert opcode in a smart contract [transaction](https://etherscan.io/tx/0x26c139ef4a0f645b1b114f45005b65fd8cd1386731c6f36369e96f9ba304f254) caused an unhandled OverflowException in the code that tried to decode the revert message. ``` 00000001000000000000000000000000000000000000000012a9d65e7d180cfcf3601b6d00000000000000000000000000000000000000000000000000000001000276a400000000000000000000000000000000000000000000000000000000000000a0000000000000000000000000000000000000000000000000000000000000006a000000000300000000000115859c410282f6600012efb47fcfcad4f96c83d4ca676842fb03ef20a4770000000015f762bdaa80f6d9dc5518ff64cb7ba5717a10dabc4be3a41acd2c2f95ee22000012a9d65e7d180cfcf3601b6df0000000000000185594dac7eb0828ff000000000000000000000000 UTF8: ��©Ö^}üó`m��v¤�� j��Aö`�ï´ÏÊÔùlÔÊghBûï ¤w��÷b½ªöÙÜUÿdË{¥qzÚ¼Kã¤Í,/î"��©Ö^}üó`mð��UÚÇë(ÿ�� ``` This bug was introduced by a Pull Request (PR) to [extract revert messages](https://github.com/NethermindEth/nethermind/pull/6226) in order to decode [Solidity 0.8.0+ checked panic error codes](https://soliditylang.org/blog/2020/12/16/solidity-v0.8.0-release-announcement/) to a more understandable message for the end user. The PR was part of JSON-RPC standardization. While parsing the revert message, the unhandled overflow exception caused the transaction to fail, consuming all of its gas. As a result, the client produced a different block proof (block hash) than the rest of the chain and caused a divergence in consensus. Parsing the revert message should not have affected the overall block, regardless of the message. This was fixed by reverting the PR and has since been fixed more elegantly to restore the UX improvement that the PR was attempting to do. # Issue discovery The issue was first discovered by Discord users. Our team lead Marek Moraczyński also noticed it and reported it through internal communication channels. This was confirmed by Mateusz Jędrzejewski, a member of our DevOps team, who identified it as an issue for all Nethermind-based validators under management. # Investigation and resolution All available members from the Nethermind client team jumped on a call to investigate the issue. Notably, Parithosh from the Ethereum Foundation DevOps and Marius from the Go-Ethereum client team made themselves available and joined the call to help. The investigation began by gathering traces in the affected block, identifying the faulty transaction and searching for a possible change that might have caused the issue. The faulty change was suspected but not immediately confirmed as the culprit. An in-depth analysis of the traces led to the identification of the opcode in the EVM that has caused the issue, confirming the initial suspicions about that change. A Docker image was created with the faulty change reverted. Upon testing, it resolved the issue and all nodes with the fix were successfully recovered. At this point, we started the release process and the fix was deployed within 4-5 hours of the initial report. # Moving forward We cannot stress enough the importance of client diversity, and are encouraged by the community’s positive response in improving the health of the network. In light of this situation, we’ll be enforcing more requirements on the EVM and consensus-touching changes, and exercising more due diligence toward them. Additionally, we're committed to improving our response capabilities through the development of internal tools. These tools would help us expedite the identification, analysis, and resolution of potential issues.