Hyperledger Besu Gas Leak (22.4.x)

# Hyperledger Besu Gas Leak (22.4.x) ## Executive Summary Because of a flaw in handling unsigned data as signed data a properly coded smart contract can create a function call that will return more gas than was passed in. This excess gas can be used to create a loop that is in effect infinite. Networks containing only Besu validators will halt when this code is executed on-chain or considered for block inclusion. This bug was patched in Hyperledger Besu version 22.7.1 and exists in versions 22.4.0 to 22.7.0. [CVE-2022-36025](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-36025) ## Gas Call Bug The origins of the bug comes from a mis-handling of unsigned data as signed data. As part of supporting [EIP-4803 Limit transaction gas to a maximum of 2^63-1](https://eips.ethereum.org/EIPS/eip-4803) Besu migrated the internal types of `Gas` from a `UInt256` object to a native `long` field, which is a signed value that can support the full range of transaction gas. In instances where the gas needed to represent a higher value Besu clamped the values to 2^63-1, aka `Long.MAX_VALUE`. Because all EVM executions will have the intrinsic gas costs deducted this clamped value will always exceed any observed gas value. [The original code](https://github.com/hyperledger/besu/blob/release-22.4.0/evm/src/main/java/org/hyperledger/besu/evm/operation/CallOperation.java#L40) used [a conversion](https://github.com/apache/incubator-tuweni/blob/2.2/units/src/main/java/org/apache/tuweni/units/bigints/UInt256.java#L828-L833) that would take any 64 bit or less value and place it directly to a signed `long`. Any 65-256 bit sized number would result in an exception that would be trapped out and clamped to the max value. But any value of exactly 64 bits would become a negative signed `long` value. ## Fuzzing This conversion error was found by the fuzzer run by Martin Holst Swende. The fuzzing that discovered the error only saw a difference in a couple of lines of trace data. The end result of the fuzzed program was a revert of the call frame including the erroneous call and both executions resulted in a top level revert. The state roots at the end of the transaction were the same, so had this been executed in a production block the consensus would not have been broken. However the Besu EVM reverted for different reason than other clients. Without the use of [EIP-3155 standard traces](https://eips.ethereum.org/EIPS/eip-3155) there is a likelihood this fuzzing finding would have been missed. This fuzzing difference, while not found in a form that broke consensus, had the potential to grow into a much more interesting exploit. The differences could surface in a way that presented a fairly standard consensus break. But there is a much deeper potential for this error. ## Leveraging the Exploit In order to elevate this to a chain-halting bug a deliberately crafted call was needed, involving some interactions with the [EIP-150](https://eips.ethereum.org/EIPS/eip-150eip-150) "all but one 64th" rule and reserving a portion of available gas for the calling contract. First, a call-series operation (`CALL`, `STATICCALL`, `CALLCODE`, `DELEGATECALL`) is made to any contract with a requested gas value between 2^64-1 and 2^63-1. The called contract is then started with the smallest of the gas requested and the gas left, with the gas requested evaluated as a negative number and hence being selected. This call will fail with an out of gas halt as the gas meter starts at less than zero. Upon returning from the call the original frame will still have the 1/64th gas reserved, which is now a positive number because of the sign converion rules. The caller now has more gas than they started with before the call. To complete the exploit a loop is executed to consume the returned gas. More elaborate contracts can structure a loop to make the gas infinite, but 8 quintillion gas should be enough for an effective attack. Here is a standard trace showing the setup to the call, the failed call, and the resulting effectively infinite loop: ``` ... {"pc":22,"op":244,"gas":"0x42d","gasCost":"0x67","memory":"0x","memSize":32,"stack":["0x0","0x1","0x2","0x3","0x4","0x5","0xf1","0x8fffffffffffffff"],"returnData":null,"depth":1,"refund":0,"opName":"DELEGATECALL","error":""} {"pc":0,"op":96,"gas":"0x8fffffffffffffff","gasCost":"0x3","memory":"0x","memSize":0,"stack":[],"returnData":null,"depth":2,"refund":0,"opName":"PUSH1","error":"Out of gas"} {"pc":23,"op":91,"gas":"0x70000000000003c7","gasCost":"0x1","memory":"0x","memSize":32,"stack":["0x0","0x1","0x0"],"returnData":null,"depth":1,"refund":0,"opName":"JUMPDEST","error":""} {"pc":24,"op":96,"gas":"0x70000000000003c6","gasCost":"0x3","memory":"0x","memSize":32,"stack":["0x0","0x1","0x0"],"returnData":null,"depth":1,"refund":0,"opName":"PUSH1","error":""} {"pc":26,"op":86,"gas":"0x70000000000003c3","gasCost":"0x8","memory":"0x","memSize":32,"stack":["0x0","0x1","0x0","0x17"],"returnData":null,"depth":1,"refund":0,"opName":"JUMP","error":""} {"pc":23,"op":91,"gas":"0x70000000000003bb","gasCost":"0x1","memory":"0x","memSize":32,"stack":["0x0","0x1","0x0"],"returnData":null,"depth":1,"refund":0,"opName":"JUMPDEST","error":""} {"pc":24,"op":96,"gas":"0x70000000000003ba","gasCost":"0x3","memory":"0x","memSize":32,"stack":["0x0","0x1","0x0"],"returnData":null,"depth":1,"refund":0,"opName":"PUSH1","error":""} {"pc":26,"op":86,"gas":"0x70000000000003b7","gasCost":"0x8","memory":"0x","memSize":32,"stack":["0x0","0x1","0x0","0x17"],"returnData":null,"depth":1,"refund":0,"opName":"JUMP","error":""} {"pc":23,"op":91,"gas":"0x70000000000003af","gasCost":"0x1","memory":"0x","memSize":32,"stack":["0x0","0x1","0x0"],"returnData":null,"depth":1,"refund":0,"opName":"JUMPDEST","error":""} ... ``` The impact of this infinite loop is any evaluation of the contract will execute effectively forever, and more elaborate contracts will truly execute forever. When a node considers the transaction for inclusion or verifies a block containing the transaction the verification code will not return and will induce a liveness failure as a block will not be produced or verified. These failures will cascade to other alternate block producers until enough nodes cause the network to totally halt. A similar setup was observed in a Hedera integration network with the same EVM code that similarly halted a leaderless consensus algorithm. ## Fixing the Exploit The [fixed code](https://github.com/hyperledger/besu/blob/release-22.7.1/evm/src/main/java/org/hyperledger/besu/evm/operation/AbstractCallOperation.java#L63) uses a [different conversion method](https://github.com/hyperledger/besu/blob/release-22.7.1/evm/src/main/java/org/hyperledger/besu/evm/internal/Words.java#L92-L106) that will "clamp" overflow values to the maximum expected values avoiding the signed translation issues. This mitigates the attack at the earliest possible point in the EVM evaluation. Losing the exact value will not impact evaluation because the value is then immediately compared to the actual gas remaining with the lower value chosen. The clamped value will always be equal or less than the EIP limited value. Here we can observe the correct value being passed into the child call, and the correct remaining gas being selected. We also observe the reserved gas returned being correctly calculated. ``` ... {"pc":22,"op":244,"gas":"0x2c3","gasCost":"0x2bf","memory":"0x","memSize":32,"stack":["0x0","0x1","0x2","0x3","0x4","0x5","0x0","0x8fffffffffffffff"],"returnData":null,"depth":1,"refund":0,"opName":"DELEGATECALL","error":""} {"pc":0,"op":0,"gas":"0x4","gasCost":"0x0","memory":"0x","memSize":0,"stack":[],"returnData":null,"depth":2,"refund":0,"opName":"STOP","error":""} {"pc":23,"op":91,"gas":"0x4","gasCost":"0x1","memory":"0x","memSize":32,"stack":["0x0","0x1","0x1"],"returnData":null,"depth":1,"refund":0,"opName":"JUMPDEST","error":""} {"pc":24,"op":96,"gas":"0x3","gasCost":"0x3","memory":"0x","memSize":32,"stack":["0x0","0x1","0x1"],"returnData":null,"depth":1,"refund":0,"opName":"PUSH1","error":""} {"pc":26,"op":86,"gas":"0x0","gasCost":"0x8","memory":"0x","memSize":32,"stack":["0x0","0x1","0x1","0x17"],"returnData":null,"depth":1,"refund":0,"opName":"JUMP","error":"Out of gas"} INSUFFICIENT_GAS ``` This was fixed in release 22.7.1, which was the release all mainnet nodes needed to update to be prepared for the merge. Any remaining active nodes necessarily have patched code as the merge has passed. Active networks Besu maintainers were aware of have upgraded vulnerable nodes to patched software. If you have nodes in the vulnerable range it is recommended you upgrade to a current version as soon as possible. ## Lessons Learned The biggest lesson demonstrated by this exploit is that the comparison of trace data in a fuzzing execution catches more bugs than simply comparing the end results. This may be only relevant to smart contract execution where fully deterministic evaluations are essential. While fuzzing may have come across a contract that would allow the gas leak comparing the traces found the root of the problem much quicker.