---
# System prepended metadata

title: Fusaka Devnet-3 MEV pipeline testing notes

---

# Fusaka Devnet-3 MEV pipeline testing notes

This document has notes for issues found during testing the MEV workflows for fusaka in devnet-3. These issues are listed in a chronological manner

1. Blocks being sent by the builder were ignored by the relay. This is because the builder was sending blocks which could be cancelled by setting the cancellations arg in the `/relay/v1/builder/blocks?cancellation=1`. To fix this, we had to enable the `ENABLE_BUILDER_CANCELLATIONS` environment variable in the mev-api to and set it to 1.
2. We found a bug in the mev-relay-api where the payloads sent for Electra were being parsed as Fulu payloads. This caused sent payload to be rejected by the mev-relay as it expected an Electra payload instead of a Fulu payload. This is because the payloads for Fulu and Electra are of the same type and the parsing logic wasn't fork aware. TODO - add links. We deployed a temporary fix for fusaka-devnet-3. But we are working on a more permanant fix.
3. We observed a lot of `error="simulation failed: block is too old, outside validation window"` errors in the mev-relay. These errors occured during block simulation. The error is thrown at: https://github.com/paradigmxyz/reth/blob/8402695f5e48d249d3d178c84b717b6d1693cc85/crates/rpc/rpc/src/validation.rs#L166 This error ensures that the latest header seen by the execution layer node and the parent header on top of which the block has been built upon do not deviate too much. The ideal default value is 3. This value is set in reth by the flag `--rpc.eth-proof-window`. The default value of this flag was set to 0. According to the validation logic we just described earlier, a default value of 0 for `--rpc.eth-proof-window` meant that the validation would only succeed if the latest header is a child block of the parent block which is incorrect. To fix this, we had to set `--rpc.eth-proof-window=3`. 
4. We started observing that the `getHeader` API calls were returning 204 `no bid found` responses back to the clients despite blocks being successfully built and sent for the given (slot, parentHash, proposerPubkey). On investigating why this was happening, we noticed that the proposer duties being queried from the beacon node for the next epoch were incorrect. We were using a prysm beacon node. We reported this issue to the prysm team and terrence raised a fix for it. Discussions can be found in this discord thread: https://discord.com/channels/595666850260713488/1398313017048830014
5. We noticed issues with block publishing with the prysm beacon node like `error="last error: got an HTTP error response: consensus validation failed: could not get parent state: state not found in the last 8192 state roots" `. This issue was raised with the prysm devs: https://discord.com/channels/595666850260713488/1398244740805300234 We unfortunately couldn't give a lot info to the prysm team, since we switched our beacon node with lighthouse. 
6. We noticed that the blocks being built were being invalidated immediately once we reached BPO1. This is because the builder was still building blocks using the Fusaka blob params which was incorrect. We fixed the builder code to pick up the blob params from the timestamp: https://github.com/flashbots/rbuilder/pull/666/files#diff-ecb850b93acffaf3d3e9622da156f6fb73848c704f0ba1556a394033678c5b27R172
7. We noticed that a lot of validators weren't still registered with the mev-relay despite being connected to it. We mainly noticed that lodestar, nimbus and caplin validators weren't registered with relay. Caplin is not controlled by the ethpandaops team, so we don't have much visibility over it. The Lodestar beacon nodes were sending the validator registrations to the relay but the registration api was failing with `error: timestamp too early`. Upon logging the validator registrations being sent by the Lodestar beacon nodes, we noticed that the timestamp in the `ValidatorRegistration` object were pre-genesis. Upon diving deeper with nico from Lodestar team, we noticed that Lodestar was caching pre-genesis ValidatorRegistration objects. The fix for this issue was raised in: https://github.com/ChainSafe/lodestar/pull/8088
8. We noticed `StateRootMismatch` errors when publishing a signed block from Nimbus. It appeared that the beacon state root was being incorrectly computed. Upon discussion with agnish, he confirmed a bug in the mev workflow and is fixing it.
9. We noticed logs like `INFO Cancelling building job reason="last block number" last_block_number=45404 block=45404 payload_id=868049159` in the rbuilder node. We initially thought something was off. This was not an error but indicated that block building was cancelled because a block had already arrived for the slot. We faced a lot of block building cancellations because there were no txs being sent during that time period.



We need to test the optimistic relaying flows too.