This document has notes for issues found during testing the MEV workflows for fusaka in devnet-3. These issues are listed in a chronological manner Blocks being sent by the builder were ignored by the relay. This is because the builder was sending blocks which could be cancelled by setting the cancellations arg in the /relay/v1/builder/blocks?cancellation=1. To fix this, we had to enable the ENABLE_BUILDER_CANCELLATIONS environment variable in the mev-api to and set it to 1. We found a bug in the mev-relay-api where the payloads sent for Electra were being parsed as Fulu payloads. This caused sent payload to be rejected by the mev-relay as it expected an Electra payload instead of a Fulu payload. This is because the payloads for Fulu and Electra are of the same type and the parsing logic wasn't fork aware. TODO - add links. We deployed a temporary fix for fusaka-devnet-3. But we are working on a more permanant fix. We observed a lot of error="simulation failed: block is too old, outside validation window" errors in the mev-relay. These errors occured during block simulation. The error is thrown at: https://github.com/paradigmxyz/reth/blob/8402695f5e48d249d3d178c84b717b6d1693cc85/crates/rpc/rpc/src/validation.rs#L166 This error ensures that the latest header seen by the execution layer node and the parent header on top of which the block has been built upon do not deviate too much. The ideal default value is 3. This value is set in reth by the flag --rpc.eth-proof-window. The default value of this flag was set to 0. According to the validation logic we just described earlier, a default value of 0 for --rpc.eth-proof-window meant that the validation would only succeed if the latest header is a child block of the parent block which is incorrect. To fix this, we had to set --rpc.eth-proof-window=3. We started observing that the getHeader API calls were returning 204 no bid found responses back to the clients despite blocks being successfully built and sent for the given (slot, parentHash, proposerPubkey). On investigating why this was happening, we noticed that the proposer duties being queried from the beacon node for the next epoch were incorrect. We were using a prysm beacon node. We reported this issue to the prysm team and terrence raised a fix for it. Discussions can be found in this discord thread: https://discord.com/channels/595666850260713488/1398313017048830014 We noticed issues with block publishing with the prysm beacon node like error="last error: got an HTTP error response: consensus validation failed: could not get parent state: state not found in the last 8192 state roots" . This issue was raised with the prysm devs: https://discord.com/channels/595666850260713488/1398244740805300234 We unfortunately couldn't give a lot info to the prysm team, since we switched our beacon node with lighthouse. We noticed that the blocks being built were being invalidated immediately once we reached BPO1. This is because the builder was still building blocks using the Fusaka blob params which was incorrect. We fixed the builder code to pick up the blob params from the timestamp: https://github.com/flashbots/rbuilder/pull/666/files#diff-ecb850b93acffaf3d3e9622da156f6fb73848c704f0ba1556a394033678c5b27R172 We noticed that a lot of validators weren't still registered with the mev-relay despite being connected to it. We mainly noticed that lodestar, nimbus and caplin validators weren't registered with relay. Caplin is not controlled by the ethpandaops team, so we don't have much visibility over it. The Lodestar beacon nodes were sending the validator registrations to the relay but the registration api was failing with error: timestamp too early. Upon logging the validator registrations being sent by the Lodestar beacon nodes, we noticed that the timestamp in the ValidatorRegistration object were pre-genesis. Upon diving deeper with nico from Lodestar team, we noticed that Lodestar was caching pre-genesis ValidatorRegistration objects. The fix for this issue was raised in: https://github.com/ChainSafe/lodestar/pull/8088 We noticed StateRootMismatch errors when publishing a signed block from Nimbus. It appeared that the beacon state root was being incorrectly computed. Upon discussion with agnish, he confirmed a bug in the mev workflow and is fixing it.

Introduction When Eth 2.0 came to full life after the merge, it allowed the seperation of concerns of block building and block validating i.e PBS. This resulted in the development of complex MEV supply chains in the Ethereum eco-system. Below, we can have a high level look at the MEV supply chain today: MEV Supply Chain On a high level, users use a wallet of their choice to send their txs which land in either the public mempool or private order flows. Searchers pick these txs and create profitable bundles which they send to builders to bid for bundle inclusion. Builders then aggregate bundles from various searchers to create a highly profitable block which they send to MEV relays to bid for block inclusion. The key observation is that, there are two main auctions in the MEV supply chain. The Searcher<>Builder auction where searchers bid to get their bundle included in a builder block. Builders are incentivized to attract more searchers as this can increases the chances of them building a profitable block. To furthur maximize their profits, builders are now integrating with searchers.

In the Ethereum Ecosystem, PBS is currently achieved using MEV relays. Block builders construct entire blocks and send them to MEV relays. Here, a one-to-one mapping exists between a builder and the block, i.e., a block built only by one builder. In parallel block building, multiple builders build a single block. The builders building the blocks are ideally unaware of the transactions contributed by the other builders. Parallel block building improves the decentralization and censorship resistance of the builder market, where multiple builders can participate in building a block. Blocks are built for every slot and are sent to the MEV relay. The MEV relay sends the highest value block to the proposers. The value of a block is determined by the payout tx included at the end of the block. This payout tx is a simple ETH transfer from the builder's coinbase address to the proposer fee recipient address. The ETH paid out is usually a portion of the MEV and priority fees, which the builder extracts from the transactions of the submitted block. The MEV and priority fees extracted by the block builders are usually the difference in the balance of the builder fee recipient before and after executing the transactions in the block. The builder can also choose to pay from their pocket to artificially increase the value of their block and improve the inclusion rate of their txs. In out-of-protocol parallel block building, payouts become tricky. When multiple actors propose partial blocks to build a single block and are unaware of the transactions proposed by each other, the transactions not included at the top of the block can potentially revert due to the state changes because of the transactions that execute before them. These unexpected reversions can reduce the MEV and priority fees extracted by actors building the partial blocks. The key idea is that we can truly know the real value(MEV + priority fees) extracted by a block only when we combine all the partial blocks into a full block and execute the full block. This means we will have to re-think validator payouts for parallel block building.

PEPC-Boost is a proposed out-of-protocol relay design where the top-of-block and rest-of-block are auctioned separately. This contrasts with the current MEV relays, where the complete block is auctioned wholly. The problem with full-block auctions is that integrated searcher-builders can easily win the auction because they have high top-of-block extraction capabilities like CEX-DEX arbitrages. This becomes a major centralising factor in the builder market. PEPC-Boost aims to reduce the competitive advantage of these top builders by splitting the full block auction into a partial block auction where the top-of-block and rest-of-block are built separately. This article contains all the technical details of PEPC-Boost. In this article, let's discuss PEPC-Boost as a product. This talks about the following: How PEPC-Boost could fit into the Ethereum eco-system. The demand-side and supply-side of PEPC-Boost. How we could roll out PEPC-Boost to get a good mass of validators, searchers and builders connecting to it. Demand and supply side of PEPC-Boost Let's analyse PEPC-Boost as a product. We will explore where it receives demand and the supply it needs to receive to cater for the demand. We will also discuss what we need to do to increase the supply and demand side of PEPC-Boost.

Hi folks, I'll be writing my final dev updates in this document! I have enjoyed EPF thoroughly! It was filled with a lot of learning and meeting new people! TLDR I began working on the project related to decoupling blobs from payloads with the Prysm team. On further discussions with them, we realised that this problem didn't exist anymore since the protocol expected KZG commitments of a blob tx to be computed out of protocol before being sent to the Ethereum protocol. I reached out to Barnabe Monnot from RIGS at EF and started to work on PEPC-Boost. As part of EPF, I designed PEPC-Boost, implemented it, documented it and deployed it to Goerli successfully. Project Abstract I worked on PEPC-Boost as part of EPF. A basic spec of PEPC-Boost, as proposed by Barnabe Monnot, can be found in the PEPC FAQ doc.

Hey folks, this is my week 15 updates. As EPF is closing and with my project being finished as per the proposal I had written. I spent more time this week writing and preparing the PPT for EPF day. This will be a much smaller dev update. Project Updates I wrote 2 hackmd articles on payouts in partial block auctions and thinking of PEPC-Boost as a product. Please find the links below Thoughts on PEPC-Boost as a product: https://hackmd.io/@bchain/rk8sepAG6 Relay payouts With parallel block auctions: https://hackmd.io/@bchain/Sy5qtmRf6 They are not reviewed by my mentors but were the thoughts I had on some open problems w.r.t PEPC-Boost.

Hey folks, This is my updates for week 13 and 14 combined! I was fairly busy at my day job so I thought it would be better if I combined my updates. TLDR; PEPC-Boost successfully proposed some blocks on Goerli! I allowed searchers to start bidding for multiple slots for PEPC-Boost and deployed it to Goerli. Blocks were successfully proposed. I started to think of PEPC-Boost from a product perspective rather then just a technical perspective. This involved reading a bunch of articles and brainstorming. I also implemented a feature in PEPC-Boost to allow rebuilding latest submitted ToB bids with previous high value RoB bids. I am currently writing an article highlighting my thoughts. I wrote a kurtosis package to launch an adminer instance to visualise postgres payloads. I integrated it with the kurtosis ethereum devnet environment. It is currently being reviewed to be upstreamed. Project Updates PEPC-Boost proposed it's first blocks! It is an example of a parallel block auction on Ethereum. Below is an example slot at which a block was proposed from PEPC-Boost: https://prater.beaconcha.in/slot/6783851 I expanded PEPC-Boost to allow searchers to submit more txs to include in the top of block

Hey folks, here are my week 12 updates! TLDR; I published a blog post on PEPC-Boost on twitter which was fairly well received! I started working on expanding the ToB slots to multiple pools of which I am done with implementation. Currently I am working on tests. Also I completed my setup on Goerli and got 24K validators connected to my setup! My setup has successfully proposed a lot of blocks too! Project Work We published a blog on PEPC-Boost on twitter which you can find at: https://twitter.com/BharathVedarth1/status/1712156150102446356 The post got good amount of traction and was fairly well received. I got connected to Max Resnick from SMG who was the inspiration behind PEPC-Boost. I started bugging him with questions too!! I am working on expanding the ToB with multiple slots. I have finished most of the work and am currently finishing up unit tests:

Contents Abstract Glossary Introduction System Architecture Implementation-Specific Details Current State Future Work Demo Video

Contents Abstract Glossary Introduction System Architecture Implementation-Specific Details Current State Future Work Demo Video

Hey folks, here is my week 11 updates! This is was a rather unproductive week compared to the previous weeks since I had to spend quite a bit of time finalized and applying for my visa to Turkey for DevConnect which I am happy is finally done! I am currently waiting for the final verdict on my visa from the consulate! TLDR of this week, I have implemented a ToB tx simulation RPC to make the ToB tx validation in PEPC-Boost robust and have integrated that with PEPC-Boost relayer. I have written a blog on PEPC-Boost which my mentors are currently reviewing. I have been studying about Skip Protocol which is sort of like PEPC for cosmos. Paritosh from EF is willing to connect some goerli validators to my relay which is exciting! Project Updates I implemented an RPC in the execution layer to robustly validate ToB txs sent to the relayer. This makes the ToB tx validations much more robust. Below are the PRs to implement the ToB tx validation RPC

Hi folks, here are my week 10 updates. Week 10 was a fairly interesting week. TLDR; I implemented state interference checks for goerli. Finalized the V0 features of PEPC-Boost. Started deploying to goerli. Started maintaining a dedicated documentation repo. Project Work I implemented an initial version of validator payouts. It's a pretty simple version which derives from the existing payout solutions in MEV relays. It is implemented in the following PRs: Code: https://github.com/bharath-123/pepc-boost-relay/pull/19 Tests: https://github.com/bharath-123/pepc-boost-relay/pull/20 I also implemented state interference checks in goerli for a uniswap v3 eth/usdc swap.

Hi folks! I ll be merging my 2 weeks updates into one this time! TLDR; I have been consistently working on PEPC-Boost. I have implemented most of the functionality for an MVP and have tested it out on devnet! There are a lot of things to be done to build a complete production ready system but I have been able to implement most of the functionality and test it on a custom devnet. I have also tried my best to document all the code by splitting the entire codebase in small logical PRs. Project I worked on re-designing PEPC-Boost to a much simpler architecture documented here. There was an important property of the system which I had misunderstood. I initially thought that seperate builders would build out ToB blocks and RoB blocks. Upon furthur discussion with my mentors, the idea is for searchers to DIRECTLY send the transactions they want to include in the top of the block. Understanding this property of the system made me re-design the entire system into a much simpler and modular system! Once I was done with the design, I began implementing it since it is the best way to iterate on your system design. I have raised a total 17 PRs across the relayer and builder with my implementation! Below are the first PRs of the relayer and builder which are the introduction PRs which contain much detail about the system and links all the subsequent PRs reviewers are encouraged to look at! PEPC-Boost Relay Introductory PR

Hi folks! In week 7, the main things I worked on were re-designing PEPC-Boost to a much simpler architecture after discussion with mentors and implementing the design. I also worked on a PR to fix a bug in prysm w.r.t validator registrations with MEV Relayers. I also had a PR merged in kurtosis which parameterizes the builder and boost image in the project. Project Updates Upon furthur discussion with Barnabe on PEPC-Boost, We realised we can simplify the architecture by changing a flows. Initially, we had designed the system in a way that the TOB bids were built by the builders where they had to search the mempool for the specific txs. But Barnabe suggested that searchers would directly submit txs to the relayer which they wanted to include in the TOB. We would expose an endpoint which would basically be a TOB bid endpoint for a given slot and parent hash. I also removed some parts of the design like adding a new field in the ValidatorRegistration object since that requires consensus layer changes. The latest design is documented in: https://docs.google.com/document/d/1DLse86yGLId2idQUIJbaiGxY_RxMxz3LhtdinbA3q4Q/edit

Hi folks, This is my week 6 updates! TLDR; Got pretty positive feedback on my PEPC-Boost designs and given the greenlight by all my mentors to start the implementation of PEPC-Boost! Started working on improving the relay-specs in flashbots Started implementing PEPC-Boost and successfully implemented a FULL_BLOCK commitment. Started using kurtosis eth2-package to rapidly develop PEPC-Boost. I also ended up contributing a PR to kurtosis eth2-package! Project Work I submitted the design of PEPC-Boost to my mentors who gave positive feedback and suggested to being implementing it.

This document contains notes being taken down during implementation of PEPC-Boost. This will be built on capella fork rather then deneb, since deneb is still in a W.I.P Repos Prysm: https://github.com/bharath-123/prysm PEPC-Boost Relay: https://github.com/bharath-123/pepc-boost-relay Builder: https://github.com/bharath-123/pepc-boost-builder Base commits

EPF-Week 5 updates This document contains my updates for week five! I started working on my project to design an out-of-protocol implementation of PEPC-Boost! I'll be dividing my updates into two sections from now on. The sections will be: Project Work: This will include updates on my project work PRs: This will include my updates on any open-source PRs I am working on I have made good progress on my project work. I have written a design doc highlighting the design of PEPC-Boost.

This document contains a project proposal to implement a POC of PEPC-Boost. Motivation Proposer-builder separation is a model which has been implemented out of protocol in Ethereum using a trusted MEV Relay. The essence of the model is that the proposer of the block can defer block building activities to an external builder. We can re-phrase this as a commitment to say that "the builder has a commitment to the proposer to produce a block whose value is more than the block which they can produce locally". Currently, there are designs which aim to enshrine proposal builder seperation in protocol to eliminate the trusted relay and use the ethereum chain for trust. PEPC and PEPC-Boost is one such proposed design which is in its very early phases. Continuing from the above "commitment", We look at PEPC. PEPC or Protocol-Enforced-Proposer-Commitment, is essentially a generalization of the proposer builder seperation where we allow the proposer to enter into any such commitment with the builder. Some example commitments could be: Full Block Auction which is the current PBS Partial Block Auction where the proposer suggests some transactions to be included in the block. Parallel Block Auctions where multiple builders suggest transactions to one block.

During Week 4, I acheived the following: Wrote a couple of articles related to Ethereum censorship, proposer builder seperation and EIP-4844 Worked on a PR in prysm which I still need to close as I got more feedback on it Studied about my new project and wrote down a project proposal which was approved by all my mentors!! Writings I wrote the following articles during my studies last week: Thoughts on Seperation of blobs from payload : I wrote this articles when I was studying more about my initial project of Seperation of blobs from payload. Through these studies, I discovered that the initial problem statement we set out to solve was not actually a problem now. That was because the initial problem statement was to seperate out blobs from payloads because home stakers wouldn't be able to compute KZG commitments and proofs of blobs on time to propose a block. But in future versions of KZG, the burden of computation of commitments and proofs have been shifted to the sender of the blob txs. Post this article, I had to pivot my project to a new project.

This article explores the idea of segmenting an Ethereum block into TOB(Top of block) and ROB(Rest of block) and understanding how it related to MEV centralization. Introduction TODO