# Engine API: A Visual Guide The Engine API is the interface that allows communication between the execution and the consensus layer of an Ethereum node. Since the merge, this API has become a vital piece of Ethereum's architecture. It coordinates important tasks such as block validation and block proposal. Even with its importance, it's hard to find resources beyond the [Engine API spec](https://github.com/ethereum/execution-apis/blob/main/src/engine) when learning about it. Trying to understand the API just by reading the spec is a challenging task. For that reason, in this article, I'll explain the most important flows of the Engine API. I'll indicate how those flows can fail, where the spec dictates them, and how they fit together. *This article is part of my [EPF project](https://github.com/eth-protocol-fellows/cohort-four/blob/master/projects/portal-network-validator.md).* ## Considerations - This is based on the [Shanghai spec](https://github.com/ethereum/execution-apis/blob/main/src/engine/shanghai.md). - The validator software is shown as part of the consensus layer client. - Every flow here assumes the validator is building blocks locally. - If you want the rendered diagrams alone, you can find them [here](https://github.com/danielrachi/diagrams/tree/main/rendered/engine-api/sequence-diagrams). ## Functions tl;dr. Here's a brief explanation of the functions we will encounter along the way: - `engine_exchangeCapabilities`: Used to exchange the Engine API methods supported by each client. - `engine_forkchoiceUpdatedV2`: Updates the fork choice of the Execution Layer Client. Also used to start a payload build process. - `engine_getPayloadV2`: Used to retrieve an `execution_payload` from a build process started in a past `engine_forkchoiceUpdatedV2` call. - `engine_newPayloadV2`: Used by the consensus layer client to send an `execution_payload` to the execution layer client for it to validate it. ## Node startup When you run the command to start a beacon node, [two Engine API endpoints are called](https://hackmd.io/@danielrachi/S1u2Veflp): `engine_exchangeCapabilities` and `engine_forkchoiceUpdatedV2`. Ideally, the following would be the communication between the *Execution Layer Client* (EL) and the *Consensus Layer Client* (CL): ```sequence Title: Node Startup, Success CL -> EL: engine_exchangeCapabilities(engine methods supported by CL) EL --> CL: engine methods supported by EL CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, null) EL --> CL: {payloadStatus: {status: VALID, ...}, payloadId: null} ``` *Note how the second argument of `engine_forkchoiceUpdatedV2` is null. This tells the EL not to start a payload build process.* ### EL is syncing There is no guarantee that both clients will finish their sync at the same time. For that reason, the CL could call an endpoint when the EL is still syncing, and that leads to the following scenario: ```sequence Title: Node Startup, EL is Syncing CL -> EL: engine_exchangeCapabilities(engine methods supported by CL) EL --> CL: engine methods supported by EL CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, null) Note over EL: I don't have all the \n necessary data available EL --> CL: {payloadStatus: {status: SYNCING, ...}, payloadId: null} ``` ## Block Building When a validator is selected to propose a block, the ideal communication between both clients is as follows: ```sequence Title: Block Building, Success Note over CL: Validator is assigned \n to propose Note over CL: Fill BeaconBlock up to \n execution_payload CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes) EL --> CL: {payloadStatus: {status: VALID, ...}, payloadId: buildProcessId} Note over EL: Build execution_payload CL -> EL: engine_getPayloadV2(PayloadId) EL --> CL: {executionPayload, blockValue} Note over CL: Put the ExecutionPayload \n on the BeaconBlock Note over CL: Compute state_root Note over CL: Propagate block ``` *Note how this time, a `PayloadAttribues` is passed as the second argument when calling `engine_forkChoiceUpdatedV2`. Then a `payloadId` is part of the response and the payload build process is started.* Let's see how this process can fail. ### `engine_forkchoiceUpdatedV2` fails #### Payload referenced by `forkchoiceState.headBlockHash` is invalid The `forkchoiceState` struct has a field called `headBlockHash`. Before updating its fork choice, the EL must check that the payload referenced by the `headBlockHash` is valid. The process to define the validity of the payload is defined in the engine API Paris spec, [here](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#payload-validation). If this process concludes that the referenced payload is invalid, this happens: ```sequence Title: Block building fails, INVALID payload Note over CL: Validator is \n assigned to propose Note over CL: Fill BeaconBlock \n up to execution_payload CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes) Note over EL: Start payload \n validation process Note over EL: Payload is invalid EL --> CL: {payloadStatus: {status: INVALID, ...}, payloadId: null} ``` #### Invalid `PayloadAttributes` Similar to how the `ForkChoiceState` needs to meet certain criteria to be considered valid, `PayloadAttributes` must fill some requirements to be valid. From point 7 of the [`engine_forkchoiceUpdatedV1` spec](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#specification-1): > Client software MUST ensure that `payloadAttributes.timestamp `is greater than timestamp of a block referenced by `forkchoiceState.headBlockHash`. If this condition isn't held client software MUST respond with `-38003: Invalid payload attributes` and MUST NOT begin a payload build process. In such an event, the `forkchoiceState` update MUST NOT be rolled back. ```sequence Title: Block Building Fails, Invalid PayloadAttributes Note over CL: Validator is assigned \n to propose Note over CL: Fill BeaconBlock up to \n execution_payload CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes) Note over EL: PayloadAttributes is not \n valid EL --> CL: -38003: Invalid payload attributes ``` #### Wrong `PayloadAttributes` version Besides being valid, `PayloadAttributes` needs to be of the right version. From the [`engine_forkchoiceUpdatedV2`](https://github.com/ethereum/execution-apis/blob/main/src/engine/shanghai.md#engine_forkchoiceupdatedv2) spec: > - `PayloadAttributesV1` MUST be used to build a payload with the timestamp value lower than the Shanghai timestamp, > - `PayloadAttributesV2` MUST be used to build a payload with the timestamp value greater or equal to the Shanghai timestamp, > - Client software MUST return `-32602: Invalid params` error if the wrong version of the structure is used in the method call. If this check fails, this occurs: ```sequence Title: Block Building Fails, Wrong PayloadAttributes Version Note over CL: Validator is assigned \n to propose Note over CL: Fill BeaconBlock up to \n execution_payload CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes) Note over EL: Wrong PayloadAttributes \n version EL --> CL: -32602: Invalid params ``` #### `ForkChoiceState` is invalid The `ForkChoiceState` passed as an argument on the FCU call needs to meet certain criteria to be considered valid. The criteria are stated in point 6 of the spec of [`engine_forkchoiceupdatedV1`](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#specification-1): > Client software MUST return `-38002: Invalid forkchoice` state error if the payload referenced by `forkchoiceState.headBlockHash` is `VALID` and a payload referenced by either `forkchoiceState.finalizedBlockHash` or `forkchoiceState.safeBlockHash` does not belong to the chain defined by `forkchoiceState.headBlockHash`. ```sequence Title: Block Building Fails, Invalid ForkChoiceState Note over CL: Validator is assigned \n to propose Note over CL: Fill BeaconBlock up to \n execution_payload CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes) Note over EL: ForkChoiceState is not \n valid EL --> CL: -38002: Invalid forkchoice state ``` #### EL is syncing Same as in [node startup](#EL-is-syncing). ```sequence Title: Block Building Fails, EL is Syncing Note over CL: Validator is assigned \n to propose Note over CL: Fill BeaconBlock up to \n execution_payload CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes) Note over EL: I don't have all the \n necessary data available EL --> CL: {payloadStatus: {status: SYNCING, ...}, payloadId: null} ``` ### `getPayloadV2` fails After a successful FCU call, the EL starts building a payload and returns a `PayloadId` to the CL. The CL then can use this `PayloadId` to ask for the payload. Of course, if the CL call attempts to retrieve the payload with the wrong ID, the EL returns an error. ```sequence Title: Block Building Fails, Unknown Payload Note over CL: Validator is assigned \n to propose Note over CL: Fill BeaconBlock up to \n execution_payload CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes) EL --> CL: {payloadStatus: {status: VALID, ...}, payloadId: buildProcessId} Note over EL: Build \n execution_payload CL -> EL: engine_getPayloadV2(PayloadId) Note over EL: There is no build \n process with that ID EL --> CL: -38001: Unknown payload ``` ## Block validation *In this section, the call to `engine_forkChoiceUpdatedV2` is not as explicit as in other flows. Keep in mind that the nodes are expected to run FCU at the start of every slot. So, any flows here assume that the EL is running on the latest fork choice.* If everything goes well, a node determines if a block is *valid, invalid, or accepted*. I made a [flow chart](https://github.com/danielrachi/diagrams/blob/main/rendered/engine-api/newPayloadV2.png) explaining how the EL handles `engine_newPayloadV2`. You can find a better level of detail there. ### VALID As the [`engine_newPayloadV1`](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#specification) spec says (this part is unchanged for `newPayloadV2`): > Client software MUST respond to this method call in the following way: > ... > - with the payload status obtained from the Payload validation process if the payload has been fully validated while processing the call So, for `newPayloadV2` to return *VALID*, the payload being validated must pass all checks before reaching the [validation process](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#payload-validation), and then it has to be considered valid during the validation process. ```sequence Title: Block Validation, VALID Note over CL: Receive new beacon \n block Note over CL: Extract \n ExecutionPayload \n from block CL -> EL: engine_newPayloadV2(ExecutionPayload) Note over EL: All requirements are \n met and the payload is \n considered valid EL --> CL: {status: VALID, ...} ``` ### INVALID For `engine_newPayloadV2` to return invalid, two things can happen: - The `blockHash` is invalid as defined by point (1) of [`newPayloadV1`](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#specification) spec. (In the spec of [`newPayloadV2`](https://github.com/ethereum/execution-apis/blob/main/src/engine/shanghai.md#response) it's stated that `INVALID_BLOCK_HASH` status value is supplanted by `INVALID`.) - The [validation process](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#payload-validation) concludes that the payload is invalid. ```sequence Title: Block Validation, INVALID Note over CL: Receive new beacon \n block Note over CL: Extract \n ExecutionPayload \n from block CL -> EL: engine_newPayloadV2(ExecutionPayload) Note over EL: BlockHash is invalid \n OR \n validation process concludes \n payload is invalid EL --> CL: {status: INVALID, ...} ``` ### SYNCING Same as in [node startup](#EL-is-syncing) and [block building](#EL-is-syncing1). ```sequence Title: Block Validation Failed, Syncing EL Note over CL: Receive new beacon \n block Note over CL: Extract \n ExecutionPayload \n from block CL -> EL: engine_newPayloadV2(ExecutionPayload) Note over EL: I don't have all the \n necessary data available EL --> CL: {status: SYNCING, ...} ``` ### Shallow State Clients This type of client holds only one version of the world state: the post-state of the head of the canonical chain. Erigon is an example of this type of client. Let's say one of these clients is running chain A and gets chain B on the FCU call. They would only know the state of chain A, and would have to validate all blocks on chain B before it becomes fully validated. In this case, the response will be VALID or INVALID. ```sequence Title: Block Validation, Shallow State Client Participant CL Participant EL (shallow) Note over EL (shallow): Last FCU nominated \n a non-canonical chain \n (chain B) Note over CL, EL (shallow): Validate all blocks on Chain B Note over CL: Receive new beacon \n block Note over CL: Extract \n ExecutionPayload \n from block CL -> EL (shallow): engine_newPayloadV2(ExecutionPayload) EL (shallow) --> CL: {status: VALID, ...} OR {status: INVALID, ...} ``` #### ACCEPTED This is the case when a shallow-state client receives a non-canonical block. In the previous example, it would mean that it gets a block for chain B when it's running chain A (Before the FCU call). ```sequence Title: Block Validation, ACCEPTED Participant CL Participant EL (shallow) Note over CL: Receive new beacon \n block Note over CL: Extract \n ExecutionPayload \n from block CL -> EL (shallow): engine_newPayloadV2(ExecutionPayload) Note over EL (shallow): Payload is \n non-canonical EL (shallow) --> CL: {status: ACCEPTED, ...} ``` ### Validation failed There is a scenario where this process can fail. #### Wrong `ExecutionPayload` version Very similar to the error caused by using the wrong `PayloadAttributes` version on [block building](#Wrong-PayloadAttributes-version). The version of the `ExecutionPayload` that we are trying to validate must be consistent with the Shanghai timestamp. > * ExecutionPayloadV1 MUST be used if the timestamp value is lower than the Shanghai timestamp, > * ExecutionPayloadV2 MUST be used if the timestamp value is greater or equal to the Shanghai timestamp, > * Client software MUST return -32602: Invalid params error if the wrong version of the structure is used in the method call. ```sequence Title: Block Validation Failed: Wrong ExecutionPayload Version Note over CL: Receive new beacon \n block Note over CL: Extract \n ExecutionPayload \n from block CL -> EL: engine_newPayloadV2(ExecutionPayload) Note over EL: Wrong \n ExecutionPayload \n version EL --> CL: -32602: Invalid params ``` ## All together Now, here's a diagram showing what the full lifetime of a validator node would look like. Something to note here is the timing of the calls relative to where slots start or end. Something that might seem odd is the call to `engine_forkchoiceUpdatedV2` with `PayloadParams` even when the validator is not selected to propose. This is an optimization validators can make to have more time to build a block (and therefore, build a more profitable block). ```sequence Title: Validator lifetime Note over CL, EL: Node is started CL -> EL: engine_exchangeCapabilities(engine methods supported by CL) EL --> CL: engine methods supported by EL CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, null) Note over EL: I don't have all the \n necessary data available, \n syncing EL --> CL: {payloadStatus: {status: SYNCING, ...}, payloadId: null} Note over CL, EL: ... Note over EL: Sync is completed CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, null) EL --> CL: {payloadStatus: {status: VALID, ...}, payloadId: null} Note over CL, EL: Slot where validator doesn't have to propose starts CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, null) EL --> CL: {payloadStatus: {status: VALID, ...}, payloadId: null} Note over CL: Receive new \n beacon block Note over CL: Extract \n ExecutionPayload \n from block CL -> EL: engine_newPayloadV2(ExecutionPayload) Note over EL: All requirements \n are met and the \n payload is considered \n valid EL --> CL: {status: VALID, ...} CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes) EL --> CL: {payloadStatus: {status: VALID, ...}, payloadId: buildProcessId} Note over EL: Start building \n execution_payload Note over CL, EL: Slot where validator doesn't have to propose ends Note over CL, EL: Slot where validator has to propose starts CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, null) EL --> CL: {payloadStatus: {status: VALID, ...}, payloadId: null} Note over CL: Fill BeaconBlock \n up to \n execution_payload CL -> EL: engine_getPayloadV2(PayloadId) EL --> CL: {executionPayload, blockValue} Note over CL: Put the \n ExecutionPayload \n on the BeaconBlock Note over CL: Compute state_root Note over CL: Propagate block Note over CL, EL: Slot where validator has to propose ends Note over CL, EL: More slots of both types would happen again until the node is shutdown ```
Last changed by  
 
Daniel Ramirez
Ex: Ethereum Protocol Fellowship
3056

Read more

Notes on Flashbots

MEV-Boost, allows out of protocol PBS but with some trust assumptions. Maybe doing this on portal would solve those trust assumptions?

12/6/2023
Lighthouse Beacon Node Startup

How Lighthouse initiates a Beacon Node. (Focusing on the execution engine.)

12/6/2023
EPF Final Dev Update

In this project, I wanted to explore how can an execution layer client could be replaced with a portal client on a staking setup.

11/15/2023
Notes on Kademlia Distributed hash table

A map is a data structure that stores <key, value> pairs and allows two fundamental operations: PUT(key, value) to add or update a value under a specific key and GET(key) to retrieve the value associated with a particular key. This makes maps highly useful for quickly retrieving specific values.

11/7/2023
Read more from Daniel Ramirez
Published on HackMD