# Engine API: A Visual Guide
The Engine API is the interface that allows communication between the execution and the consensus layer of an Ethereum node. Since the merge, this API has become a vital piece of Ethereum's architecture. It coordinates important tasks such as block validation and block proposal.
Even with its importance, it's hard to find resources beyond the [Engine API spec](https://github.com/ethereum/execution-apis/blob/main/src/engine) when learning about it.
Trying to understand the API just by reading the spec is a challenging task. For that reason, in this article, I'll explain the most important flows of the Engine API. I'll indicate how those flows can fail, where the spec dictates them, and how they fit together.
*This article is part of my [EPF project](https://github.com/eth-protocol-fellows/cohort-four/blob/master/projects/portal-network-validator.md).*
## Considerations
- This is based on the [Shanghai spec](https://github.com/ethereum/execution-apis/blob/main/src/engine/shanghai.md).
- The validator software is shown as part of the consensus layer client.
- Every flow here assumes the validator is building blocks locally.
- If you want the rendered diagrams alone, you can find them [here](https://github.com/danielrachi/diagrams/tree/main/rendered/engine-api/sequence-diagrams).
## Functions tl;dr.
Here's a brief explanation of the functions we will encounter along the way:
- `engine_exchangeCapabilities`: Used to exchange the Engine API methods supported by each client.
- `engine_forkchoiceUpdatedV2`: Updates the fork choice of the Execution Layer Client. Also used to start a payload build process.
- `engine_getPayloadV2`: Used to retrieve an `execution_payload` from a build process started in a past `engine_forkchoiceUpdatedV2` call.
- `engine_newPayloadV2`: Used by the consensus layer client to send an `execution_payload` to the execution layer client for it to validate it.
## Node startup
When you run the command to start a beacon node, [two Engine API endpoints are called](https://hackmd.io/@danielrachi/S1u2Veflp): `engine_exchangeCapabilities` and `engine_forkchoiceUpdatedV2`.
Ideally, the following would be the communication between the *Execution Layer Client* (EL) and the *Consensus Layer Client* (CL):
```sequence
Title: Node Startup, Success
CL -> EL: engine_exchangeCapabilities(engine methods supported by CL)
EL --> CL: engine methods supported by EL
CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, null)
EL --> CL: {payloadStatus: {status: VALID, ...}, payloadId: null}
```
*Note how the second argument of `engine_forkchoiceUpdatedV2` is null. This tells the EL not to start a payload build process.*
### EL is syncing
There is no guarantee that both clients will finish their sync at the same time. For that reason, the CL could call an endpoint when the EL is still syncing, and that leads to the following scenario:
```sequence
Title: Node Startup, EL is Syncing
CL -> EL: engine_exchangeCapabilities(engine methods supported by CL)
EL --> CL: engine methods supported by EL
CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, null)
Note over EL: I don't have all the \n necessary data available
EL --> CL: {payloadStatus: {status: SYNCING, ...}, payloadId: null}
```
## Block Building
When a validator is selected to propose a block, the ideal communication between both clients is as follows:
```sequence
Title: Block Building, Success
Note over CL: Validator is assigned \n to propose
Note over CL: Fill BeaconBlock up to \n execution_payload
CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes)
EL --> CL: {payloadStatus: {status: VALID, ...}, payloadId: buildProcessId}
Note over EL: Build execution_payload
CL -> EL: engine_getPayloadV2(PayloadId)
EL --> CL: {executionPayload, blockValue}
Note over CL: Put the ExecutionPayload \n on the BeaconBlock
Note over CL: Compute state_root
Note over CL: Propagate block
```
*Note how this time, a `PayloadAttribues` is passed as the second argument when calling `engine_forkChoiceUpdatedV2`. Then a `payloadId` is part of the response and the payload build process is started.*
Let's see how this process can fail.
### `engine_forkchoiceUpdatedV2` fails
#### Payload referenced by `forkchoiceState.headBlockHash` is invalid
The `forkchoiceState` struct has a field called `headBlockHash`. Before updating its fork choice, the EL must check that the payload referenced by the `headBlockHash` is valid. The process to define the validity of the payload is defined in the engine API Paris spec, [here](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#payload-validation).
If this process concludes that the referenced payload is invalid, this happens:
```sequence
Title: Block building fails, INVALID payload
Note over CL: Validator is \n assigned to propose
Note over CL: Fill BeaconBlock \n up to execution_payload
CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes)
Note over EL: Start payload \n validation process
Note over EL: Payload is invalid
EL --> CL: {payloadStatus: {status: INVALID, ...}, payloadId: null}
```
#### Invalid `PayloadAttributes`
Similar to how the `ForkChoiceState` needs to meet certain criteria to be considered valid, `PayloadAttributes` must fill some requirements to be valid.
From point 7 of the [`engine_forkchoiceUpdatedV1` spec](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#specification-1):
> Client software MUST ensure that `payloadAttributes.timestamp `is greater than timestamp of a block referenced by `forkchoiceState.headBlockHash`. If this condition isn't held client software MUST respond with `-38003: Invalid payload attributes` and MUST NOT begin a payload build process. In such an event, the `forkchoiceState` update MUST NOT be rolled back.
```sequence
Title: Block Building Fails, Invalid PayloadAttributes
Note over CL: Validator is assigned \n to propose
Note over CL: Fill BeaconBlock up to \n execution_payload
CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes)
Note over EL: PayloadAttributes is not \n valid
EL --> CL: -38003: Invalid payload attributes
```
#### Wrong `PayloadAttributes` version
Besides being valid, `PayloadAttributes` needs to be of the right version.
From the [`engine_forkchoiceUpdatedV2`](https://github.com/ethereum/execution-apis/blob/main/src/engine/shanghai.md#engine_forkchoiceupdatedv2) spec:
> - `PayloadAttributesV1` MUST be used to build a payload with the timestamp value lower than the Shanghai timestamp,
> - `PayloadAttributesV2` MUST be used to build a payload with the timestamp value greater or equal to the Shanghai timestamp,
> - Client software MUST return `-32602: Invalid params` error if the wrong version of the structure is used in the method call.
If this check fails, this occurs:
```sequence
Title: Block Building Fails, Wrong PayloadAttributes Version
Note over CL: Validator is assigned \n to propose
Note over CL: Fill BeaconBlock up to \n execution_payload
CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes)
Note over EL: Wrong PayloadAttributes \n version
EL --> CL: -32602: Invalid params
```
#### `ForkChoiceState` is invalid
The `ForkChoiceState` passed as an argument on the FCU call needs to meet certain criteria to be considered valid. The criteria are stated in point 6 of the spec of [`engine_forkchoiceupdatedV1`](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#specification-1):
> Client software MUST return `-38002: Invalid forkchoice` state error if the payload referenced by `forkchoiceState.headBlockHash` is `VALID` and a payload referenced by either `forkchoiceState.finalizedBlockHash` or `forkchoiceState.safeBlockHash` does not belong to the chain defined by `forkchoiceState.headBlockHash`.
```sequence
Title: Block Building Fails, Invalid ForkChoiceState
Note over CL: Validator is assigned \n to propose
Note over CL: Fill BeaconBlock up to \n execution_payload
CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes)
Note over EL: ForkChoiceState is not \n valid
EL --> CL: -38002: Invalid forkchoice state
```
#### EL is syncing
Same as in [node startup](#EL-is-syncing).
```sequence
Title: Block Building Fails, EL is Syncing
Note over CL: Validator is assigned \n to propose
Note over CL: Fill BeaconBlock up to \n execution_payload
CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes)
Note over EL: I don't have all the \n necessary data available
EL --> CL: {payloadStatus: {status: SYNCING, ...}, payloadId: null}
```
### `getPayloadV2` fails
After a successful FCU call, the EL starts building a payload and returns a `PayloadId` to the CL. The CL then can use this `PayloadId` to ask for the payload. Of course, if the CL call attempts to retrieve the payload with the wrong ID, the EL returns an error.
```sequence
Title: Block Building Fails, Unknown Payload
Note over CL: Validator is assigned \n to propose
Note over CL: Fill BeaconBlock up to \n execution_payload
CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes)
EL --> CL: {payloadStatus: {status: VALID, ...}, payloadId: buildProcessId}
Note over EL: Build \n execution_payload
CL -> EL: engine_getPayloadV2(PayloadId)
Note over EL: There is no build \n process with that ID
EL --> CL: -38001: Unknown payload
```
## Block validation
*In this section, the call to `engine_forkChoiceUpdatedV2` is not as explicit as in other flows. Keep in mind that the nodes are expected to run FCU at the start of every slot. So, any flows here assume that the EL is running on the latest fork choice.*
If everything goes well, a node determines if a block is *valid, invalid, or accepted*.
I made a [flow chart](https://github.com/danielrachi/diagrams/blob/main/rendered/engine-api/newPayloadV2.png) explaining how the EL handles `engine_newPayloadV2`. You can find a better level of detail there.
### VALID
As the [`engine_newPayloadV1`](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#specification) spec says (this part is unchanged for `newPayloadV2`):
> Client software MUST respond to this method call in the following way:
> ...
> - with the payload status obtained from the Payload validation process if the payload has been fully validated while processing the call
So, for `newPayloadV2` to return *VALID*, the payload being validated must pass all checks before reaching the [validation process](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#payload-validation), and then it has to be considered valid during the validation process.
```sequence
Title: Block Validation, VALID
Note over CL: Receive new beacon \n block
Note over CL: Extract \n ExecutionPayload \n from block
CL -> EL: engine_newPayloadV2(ExecutionPayload)
Note over EL: All requirements are \n met and the payload is \n considered valid
EL --> CL: {status: VALID, ...}
```
### INVALID
For `engine_newPayloadV2` to return invalid, two things can happen:
- The `blockHash` is invalid as defined by point (1) of [`newPayloadV1`](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#specification) spec. (In the spec of [`newPayloadV2`](https://github.com/ethereum/execution-apis/blob/main/src/engine/shanghai.md#response) it's stated that `INVALID_BLOCK_HASH` status value is supplanted by `INVALID`.)
- The [validation process](https://github.com/ethereum/execution-apis/blob/main/src/engine/paris.md#payload-validation) concludes that the payload is invalid.
```sequence
Title: Block Validation, INVALID
Note over CL: Receive new beacon \n block
Note over CL: Extract \n ExecutionPayload \n from block
CL -> EL: engine_newPayloadV2(ExecutionPayload)
Note over EL: BlockHash is invalid \n OR \n validation process concludes \n payload is invalid
EL --> CL: {status: INVALID, ...}
```
### SYNCING
Same as in [node startup](#EL-is-syncing) and [block building](#EL-is-syncing1).
```sequence
Title: Block Validation Failed, Syncing EL
Note over CL: Receive new beacon \n block
Note over CL: Extract \n ExecutionPayload \n from block
CL -> EL: engine_newPayloadV2(ExecutionPayload)
Note over EL: I don't have all the \n necessary data available
EL --> CL: {status: SYNCING, ...}
```
### Shallow State Clients
This type of client holds only one version of the world state: the post-state of the head of the canonical chain. Erigon is an example of this type of client.
Let's say one of these clients is running chain A and gets chain B on the FCU call. They would only know the state of chain A, and would have to validate all blocks on chain B before it becomes fully validated.
In this case, the response will be VALID or INVALID.
```sequence
Title: Block Validation, Shallow State Client
Participant CL
Participant EL (shallow)
Note over EL (shallow): Last FCU nominated \n a non-canonical chain \n (chain B)
Note over CL, EL (shallow): Validate all blocks on Chain B
Note over CL: Receive new beacon \n block
Note over CL: Extract \n ExecutionPayload \n from block
CL -> EL (shallow): engine_newPayloadV2(ExecutionPayload)
EL (shallow) --> CL: {status: VALID, ...} OR {status: INVALID, ...}
```
#### ACCEPTED
This is the case when a shallow-state client receives a non-canonical block.
In the previous example, it would mean that it gets a block for chain B when it's running chain A (Before the FCU call).
```sequence
Title: Block Validation, ACCEPTED
Participant CL
Participant EL (shallow)
Note over CL: Receive new beacon \n block
Note over CL: Extract \n ExecutionPayload \n from block
CL -> EL (shallow): engine_newPayloadV2(ExecutionPayload)
Note over EL (shallow): Payload is \n non-canonical
EL (shallow) --> CL: {status: ACCEPTED, ...}
```
### Validation failed
There is a scenario where this process can fail.
#### Wrong `ExecutionPayload` version
Very similar to the error caused by using the wrong `PayloadAttributes` version on [block building](#Wrong-PayloadAttributes-version). The version of the `ExecutionPayload` that we are trying to validate must be consistent with the Shanghai timestamp.
> * ExecutionPayloadV1 MUST be used if the timestamp value is lower than the Shanghai timestamp,
> * ExecutionPayloadV2 MUST be used if the timestamp value is greater or equal to the Shanghai timestamp,
> * Client software MUST return -32602: Invalid params error if the wrong version of the structure is used in the method call.
```sequence
Title: Block Validation Failed: Wrong ExecutionPayload Version
Note over CL: Receive new beacon \n block
Note over CL: Extract \n ExecutionPayload \n from block
CL -> EL: engine_newPayloadV2(ExecutionPayload)
Note over EL: Wrong \n ExecutionPayload \n version
EL --> CL: -32602: Invalid params
```
## All together
Now, here's a diagram showing what the full lifetime of a validator node would look like. Something to note here is the timing of the calls relative to where slots start or end.
Something that might seem odd is the call to `engine_forkchoiceUpdatedV2` with `PayloadParams` even when the validator is not selected to propose. This is an optimization validators can make to have more time to build a block (and therefore, build a more profitable block).
```sequence
Title: Validator lifetime
Note over CL, EL: Node is started
CL -> EL: engine_exchangeCapabilities(engine methods supported by CL)
EL --> CL: engine methods supported by EL
CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, null)
Note over EL: I don't have all the \n necessary data available, \n syncing
EL --> CL: {payloadStatus: {status: SYNCING, ...}, payloadId: null}
Note over CL, EL: ...
Note over EL: Sync is completed
CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, null)
EL --> CL: {payloadStatus: {status: VALID, ...}, payloadId: null}
Note over CL, EL: Slot where validator doesn't have to propose starts
CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, null)
EL --> CL: {payloadStatus: {status: VALID, ...}, payloadId: null}
Note over CL: Receive new \n beacon block
Note over CL: Extract \n ExecutionPayload \n from block
CL -> EL: engine_newPayloadV2(ExecutionPayload)
Note over EL: All requirements \n are met and the \n payload is considered \n valid
EL --> CL: {status: VALID, ...}
CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes)
EL --> CL: {payloadStatus: {status: VALID, ...}, payloadId: buildProcessId}
Note over EL: Start building \n execution_payload
Note over CL, EL: Slot where validator doesn't have to propose ends
Note over CL, EL: Slot where validator has to propose starts
CL -> EL: engine_forkchoiceUpdatedV2(ForkchoiceState, null)
EL --> CL: {payloadStatus: {status: VALID, ...}, payloadId: null}
Note over CL: Fill BeaconBlock \n up to \n execution_payload
CL -> EL: engine_getPayloadV2(PayloadId)
EL --> CL: {executionPayload, blockValue}
Note over CL: Put the \n ExecutionPayload \n on the BeaconBlock
Note over CL: Compute state_root
Note over CL: Propagate block
Note over CL, EL: Slot where validator has to propose ends
Note over CL, EL: More slots of both types would happen again until the node is shutdown
```
Sign in with Wallet
Connect another wallet