The Engine API is the interface that allows communication between the execution and the consensus layer of an Ethereum node. Since the merge, this API has become a vital piece of Ethereum's architecture. It coordinates important tasks such as block validation and block proposal.
Even with its importance, it's hard to find resources beyond the Engine API spec when learning about it.
Trying to understand the API just by reading the spec is a challenging task. For that reason, in this article, I'll explain the most important flows of the Engine API. I'll indicate how those flows can fail, where the spec dictates them, and how they fit together.
This article is part of my EPF project.
Here's a brief explanation of the functions we will encounter along the way:
engine_exchangeCapabilities
: Used to exchange the Engine API methods supported by each client.engine_forkchoiceUpdatedV2
: Updates the fork choice of the Execution Layer Client. Also used to start a payload build process.engine_getPayloadV2
: Used to retrieve an execution_payload
from a build process started in a past engine_forkchoiceUpdatedV2
call.engine_newPayloadV2
: Used by the consensus layer client to send an execution_payload
to the execution layer client for it to validate it.When you run the command to start a beacon node, two Engine API endpoints are called: engine_exchangeCapabilities
and engine_forkchoiceUpdatedV2
.
Ideally, the following would be the communication between the Execution Layer Client (EL) and the Consensus Layer Client (CL):
Note how the second argument of engine_forkchoiceUpdatedV2
is null. This tells the EL not to start a payload build process.
There is no guarantee that both clients will finish their sync at the same time. For that reason, the CL could call an endpoint when the EL is still syncing, and that leads to the following scenario:
When a validator is selected to propose a block, the ideal communication between both clients is as follows:
Note how this time, a PayloadAttribues
is passed as the second argument when calling engine_forkChoiceUpdatedV2
. Then a payloadId
is part of the response and the payload build process is started.
Let's see how this process can fail.
engine_forkchoiceUpdatedV2
failsforkchoiceState.headBlockHash
is invalidThe forkchoiceState
struct has a field called headBlockHash
. Before updating its fork choice, the EL must check that the payload referenced by the headBlockHash
is valid. The process to define the validity of the payload is defined in the engine API Paris spec, here.
If this process concludes that the referenced payload is invalid, this happens:
PayloadAttributes
Similar to how the ForkChoiceState
needs to meet certain criteria to be considered valid, PayloadAttributes
must fill some requirements to be valid.
From point 7 of the engine_forkchoiceUpdatedV1
spec:
Client software MUST ensure that
payloadAttributes.timestamp
is greater than timestamp of a block referenced byforkchoiceState.headBlockHash
. If this condition isn't held client software MUST respond with-38003: Invalid payload attributes
and MUST NOT begin a payload build process. In such an event, theforkchoiceState
update MUST NOT be rolled back.
PayloadAttributes
versionBesides being valid, PayloadAttributes
needs to be of the right version.
From the engine_forkchoiceUpdatedV2
spec:
PayloadAttributesV1
MUST be used to build a payload with the timestamp value lower than the Shanghai timestamp,PayloadAttributesV2
MUST be used to build a payload with the timestamp value greater or equal to the Shanghai timestamp,- Client software MUST return
-32602: Invalid params
error if the wrong version of the structure is used in the method call.
If this check fails, this occurs:
ForkChoiceState
is invalidThe ForkChoiceState
passed as an argument on the FCU call needs to meet certain criteria to be considered valid. The criteria are stated in point 6 of the spec of engine_forkchoiceupdatedV1
:
Client software MUST return
-38002: Invalid forkchoice
state error if the payload referenced byforkchoiceState.headBlockHash
isVALID
and a payload referenced by eitherforkchoiceState.finalizedBlockHash
orforkchoiceState.safeBlockHash
does not belong to the chain defined byforkchoiceState.headBlockHash
.
Same as in node startup.
getPayloadV2
failsAfter a successful FCU call, the EL starts building a payload and returns a PayloadId
to the CL. The CL then can use this PayloadId
to ask for the payload. Of course, if the CL call attempts to retrieve the payload with the wrong ID, the EL returns an error.
In this section, the call to engine_forkChoiceUpdatedV2
is not as explicit as in other flows. Keep in mind that the nodes are expected to run FCU at the start of every slot. So, any flows here assume that the EL is running on the latest fork choice.
If everything goes well, a node determines if a block is valid, invalid, or accepted.
I made a flow chart explaining how the EL handles engine_newPayloadV2
. You can find a better level of detail there.
As the engine_newPayloadV1
spec says (this part is unchanged for newPayloadV2
):
Client software MUST respond to this method call in the following way:
…
- with the payload status obtained from the Payload validation process if the payload has been fully validated while processing the call
So, for newPayloadV2
to return VALID, the payload being validated must pass all checks before reaching the validation process, and then it has to be considered valid during the validation process.
For engine_newPayloadV2
to return invalid, two things can happen:
blockHash
is invalid as defined by point (1) of newPayloadV1
spec. (In the spec of newPayloadV2
it's stated that INVALID_BLOCK_HASH
status value is supplanted by INVALID
.)Same as in node startup and block building.
This type of client holds only one version of the world state: the post-state of the head of the canonical chain. Erigon is an example of this type of client.
Let's say one of these clients is running chain A and gets chain B on the FCU call. They would only know the state of chain A, and would have to validate all blocks on chain B before it becomes fully validated.
In this case, the response will be VALID or INVALID.
This is the case when a shallow-state client receives a non-canonical block.
In the previous example, it would mean that it gets a block for chain B when it's running chain A (Before the FCU call).
There is a scenario where this process can fail.
ExecutionPayload
versionVery similar to the error caused by using the wrong PayloadAttributes
version on block building. The version of the ExecutionPayload
that we are trying to validate must be consistent with the Shanghai timestamp.
- ExecutionPayloadV1 MUST be used if the timestamp value is lower than the Shanghai timestamp,
- ExecutionPayloadV2 MUST be used if the timestamp value is greater or equal to the Shanghai timestamp,
- Client software MUST return -32602: Invalid params error if the wrong version of the structure is used in the method call.
Now, here's a diagram showing what the full lifetime of a validator node would look like. Something to note here is the timing of the calls relative to where slots start or end.
Something that might seem odd is the call to engine_forkchoiceUpdatedV2
with PayloadParams
even when the validator is not selected to propose. This is an optimization validators can make to have more time to build a block (and therefore, build a more profitable block).