Try   HackMD

Engine API: A Visual Guide

The Engine API is the interface that allows communication between the execution and the consensus layer of an Ethereum node. Since the merge, this API has become a vital piece of Ethereum's architecture. It coordinates important tasks such as block validation and block proposal.

Even with its importance, it's hard to find resources beyond the Engine API spec when learning about it.

Trying to understand the API just by reading the spec is a challenging task. For that reason, in this article, I'll explain the most important flows of the Engine API. I'll indicate how those flows can fail, where the spec dictates them, and how they fit together.

This article is part of my EPF project.

Considerations

  • This is based on the Shanghai spec.
  • The validator software is shown as part of the consensus layer client.
  • Every flow here assumes the validator is building blocks locally.
  • If you want the rendered diagrams alone, you can find them here.

Functions tl;dr.

Here's a brief explanation of the functions we will encounter along the way:

  • engine_exchangeCapabilities: Used to exchange the Engine API methods supported by each client.
  • engine_forkchoiceUpdatedV2: Updates the fork choice of the Execution Layer Client. Also used to start a payload build process.
  • engine_getPayloadV2: Used to retrieve an execution_payload from a build process started in a past engine_forkchoiceUpdatedV2 call.
  • engine_newPayloadV2: Used by the consensus layer client to send an execution_payload to the execution layer client for it to validate it.

Node startup

When you run the command to start a beacon node, two Engine API endpoints are called: engine_exchangeCapabilities and engine_forkchoiceUpdatedV2.

Ideally, the following would be the communication between the Execution Layer Client (EL) and the Consensus Layer Client (CL):

Created with Raphaël 2.2.0Node Startup, SuccessCLCLELELengine_exchangeCapabilities(engine methods supported by CL)engine methods supported by ELengine_forkchoiceUpdatedV2(ForkchoiceState, null){payloadStatus: {status: VALID, ...}, payloadId: null}

Note how the second argument of engine_forkchoiceUpdatedV2 is null. This tells the EL not to start a payload build process.

EL is syncing

There is no guarantee that both clients will finish their sync at the same time. For that reason, the CL could call an endpoint when the EL is still syncing, and that leads to the following scenario:

Created with Raphaël 2.2.0Node Startup, EL is SyncingCLCLELELengine_exchangeCapabilities(engine methods supported by CL)engine methods supported by ELengine_forkchoiceUpdatedV2(ForkchoiceState, null)I don't have all the  necessary data available{payloadStatus: {status: SYNCING, ...}, payloadId: null}

Block Building

When a validator is selected to propose a block, the ideal communication between both clients is as follows:

Created with Raphaël 2.2.0Block Building, SuccessCLCLELELValidator is assigned  to proposeFill BeaconBlock up to  execution_payloadengine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes){payloadStatus: {status: VALID, ...}, payloadId: buildProcessId}Build execution_payloadengine_getPayloadV2(PayloadId){executionPayload, blockValue}Put the ExecutionPayload  on the BeaconBlockCompute state_rootPropagate block

Note how this time, a PayloadAttribues is passed as the second argument when calling engine_forkChoiceUpdatedV2. Then a payloadId is part of the response and the payload build process is started.

Let's see how this process can fail.

engine_forkchoiceUpdatedV2 fails

Payload referenced by forkchoiceState.headBlockHash is invalid

The forkchoiceState struct has a field called headBlockHash. Before updating its fork choice, the EL must check that the payload referenced by the headBlockHash is valid. The process to define the validity of the payload is defined in the engine API Paris spec, here.

If this process concludes that the referenced payload is invalid, this happens:

Created with Raphaël 2.2.0Block building fails, INVALID payloadCLCLELELValidator is  assigned to proposeFill BeaconBlock  up to execution_payloadengine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes)Start payload  validation processPayload is invalid{payloadStatus: {status: INVALID, ...}, payloadId: null}

Invalid PayloadAttributes

Similar to how the ForkChoiceState needs to meet certain criteria to be considered valid, PayloadAttributes must fill some requirements to be valid.

From point 7 of the engine_forkchoiceUpdatedV1 spec:

Client software MUST ensure that payloadAttributes.timestamp is greater than timestamp of a block referenced by forkchoiceState.headBlockHash. If this condition isn't held client software MUST respond with -38003: Invalid payload attributes and MUST NOT begin a payload build process. In such an event, the forkchoiceState update MUST NOT be rolled back.

Created with Raphaël 2.2.0Block Building Fails, Invalid PayloadAttributesCLCLELELValidator is assigned  to proposeFill BeaconBlock up to  execution_payloadengine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes)PayloadAttributes is not  valid-38003: Invalid payload attributes

Wrong PayloadAttributes version

Besides being valid, PayloadAttributes needs to be of the right version.

From the engine_forkchoiceUpdatedV2 spec:

  • PayloadAttributesV1 MUST be used to build a payload with the timestamp value lower than the Shanghai timestamp,
  • PayloadAttributesV2 MUST be used to build a payload with the timestamp value greater or equal to the Shanghai timestamp,
  • Client software MUST return -32602: Invalid params error if the wrong version of the structure is used in the method call.

If this check fails, this occurs:

Created with Raphaël 2.2.0Block Building Fails, Wrong PayloadAttributes VersionCLCLELELValidator is assigned  to proposeFill BeaconBlock up to  execution_payloadengine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes)Wrong PayloadAttributes  version-32602: Invalid params

ForkChoiceState is invalid

The ForkChoiceState passed as an argument on the FCU call needs to meet certain criteria to be considered valid. The criteria are stated in point 6 of the spec of engine_forkchoiceupdatedV1:

Client software MUST return -38002: Invalid forkchoice state error if the payload referenced by forkchoiceState.headBlockHash is VALID and a payload referenced by either forkchoiceState.finalizedBlockHash or forkchoiceState.safeBlockHash does not belong to the chain defined by forkchoiceState.headBlockHash.

Created with Raphaël 2.2.0Block Building Fails, Invalid ForkChoiceStateCLCLELELValidator is assigned  to proposeFill BeaconBlock up to  execution_payloadengine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes)ForkChoiceState is not  valid-38002: Invalid forkchoice state

EL is syncing

Same as in node startup.

Created with Raphaël 2.2.0Block Building Fails, EL is SyncingCLCLELELValidator is assigned  to proposeFill BeaconBlock up to  execution_payloadengine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes)I don't have all the  necessary data available{payloadStatus: {status: SYNCING, ...}, payloadId: null}

getPayloadV2 fails

After a successful FCU call, the EL starts building a payload and returns a PayloadId to the CL. The CL then can use this PayloadId to ask for the payload. Of course, if the CL call attempts to retrieve the payload with the wrong ID, the EL returns an error.

Created with Raphaël 2.2.0Block Building Fails, Unknown PayloadCLCLELELValidator is assigned  to proposeFill BeaconBlock up to  execution_payloadengine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes){payloadStatus: {status: VALID, ...}, payloadId: buildProcessId}Build  execution_payloadengine_getPayloadV2(PayloadId)There is no build  process with that ID-38001: Unknown payload

Block validation

In this section, the call to engine_forkChoiceUpdatedV2 is not as explicit as in other flows. Keep in mind that the nodes are expected to run FCU at the start of every slot. So, any flows here assume that the EL is running on the latest fork choice.

If everything goes well, a node determines if a block is valid, invalid, or accepted.

I made a flow chart explaining how the EL handles engine_newPayloadV2. You can find a better level of detail there.

VALID

As the engine_newPayloadV1 spec says (this part is unchanged for newPayloadV2):

Client software MUST respond to this method call in the following way:

  • with the payload status obtained from the Payload validation process if the payload has been fully validated while processing the call

So, for newPayloadV2 to return VALID, the payload being validated must pass all checks before reaching the validation process, and then it has to be considered valid during the validation process.

Created with Raphaël 2.2.0Block Validation, VALIDCLCLELELReceive new beacon  blockExtract  ExecutionPayload  from blockengine_newPayloadV2(ExecutionPayload)All requirements are  met and the payload is  considered valid{status: VALID, ...}

INVALID

For engine_newPayloadV2 to return invalid, two things can happen:

  • The blockHash is invalid as defined by point (1) of newPayloadV1 spec. (In the spec of newPayloadV2 it's stated that INVALID_BLOCK_HASH status value is supplanted by INVALID.)
  • The validation process concludes that the payload is invalid.
Created with Raphaël 2.2.0Block Validation, INVALIDCLCLELELReceive new beacon  blockExtract  ExecutionPayload  from blockengine_newPayloadV2(ExecutionPayload)BlockHash is invalid  OR  validation process concludes  payload is invalid{status: INVALID, ...}

SYNCING

Same as in node startup and block building.

Created with Raphaël 2.2.0Block Validation Failed, Syncing ELCLCLELELReceive new beacon  blockExtract  ExecutionPayload  from blockengine_newPayloadV2(ExecutionPayload)I don't have all the  necessary data available{status: SYNCING, ...}

Shallow State Clients

This type of client holds only one version of the world state: the post-state of the head of the canonical chain. Erigon is an example of this type of client.

Let's say one of these clients is running chain A and gets chain B on the FCU call. They would only know the state of chain A, and would have to validate all blocks on chain B before it becomes fully validated.

In this case, the response will be VALID or INVALID.

Created with Raphaël 2.2.0Block Validation, Shallow State ClientCLCLEL (shallow)EL (shallow)Last FCU nominated  a non-canonical chain  (chain B)Validate all blocks on Chain BReceive new beacon  blockExtract  ExecutionPayload  from blockengine_newPayloadV2(ExecutionPayload){status: VALID, ...} OR {status: INVALID, ...}

ACCEPTED

This is the case when a shallow-state client receives a non-canonical block.

In the previous example, it would mean that it gets a block for chain B when it's running chain A (Before the FCU call).

Created with Raphaël 2.2.0Block Validation, ACCEPTEDCLCLEL (shallow)EL (shallow)Receive new beacon  blockExtract  ExecutionPayload  from blockengine_newPayloadV2(ExecutionPayload)Payload is  non-canonical{status: ACCEPTED, ...}

Validation failed

There is a scenario where this process can fail.

Wrong ExecutionPayload version

Very similar to the error caused by using the wrong PayloadAttributes version on block building. The version of the ExecutionPayload that we are trying to validate must be consistent with the Shanghai timestamp.

  • ExecutionPayloadV1 MUST be used if the timestamp value is lower than the Shanghai timestamp,
  • ExecutionPayloadV2 MUST be used if the timestamp value is greater or equal to the Shanghai timestamp,
  • Client software MUST return -32602: Invalid params error if the wrong version of the structure is used in the method call.
Created with Raphaël 2.2.0Block Validation Failed: Wrong ExecutionPayload VersionCLCLELELReceive new beacon  blockExtract  ExecutionPayload  from blockengine_newPayloadV2(ExecutionPayload)Wrong  ExecutionPayload  version-32602: Invalid params

All together

Now, here's a diagram showing what the full lifetime of a validator node would look like. Something to note here is the timing of the calls relative to where slots start or end.

Something that might seem odd is the call to engine_forkchoiceUpdatedV2 with PayloadParams even when the validator is not selected to propose. This is an optimization validators can make to have more time to build a block (and therefore, build a more profitable block).

Created with Raphaël 2.2.0Validator lifetimeCLCLELELNode is startedengine_exchangeCapabilities(engine methods supported by CL)engine methods supported by ELengine_forkchoiceUpdatedV2(ForkchoiceState, null)I don't have all the  necessary data available,  syncing{payloadStatus: {status: SYNCING, ...}, payloadId: null}...Sync is completedengine_forkchoiceUpdatedV2(ForkchoiceState, null){payloadStatus: {status: VALID, ...}, payloadId: null}Slot where validator doesn't have to propose startsengine_forkchoiceUpdatedV2(ForkchoiceState, null){payloadStatus: {status: VALID, ...}, payloadId: null}Receive new  beacon blockExtract  ExecutionPayload  from blockengine_newPayloadV2(ExecutionPayload)All requirements  are met and the  payload is considered  valid{status: VALID, ...}engine_forkchoiceUpdatedV2(ForkchoiceState, PayloadAttributes){payloadStatus: {status: VALID, ...}, payloadId: buildProcessId}Start building  execution_payloadSlot where validator doesn't have to propose endsSlot where validator has to propose startsengine_forkchoiceUpdatedV2(ForkchoiceState, null){payloadStatus: {status: VALID, ...}, payloadId: null}Fill BeaconBlock  up to  execution_payloadengine_getPayloadV2(PayloadId){executionPayload, blockValue}Put the  ExecutionPayload  on the BeaconBlockCompute state_rootPropagate blockSlot where validator has to propose endsMore slots of both types would happen again until the node is shutdown