# Engine API/Consensus Layer Mock Simulator - Test Results Tracker

## Description

Tests described in this document run using the hive simulator [ethereum/engine](https://github.com/ethereum/hive/tree/master/simulators/ethereum/engine).

Tests use a Consensus Layer Mock to simulate test scenarios on the Execution Clients.

Merge-compatible client must be used in hive to be able to run these tests (Example: [merge-go-ethereum](https://github.com/ethereum/hive/tree/master/clients/merge-go-ethereum))

## Versions used for testing:

Geth: github.com/MariusVanDerWijden/go-ethereum/commits/merge-kiln-v2 @23ce1ab

Nethermind: Docker [nethermindeth/nethermind:kiln_0.5 @ cb1a890ef672c6827d7b10bf1af5c7a6e29a8e6d5cb1f93d315e7d6b6ab5745a](https://hub.docker.com/layers/nethermindeth/nethermind/kiln_0.5/images/sha256-cb1a890ef672c6827d7b10bf1af5c7a6e29a8e6d5cb1f93d315e7d6b6ab5745a?context=explore)

EthereumJS: https://github.com/ethereumjs/ethereumjs-monorepo @739f839

Test Vectors:
Kiln Public Testnet (Kiln v1 + prevRandao rename): github.com/marioevz/hive/tree/engine-api-simulator-kiln-public-testnet @777aced

Previous Test Vectors:
Stable (Kiln v1): github.com/ethereum/hive @875e6ac

## Current Test Case Results

(🔥🧱 == Test case is running successfully)

### Totals
| Geth | Nethermind | Besu | Erigon | EthereumJS |
| -------- | -------- | -------- | -------- | -------- |
| 42/43 | 33/43 | | |31/43

### Engine API Negative Test Cases:
| Test Case | Geth | Nethermind | Besu | Erigon | EthereumJS |
| -------- | -------- | -------- | -------- | -------- | -------- |
| Invalid Terminal Block in ForkchoiceUpdated | 🔥🧱| 🔥🧱 | | |🔥🧱
| Invalid GetPayload Under PoW | 🔥🧱| 🔥🧱 | | | 🔥🧱
| Invalid Terminal Block in NewPayload | 🔥🧱| 🔥🧱 | | |🔥🧱
| Unknown HeadBlockHash | 🔥🧱| 🔥🧱 | | | 🔥🧱
| Unknown SafeBlockHash | 🔥🧱| 🔥🧱 | | |🔥🧱
| Unknown FinalizedBlockHash | 🔥🧱| 🔥🧱| | |🔥🧱
| Pre-TTD Block Hash | 🔥🧱| 🔥🧱 | | | 🔥🧱
| Bad blockhash on NewPayload | 🔥🧱| 🔥🧱 | | | 🔥🧱
| ParentHash==BlockHash on NewPayload | 🔥🧱| 🔥🧱 | | | 🔥🧱
| Invalid Field in NewPayload / ParentHash | 🔥🧱| 🔥🧱 | | |
| Invalid Field in NewPayload / StateRoot | 🔥🧱| 🔥🧱 | | | 🔥🧱
| Invalid Field in NewPayload / ReceiptsRoot | 🔥🧱| 🔥🧱 | | | 🔥🧱
| Invalid Field in NewPayload / BlockNumber | 🔥🧱| 🔥🧱 | | | 🔥🧱
| Invalid Field in NewPayload / GasLimit | 🔥🧱| 🔥🧱 | | | 🔥🧱
| Invalid Field in NewPayload / GasUsed | 🔥🧱| 🔥🧱 | | | 🔥🧱
| Invalid Field in NewPayload / Timestamp | 🔥🧱| 🔥🧱 | | | 🔥🧱
| Invalid Field in NewPayload / PrevRandao | 🔥🧱| 🔥🧱 | | | 🔥🧱
| Invalid Field in NewPayload / Incomplete Transactions | 🔥🧱| 🔥🧱 | | | 🔥🧱
| Invalid Field in NewPayload Transaction / Signature | 🔥🧱| 🔥🧱 | | | 🔥🧱
| Invalid Field in NewPayload Transaction / Nonce | 🔥🧱| 🔥🧱| | | 🔥🧱
| Invalid Field in NewPayload Transaction / GasPrice | 🔥🧱| 🔥🧱| | | 🔥🧱
| Invalid Field in NewPayload Transaction / Gas | 🔥🧱| 🔥🧱| | | 🔥🧱
| Invalid Field in NewPayload Transaction / Value | 🔥🧱| 🔥🧱| | | 🔥🧱

### Eth RPC Status on ForkchoiceUpdated Events:
| Test Case | Geth | Nethermind | Besu | Erigon | EthereumJS |
| -------- | -------- | -------- | -------- | -------- | -------- |
| Latest Block after NewPayload | 🔥🧱| 🔥🧱| | | 🔥🧱
| Latest Block after New HeadBlock | 🔥🧱| 🔥🧱| | | 🔥🧱
| Latest Block after New SafeBlock | 🔥🧱| 🔥🧱| | | 🔥🧱
| Latest Block after New FinalizedBlock | 🔥🧱| 🔥🧱| | | 🔥🧱
| Latest Block after Reorg | 🔥🧱| | | | 🔥🧱

### Payload Execution
| Test Case | Geth | Nethermind | Besu | Erigon | EthereumJS |
| -------- | -------- | -------- | -------- | -------- | -------- |
| Re-Execute Payload | 🔥🧱| 🔥🧱| | | 🔥🧱
| Multiple New Payloads Extending Canonical Chain | 🔥🧱| 🔥🧱| | | 🔥🧱
| Out of Order Payload Execution | | | | |

### Transaction Reorg using Engine API
| Test Case | Geth | Nethermind | Besu | Erigon | EthereumJS |
| -------- | -------- | -------- | -------- | -------- | -------- |
| Transaction Reorg using ForkchoiceUpdated | 🔥🧱|
| Sidechain Reorg | 🔥🧱| 🔥🧱 | | | 🔥🧱

### Suggested Fee Recipient in Payload creation
| Test Case | Geth | Nethermind | Besu | Erigon | EthereumJS |
| -------- | -------- | -------- | -------- | -------- | -------- |
| Suggested Fee Recipient Test | 🔥🧱| 🔥🧱 | | | 🔥🧱

### PrevRandao Opcode
| Test Case | Geth | Nethermind | Besu | Erigon | EthereumJS |
| -------- | -------- | -------- | -------- | -------- | -------- |
| PrevRandao Opcode Transactions | 🔥🧱| 🔥🧱 | | | 🔥🧱

### Sync Tests
| Test Case | Geth | Nethermind | Besu | Erigon | EthereumJS |
| -------- | -------- | -------- | -------- | -------- | -------- |
| Sync Client Post Merge | 🔥🧱|

### Merge Tests
| Test Case | Geth | Nethermind | Besu | Erigon | EthereumJS |
| -------- | -------- | -------- | -------- | -------- | -------- |
| Single Block PoW Re-org to Higher-Total-Difficulty Chain, Equal Height | 🔥🧱|
| Single Block PoW Re-org to Lower-Total-Difficulty Chain, Equal Height | 🔥🧱|
| Two Block PoW Re-org to Higher-Total-Difficulty Chain, Equal Height | 🔥🧱|
| Two Block PoW Re-org to Lower-Total-Difficulty Chain, Equal Height | 🔥🧱|
| Two Block PoW Re-org to Higher-Height Chain | 🔥🧱|
| Two Block PoW Re-org to Lower-Height Chain |🔥🧱|
| Halt following PoW chain | 🔥🧱| 🔥🧱

---

## Full Description of Test Cases
### Engine API Negative Test Cases:
- Invalid Terminal Block in ForkchoiceUpdated:
Client should reject ForkchoiceUpdated directives if the referenced HeadBlockHash does not meet the TTD requirement.

- Invalid GetPayload Under PoW:
Client must reject GetPayload directives under PoW.

- Invalid Terminal Block in NewPayload:
Client must reject NewPayload directives if the referenced ParentHash does not meet the TTD requirement.

- Unknown HeadBlockHash:
Perform a forkchoiceUpdated call with an unknown (random) HeadBlockHash, the client should initiate the syncing process.

- Unknown SafeBlockHash:
Perform a forkchoiceUpdated call with an unknown (random) SafeBlockHash, the client should throw an error since the hash is not an ancestor to the HeadBlockHash.

- Unknown FinalizedBlockHash:
Perform a forkchoiceUpdated call with an unknown (random) FinalizedBlockHash, the client should throw an error.

- Pre-TTD Block Hash:
Perform a forkchoiceUpdated call using a block hash part of the canonical chain that precedes the block where the TTD occurred. (Behavior is undefined for this edge case and not verified, but should not produce unrecoverable error)

- Bad blockhash on NewPayload:
Send a NewPayload directive to the client including an incorrect BlockHash, should result in an error.

- ParentHash==BlockHash on NewPayload:
Send a NewPayload directive to the client including ParentHash that is equal to the BlockHash (Incorrect hash).

- Invalid Field in NewPayload:
Modify fields of the ExecutablePayload while maintaining a valid BlockHash, including:
- ParentHash
- StateRoot
- ReceiptsRoot
- BlockNumber
- GasLimit
- GasUsed
- Timestamp
- PrevRandao
- Removed Transaction
- Transaction with incorrect fields:
- Signature
- Nonce
- GasPrice
- Gas
- Value

### Eth RPC Status on ForkchoiceUpdated Events:
- Latest Block after NewPayload:
Verify the Block returned by the Eth RPC after a new payload is executed. Eth RPC should still return previous block.

- Latest Block after New HeadBlock:
Verify the Block returned by the Eth RPC after a new HeadBlockHash is set using forkchoiceUpdated. Eth RPC should return new block.

- Latest Block after New SafeBlock:
Verify the Block returned by the Eth RPC after a new SafeBlockHash is set using forkchoiceUpdated. Eth RPC should return new block.

- Latest Block after New FinalizedBlock:
Verify the Block returned by the Eth RPC after a new FinalizedBlockHash is set using forkchoiceUpdated. Eth RPC should return new block.

- Latest Block after Reorg:
Verify the Block returned by the Eth RPC after a forkchoiceUpdated reorgs HeadBlockHash/SafeBlockHash to their previous value. Eth RPC should return previous block.

### Payload Execution
- Re-Execute Payload:
Re-execute already executed payloads (10) and verify that no errors occur.

- Multiple New Payloads Extending Canonical Chain:
Send 80 valid NewPayload directives extending the canonical chain. Only one of the payloads is selected using ForkchoiceUpdated directive.

- Out of Order Payload Execution:
Launch a first client and produce N payloads.
Launch a second client and send payloads (NewPayload) in reverse order (N, N - 1, ..., 1).
The payloads should be ACCEPTED/SYNCING, and the last payload should be VALID (since payload 1 successfully links the chain with the Genesis).

### Transaction Reorg using Engine API
- Transaction Reorg using ForkchoiceUpdated:
Send transactions that modify the state tree after the PoS switch and verify that the modifications are correctly rolled back when a ForkchoiceUpdated event occurs with a block older than the block where the transaction was included.

- Sidechain Reorg:
Send a transaction that modifies the state, ForkchoiceUpdate to the payload containing this transaction. Then send an alternative payload, that produces a different state with the same transaction, and use ForkchoiceUpdated to change into this sidechain.

### Suggested Fee Recipient in Payload creation
- Suggested Fee Recipient Test:
Set the fee recipient to a custom address and verify that (a) balance is not increased when no fees are collected (b) balance is increased appropriately when fees are collected.

### PrevRandao Opcode:
- PrevRandao Opcode Transactions:
Send transactions that modify the state to the value of the 'DIFFICULTY' opcode and verify that:
(a) the state is equal to the difficulty on blocks before the TTD is crossed
(b) the state is equal to the PrevRandao value provided using forkchoiceUpdated after PoS transition.

### Sync Tests:
- Sync Client Post Merge:
Launch a first client and verify that the transition to PoS occurs.
Launch a second client and verify that it successfully syncs to the first client's chain.

### Engine API Merge Tests:

Test cases using multiple Proof of Work chains to test the client's behavior when the Terminal Total Difficulty is reached by two different blocks simultaneously and the Engine API takes over.

- Single Block PoW Re-org to Higher-Total-Difficulty Chain, Equal Height:
Client 1 starts with chain G -> A, Client 2 starts with chain G -> B.
Blocks A and B reach TTD, but block B has higher difficulty than A.
ForkchoiceUpdated is sent to Client 1 with A as Head.
ForkchoiceUpdated is sent to Client 1 with B as Head.
Verification is made that Client 1 Re-orgs to chain G -> B.

- Single Block PoW Re-org to Lower-Total-Difficulty Chain, Equal Height:
Client 1 starts with chain G -> A, Client 2 starts with chain G -> B.
Blocks A and B reach TTD, but block A has higher difficulty than B.
ForkchoiceUpdated is sent to Client 1 with A as Head.
ForkchoiceUpdated is sent to Client 1 with B as Head.
Verification is made that Client 1 Re-orgs to chain G -> B.

- Two Block PoW Re-org to Higher-Total-Difficulty Chain, Equal Height:
Client 1 starts with chain G -> A -> B, Client 2 starts with chain G -> C -> D.
Blocks B and D reach TTD, but block D has higher total difficulty than B.
ForkchoiceUpdated is sent to Client 1 with B as Head.
ForkchoiceUpdated is sent to Client 1 with D as Head.
Verification is made that Client 1 Re-orgs to chain G -> C -> D.

- Two Block PoW Re-org to Lower-Total-Difficulty Chain, Equal Height:
Client 1 starts with chain G -> A -> B, Client 2 starts with chain G -> C -> D.
Blocks B and D reach TTD, but block B has higher total difficulty than D.
ForkchoiceUpdated is sent to Client 1 with B as Head.
ForkchoiceUpdated is sent to Client 1 with D as Head.
Verification is made that Client 1 Re-orgs to chain G -> C -> D.

- Two Block PoW Re-org to Higher-Height Chain:
Client 1 starts with chain G -> A, Client 2 starts with chain G -> B -> C.
Blocks A and C reach TTD, but block C has higher height than A.
ForkchoiceUpdated is sent to Client 1 with A as Head.
ForkchoiceUpdated is sent to Client 1 with C as Head.
Verification is made that Client 1 Re-orgs to chain G -> B -> C.

- Two Block PoW Re-org to Lower-Height Chain:
Client 1 starts with chain G -> A -> B, Client 2 starts with chain G -> C.
Blocks B and C reach TTD, but block B has higher height than C.
ForkchoiceUpdated is sent to Client 1 with B as Head.
ForkchoiceUpdated is sent to Client 1 with C as Head.
Verification is made that Client 1 Re-orgs to chain G -> C.

- Halt following PoW chain:
Client 1 starts with chain G -> A, Client 2 starts with chain G -> A -> B.
Block A reaches TTD, but Client 2 has a higher TTD and accepts block B (simulating a client not complying with the merge).
Verification is made that Client 1 does not follow Client 2 chain to block B.