# Antithesis Report 25.09.23
This dencun-devnet-8 report includes the results of our sync testing experiments. In these experiments we stopped and started containers at random for a random amount of time. Below is a summary of the most interesting findings.
The repository and branch of the clients used:
```dockerfile
# Consensus Clients
ARG LIGHTHOUSE_REPO="https://github.com/sigp/lighthouse"
ARG LIGHTHOUSE_BRANCH="ce824e00a3566e7e94e77600d97aab5be3b9a99c"
ARG PRYSM_REPO="https://github.com/prysmaticlabs/prysm.git"
ARG PRYSM_BRANCH="c728e0923fc451c151c7ffe373fad3b57fab018e"
ARG NIMBUS_ETH2_REPO="https://github.com/status-im/nimbus-eth2.git"
ARG NIMBUS_ETH2_BRANCH="8993d57"
ARG TEKU_REPO="https://github.com/ConsenSys/teku.git"
ARG TEKU_BRANCH="b49165a76c6aaa56fad01eaa2a80c18b4084c81f"
# Execution Clients
ARG BESU_REPO="https://github.com/hyperledger/besu.git"
ARG BESU_BRANCH="c16a3274b30a96678388dd616006d5919085200e"
ARG GETH_REPO="https://github.com/lightclient/go-ethereum.git"
ARG GETH_BRANCH="79f3c2d9c96bd82320d44851074da76587e41887"
ARG NETHERMIND_REPO="https://github.com/NethermindEth/nethermind.git"
ARG NETHERMIND_BRANCH="84d41bd174fd6c4f8350c5c6f0d4b746acee1c51"
```
The client pairs we launched each with 10 validators attached:
* 10.0.20.10 prysm-geth
* 10.0.20.20 teku-geth
* 10.0.20.50 nimbus-geth
* 10.0.20.80 lighthouse-besu
* 10.0.20.110 prysm-nethermind
* 10.0.20.130 lighthouse-nethermind
* 10.0.20.150 nimbus-nethermind
## Consensus
### Lighthouse
1. Lighthouse-bn (Besu)
Invalid block proposed by lighthouse-besu. This is similar to the error in the last report, but this error did not result in a CRIT on the side of lighthouse-besu. Below are the logs of lighthouse-nethermind trying to validate the invalid block.
```l!
[ 0] [ 802.147119] [service_lighthouse-nethermind-0--nethermind] [I] 12 Sep 14:09:56 | Received new block: 41 (0x400921...e29774)
[ 0] [ 802.150677] [service_lighthouse-nethermind-0--nethermind] [I] 12 Sep 14:09:56 | InvalidBlockHash. Result of new block: 41 (0x400921...e29774).
[ 0] [ 802.385225] [service_lighthouse-nethermind-0--nethermind] [I] 12 Sep 14:09:56 | Invalid block hash 0x400921b131d79e63aebb231c2d70e035f7bb6fa028372715d3631f72cce29774, expected: 0x3224d98cde8f6ccdc856f2a8cb6cf289f2ee1a3a516ec2d6e31f2cbc0a302bcb., RLP: f9023da09ac8d7e72abc43301454f076d4af573629a4fc753d4125609879f122bf784eb1a01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d493479400000000219ab540356cbb839cbe05303d7705faa0fe495f78696a18268cfef2e6180285678874ab9656fafd2b25a7446e3b89ff4ea056e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421a056e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421b901000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000080298401815e5580846500712e80a0e45cc5932b18e708a8cee26e934e091b16dacb4e88daa8ab6f96e3d761e7ae36880000000000000000833ff394a056e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b4218080a0b448770cdcd0e610f7aa8d17edaf5282f2251bf4ae431fbebbe1b36ec2f69cf1
[ 0] [ 802.413300] [service_lighthouse-nethermind-0--lighthouse-bn] [I] Sep 12 14:09:56.291 WARN Invalid execution payload method: new_payload, slot: 64, proposer_index: 32, graffiti: lighthouse-besu-0, root: 0x4201d86999bd621815c3c549e6fa85b6add586d9945b42a84c17b368d45a111b, execution_block_hash: 0x400921b131d79e63aebb231c2d70e035f7bb6fa028372715d3631f72cce29774, latest_valid_hash: None, validation_error: Some("Invalid block hash 0x400921b131d79e63aebb231c2d70e035f7bb6fa028372715d3631f72cce29774."), service: beacon
```
The invalid block headers:
```json
{
"isGenesis": false,
"parentHash": "0x9ac8d7e72abc43301454f076d4af573629a4fc753d4125609879f122bf784eb1",
"unclesHash": "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
"author": "0x00000000219ab540356cbb839cbe05303d7705fa",
"beneficiary": "0x00000000219ab540356cbb839cbe05303d7705fa",
"gasBeneficiary": "0x00000000219ab540356cbb839cbe05303d7705fa",
"stateRoot": "0xfe495f78696a18268cfef2e6180285678874ab9656fafd2b25a7446e3b89ff4e",
"txRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
"receiptsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
"bloom": "0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
"difficulty": "0x0",
"number": "0x29",
"gasUsed": "0x0",
"gasLimit": "0x1815e55",
"timestamp": "0x6500712e",
"timestampDate": "2023-09-12T14:09:50+00:00",
"extraData": "0x",
"mixHash": "0xe45cc5932b18e708a8cee26e934e091b16dacb4e88daa8ab6f96e3d761e7ae36",
"random": "0xe45cc5932b18e708a8cee26e934e091b16dacb4e88daa8ab6f96e3d761e7ae36",
"nonce": "0x0",
"hash": "0x400921b131d79e63aebb231c2d70e035f7bb6fa028372715d3631f72cce29774",
"baseFeePerGas": "0x3ff394",
"withdrawalsRoot": "0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421",
"parentBeaconBlockRoot": "0xb448770cdcd0e610f7aa8d17edaf5282f2251bf4ae431fbebbe1b36ec2f69cf1",
"blobGasUsed": "0x0",
"excessBlobGas": "0x0",
"hasBody": false,
"hasTransactions": false,
"sealEngineType": "Ethash",
"isPostMerge": true
}
```
2. Lighthouse-vc (Besu) What does it mean if the payload id is unavailable?
```l!
Sep 12 14:24:28.613 CRIT Error whilst producing block info: proposal did not use a builder, block_slot: Slot(137), message: Recoverable("Some endpoints failed, num_failed: 1 http://127.0.0.1:5000/ => RequestFailed(Recoverable(\"Error from beacon node when producing block: ServerMessage(ErrorMessage { code: 500, message: \\\"UNHANDLED_ERROR: GetPayloadFailed(EngineError(Api { error: PayloadIdUnavailable }))\\\", stacktraces: [] })\"))"), service: block
```
[Antithesis Report](https://ethereum.antithesis.com/report/yUHgDq98Keh3fhF6BFYqb7HYWEHjqOo0/eJMMr80ZZQuKOfbmjkPjo1m1muT7EB8rg4ARzCBKYPE.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiZUpNTXI4MFpaUXVLT2ZibWprUGpvMW0xbXVUN0VCOHJnNEFSekNCS1lQRS5odG1sIiwicmVwb3J0X2lkIjoieVVIZ0RxOThLZWgzZmhGNkJGWXFiN0hZV0VIanFPbzAifX0sIm5iZiI6IjIwMjMtMDktMjVUMDk6MjE6MzVaIn1fYmQnQR2erVTqANKYDaW4Dwa0tQ7M_1IM7abbMeRgT5bMymg4-Hn-6pKOY1C-37uij1YwKAXcpINP0FT9UpoE)
3. Lighthouse-vc (Besu)
```l!
Sep 12 14:34:54.028 CRIT Failed to sign attestation slot: 189, committee_index: 0, validator: 0x8bb0b972c71fe3c36b49924ccf59126f4d4721b634d18d6280060f672a5af07d2ec6cd5e6f7ed181ad8d6b947c8a93ad, error: Slashable(InvalidAttestation(DoubleVote(SignedAttestation { source_epoch: Epoch(3), target_epoch: Epoch(5), signing_root: SigningRoot(0x447baf5d60b158547e342ef21f6f027dda633437540972bfb97c65c5683cc141) }))), service: attestation
```
[Antithesis Report](https://ethereum.antithesis.com/report/Vw2j9YgkUd20pjVrBWp68AL_YhV21jm6/TLZ8M7Q9LdGlgSxGH4X629N9ohztKG4OeltZ72tD28o.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiVExaOE03UTlMZEdsZ1N4R0g0WDYyOU45b2h6dEtHNE9lbHRaNzJ0RDI4by5odG1sIiwicmVwb3J0X2lkIjoiVncyajlZZ2tVZDIwcGpWckJXcDY4QUxfWWhWMjFqbTYifX0sIm5iZiI6IjIwMjMtMDktMjVUMDk6MzY6NTBaIn2Dr6tKt0QhWG7dNO1CIXNIRQCZAPtg2IhO8NfJVr29Wf_wx-fSEzSLXaqfhd-pHWGp-kB6o0b40NiwcJKosGoJ)
[Antithesis Report](https://ethereum.antithesis.com/report/EyH4_RGFpMwWVzVa8Zy-FLasIEtyG6Z0/BhRUaHrkvASJgmy_2Xzbx5wdwTXUx8aapqm3gkxvXzM.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiQmhSVWFIcmt2QVNKZ215XzJYemJ4NXdkd1RYVXg4YWFwcW0zZ2t4dlh6TS5odG1sIiwicmVwb3J0X2lkIjoiRXlINF9SR0ZwTXdXVnpWYThaeS1GTGFzSUV0eUc2WjAifX0sIm5iZiI6IjIwMjMtMDktMjBUMDk6MzY6MTFaIn3Nhsd22h_2MUFia9DC2euxqnxY-YEbGcowLdvXeuOXimNq6yBXix6RPcRVb99cuvKM1YZaj8xk-tPuXUfIjwEK)
### Prysm
1. Prysm-bn (Geth)
It seems that the function to validate the blob sidecars is called with an empty array. Might be worth looking into why SaveBlobSideCar is being called that way.
```l!
time="2023-09-12 14:22:19" level=warning msg="Failed to save blob sidecar" error="nil or empty blob sidecars" prefix=initial-sync
```
[Antithesis Report](https://ethereum.antithesis.com/report/ZbOb0LBm1Nb3W1vgnYz7gJo31knh4jrF/zMQdoI9uKUY1lo98VOXIDYiNTkIYsvEY_RrU50kD7gU.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0Ijoiek1RZG9JOXVLVVkxbG85OFZPWElEWWlOVGtJWXN2RVlfUnJVNTBrRDdnVS5odG1sIiwicmVwb3J0X2lkIjoiWmJPYjBMQm0xTmIzVzF2Z25ZejdnSm8zMWtuaDRqckYifX0sIm5iZiI6IjIwMjMtMDktMjBUMTM6MDc6MzZaIn1hxxX9_qWoJ_l1MYvM9iEDTKm7-28A5EeVTLFHfIu6CM03-cZW9PdhjIr8sdbi8HA65LolwzyUn9rr8h_b51YH)
2. Prysm-bn (Geth)
Prysm node continues to sync even if the blob sidecar is not available. Payload number 94 hash 033320 slot 135
```l!
[ 272886] [ 1649.116760] [service_prysm-nethermind-0--prysm-bn] [I] time="2023-09-12 14:24:03" level=debug msg="Block proposal received via RPC" prefix="rpc/validator" slot=135
time="2023-09-12 14:24:02" level=debug msg="Synced new block" block=0x8214dc1f... chainServiceProcessedTime=325.866043ms deposits=0 epoch=4 finalizedEpoch=2 finalizedRoot=0xb448770c... justifiedEpoch=3 justifiedRoot=0x898b48de... parentRoot=0x03cbbe7e... prefix=blockchain sinceSlotStartTime=708.871066ms slot=135 slotInEpoch=7 version=deneb
level=debug msg="BlobSidecar not found in db, root=8214dc1ffd8880c995b6d690316b639e1c73d5258a8254d914e9a9b08bacf458, index=0" error="not found in db" handler="blob_sidecars_by_root" prefix=sync
time="2023-09-12 14:24:15" level=debug msg="Synced new block" block=0x5354a2f3... chainServiceProcessedTime=341.47501ms deposits=0 epoch=4 finalizedEpoch=2 finalizedRoot=0xb448770c... justifiedEpoch=3 justifiedRoot=0x898b48de... parentRoot=0x8214dc1f... prefix=blockchain sinceSlotStartTime=1.457941412s slot=136 slotInEpoch=8 version=deneb
time="2023-09-12 14:24:31" level=debug msg="BlobSidecar not found in db, root=8214dc1ffd8880c995b6d690316b639e1c73d5258a8254d914e9a9b08bacf458, index=0" error="not found in db" handler="blob_sidecars_by_root" prefix=sync
Number: 94
Parent: 0x1d38c60170a18ff8f937da6ac2bf21dcc2a84ecb7e8c2e3c4c554cf8f3948093
Beneficiary: 0x00000000219ab540356cbb839cbe05303d7705fa
Gas Limit: 25928856
Gas Used: 477000
Timestamp: 1694528642
Extra Data: 4e65746865726d696e64
Difficulty: 0
Mix Hash: 0x49f5af35a960b4217e4cfa36826784ae279b1448ed58fb72d783801cd3a30371
Nonce: 0
Uncles Hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347
Tx Root: 0x9224ca3569a733058fdbcf01dd6f331a3d32ae44517aa7559824acd8d6475169
Receipts Root: 0xf3d2f1d771c68170248a4a2b67ca20e818cea9f6dcffe2a2159f2788baa8005c
State Root: 0xd700ba72f70ed6c89698acc791ebad97b580fb2b54a2a86bb9a40dba56689260
BaseFeePerGas: 20447
WithdrawalsRoot: 0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421
ParentBeaconBlockRoot: 0x03cbbe7ea4b7def6ba9cf17c9376a3c9f3f0760dc00d49780dadcf8008bf7359
BlobGasUsed: 0
ExcessBlobGas: 393216
IsPostMerge: True
TotalDifficulty: 1
```
[Antithesis Report](https://ethereum.antithesis.com/report/oNcpUvDTlBDcpMcL2O-36sqptt9gMc8v/Ziii0FMeR2efYVe7dPH6Md3ee4XhIVBNt41hfOOsiOc.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTIxVDE1OjAyOjUwWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJaaWlpMEZNZVIyZWZZVmU3ZFBINk1kM2VlNFhoSVZCTnQ0MWhmT09zaU9jLmh0bWwiLCJyZXBvcnRfaWQiOiJvTmNwVXZEVGxCRGNwTWNMMk8tMzZzcXB0dDlnTWM4diJ9fX0hngGMAg_vi5aR2kFvTvqQMyfujvkHeZmvXSF4jQ9qzmpKX7AXB2cy1KPfC6yq0KPr62KyxJGm79gnH-U7vTEC)
3. Prysm-bn (Geth)
```l!
time="2023-09-12 14:21:51" level=debug msg="Invalid status message from peer" error="invalid finalized root" handler=status peer=16Uiu2HAmLjUm9UUuDnjpbdiwGxstYn5gV2Xf3jtJwJvmfQtQWQRv prefix=sync
```
[Antithesis Report](https://ethereum.antithesis.com/report/ok_MRShTqYDuXsNaLC5KRGJ9eMSG-snp/2DWiIk1IgBScZi-ATkzbwZQq4rMTi8f1Me3vOq-OX6s.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTI1VDA4OjQ2OjQ2WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiIyRFdpSWsxSWdCU2NaaS1BVGt6YndaUXE0ck1UaThmMU1lM3ZPcS1PWDZzLmh0bWwiLCJyZXBvcnRfaWQiOiJva19NUlNoVHFZRHVYc05hTEM1S1JHSjllTVNHLXNucCJ9fX3KdcCRPvvpdc8ynjKiOjwdtJECrEkCRV_MkhMD6eGxOG2ED5BMgeTjABIdnVzIRz6l_5ILvZvO59DNmfc3MskL)
5. Prysm-bn (Nethermind) It seems that blob_sidecars_by_root is not yet supported at the commit version we are running.
```l!
[ 282361] [ 1755.457751] [service_prysm-nethermind-0--prysm-bn] [I] time="2023-09-12 14:25:49" level=debug msg="Sending blob sidecar request" prefix=sync topic="/eth2/beacon_chain/req/blob_sidecars_by_root/1"
[ 282361] [ 1755.461912] [service_prysm-nethermind-0--prysm-bn] [I] time="2023-09-12 14:25:49" level=debug msg="Could not request pending blobs" error="protocols not supported: [/eth2/beacon_chain/req/blob_sidecars_by_root/1/ssz_snappy]" prefix=sync slot=140
```
[Antithesis Report](https://ethereum.antithesis.com/report/IzZgBJHm7LbqFKu6n6Gczw498n2oxgZN/VEp3q9st0tjVwdk40abANCe7DtABOPzew2Aqb0Va9Cg.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiVkVwM3E5c3QwdGpWd2RrNDBhYkFOQ2U3RHRBQk9QemV3MkFxYjBWYTlDZy5odG1sIiwicmVwb3J0X2lkIjoiSXpaZ0JKSG03TGJxRkt1Nm42R2N6dzQ5OG4yb3hnWk4ifX0sIm5iZiI6IjIwMjMtMDktMjVUMDk6MDc6MTdaIn1T_gfDfuaCq9GPocRadWS0KntgwMlh5zdB0AuyMFJV_KU3EZpuEkQ5h-4i97-jA1v1V2X0Sjpdk5yW2TWREwAF)
### Nimbus
1. Nimbus (Geth)
```l!
WRN 2023-09-12 14:45:58.094+00:00 Aggregated attestation failed validation topics="message_router" attestation="(aggregation_bits: 0b10, data: (slot: 244, index: 0, beacon_block_root: \"e36c0532\", source: \"4:07ebcea8\", target: \"7:e36c0532\"), signature: \"aa1b3738\")" aggregator_index=26 signature=8c1d7c2a error="(Reject, \"Aggregate: aggregator\\'s validator index not in committee\")"
```
[Antithesis Report](https://ethereum.antithesis.com/report/haOBuaLyzUvP-1__9FigNahAbGlDLbNr/SZU4KPebeJpK2TMy1lKF2weLxX76Ug8APijIZCVuvUE.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiU1pVNEtQZWJlSnBLMlRNeTFsS0Yyd2VMeFg3NlVnOEFQaWpJWkNWdXZVRS5odG1sIiwicmVwb3J0X2lkIjoiaGFPQnVhTHl6VXZQLTFfXzlGaWdOYWhBYkdsRExiTnIifX0sIm5iZiI6IjIwMjMtMDktMjVUMDk6MTQ6MDlaIn1jFBIq7qfYPtrZzLrVQR9UcO5EMjdfJzqnhMiWOrOJwjYzXzqd7toAO3oRbdpZP4WC27IPVrckJiJAkg2iJwMD)
2. Nimbus (Nethermind)
```l!
DBG 2023-09-12 15:10:44.727+00:00 Dropping blob topics="gossip_eth2" error="(Ignore, \"SignedBlobSidecar: already have block\")" blob="(block_root: \"b0f0dcc9\", index: 3, slot: 368, block_parent_root: \"10a058e9\", proposer_index: 2, bloblen: 131072)" wallSlot=368 signature=98f4a471
```
[Antithesis Report](https://ethereum.antithesis.com/report/bNzvohRWD-j-dx0vrh-i4WhMzvJI2BPw/Oe17HS3pNPEoGPx3eexWUsbBzSHGQLAkIoMpa5rOeYY.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiT2UxN0hTM3BOUEVvR1B4M2VleFdVc2JCelNIR1FMQWtJb01wYTVyT2VZWS5odG1sIiwicmVwb3J0X2lkIjoiYk56dm9oUldELWotZHgwdnJoLWk0V2hNenZKSTJCUHcifX0sIm5iZiI6IjIwMjMtMDktMjVUMDk6MTc6NTFaIn3UA2tZCP_LEVwDq2aGVOzJJrRu_E46c4YYvEZW48j1JfhCi2CU5CUG8HX_xlcgeDM44qsd96wbqcIHwJ3FWWEB)
### Teku
1. Teku (Geth) It seems that teku geth does not recover well and starts getting blocked by other peers.
```l!
Sep 12 14:42:48.054 WARN Invalid parent chain block_peer_id: 16Uiu2HAkvimJL7CDoPoWaJndLMnh818Vb4hrC7HFVwRtHf6VPrFa, outcome: WouldRevertFinalizedSlot { block_slot: Slot(114), finalized_slot: Slot(160) }, score_adjustment: Mid Tolerance Error, service: sync
```
[Antithesis Report](https://ethereum.antithesis.com/report/70etT7VLNaM4lRlZuI47skgQXon9KlMJ/_ttlTuaoo0kr9ngrVGKeFMZvvriXphmY2pP6rVt--NI.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTIwVDA5OjMyOjE0WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJfdHRsVHVhb28wa3I5bmdyVkdLZUZNWnZ2cmlYcGhtWTJwUDZyVnQtLU5JLmh0bWwiLCJyZXBvcnRfaWQiOiI3MGV0VDdWTE5hTTRsUmxadUk0N3NrZ1FYb245S2xNSiJ9fX0h7CreGZW0-qFGDOFvWp_Z460i7AiuH3Zt5S0OMHaBC3On-Et3ZFsReDEZ7BPZsgpKZ0p4THXab8O4gpXVi4kD)
2. Teku (Geth) Is it technically possible to sign a second attestation at the same height if the previous attestation becomes invalid due to a reorg event?
```l!
2023-09-12 14:13:30.003 INFO - Reorg Event *** New Head: 65287aecb35e4302c320e1119501fbfc94537c6d9ca2bc04452a1249ff8b8731 (82), Previous Head: 124f13387ee5d6aaf748596aa5d8614c16488405d3857ae75d73800e7760ea2c (81), Common Ancestor: bf7d0db852dd696b71d0c3930dc7bd2151eaf04a650e2cc2f974b0163def2cf7 (80)
2023-09-12 14:13:30.033 INFO - Slot Event *** Slot: 82, Block: 65287aecb35e4302c320e1119501fbfc94537c6d9ca2bc04452a1249ff8b8731, Justified: 0, Finalized: 0, Peers: 6
2023-09-12 14:14:52.174 ERROR - Refusing to sign attestation: source epoch (0) is safe, target epoch (2) is unsafe. Previously signed source epoch 0, target epoch 2
2023-09-12 14:14:52.184 ERROR - Validator *** Failed to produce attestation Slot: 89 Validator: 87b8baa
```
[Antithesis Report](https://ethereum.antithesis.com/report/SPDzoDW1FtGGAg-Lx5oelK03e7dVdS16/mL858BTqX_5FHapUBG-kcYlIAyKixltdwXAD6o1qALU.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTIwVDA5OjAwOjQzWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJtTDg1OEJUcVhfNUZIYXBVQkcta2NZbElBeUtpeGx0ZHdYQUQ2bzFxQUxVLmh0bWwiLCJyZXBvcnRfaWQiOiJTUER6b0RXMUZ0R0dBZy1MeDVvZWxLMDNlN2RWZFMxNiJ9fX0CUbwtjlCIite9pZqXOFu-Uf_FHo7idMQZPrG8phazrBPhMTRX4rvtxIbzDYA-Mqh8FeGSVowGhcc6pL7mCe0N)
## Execution
### Nethermind
1. Nethermind (lighthouse) - Is there a retry mechanism for exchanging capabilities between the CL and EL. It seems the operation timed out and then there was a failed eth_getLogs call. I am not sure yet if they are related.
```l!
[lighthouse-bn] [I] Sep 12 13:59:26.488 WARN Error during exchange capabilities error: HttpClient(url: http://127.0.0.1:8551/, kind: timeout, detail: operation timed out), service: exec
[nethermind] [I] 12 Sep 13:59:26 | Request starting HTTP/1.1 POST http://127.0.0.1:8551/ application/json 229
[nethermind] [I] 12 Sep 13:59:26 | Request finished HTTP/1.1 POST http://127.0.0.1:8551/ application/json 364 - 200 - application/json 1125.4642ms
[lighthouse-bn] [I] Sep 12 13:59:26.899 ERRO Error updating deposit contract cache error: Failed to update deposit cache: GetDepositLogsFailed("eth_getLogs call failed ServerMessage { code: -32001, message: \"1 could not be found\" }"), retry_millis: 60000, service: deposit_contract_rpc
[nethermind] [I] 12 Sep 13:59:26 | Error when handling ID 1, eth_getLogs({
[nethermind] [I] "address": "0x8c594691c0e592ffa21f153a16ae41db5befcaaa",
[nethermind] [I] "fromBlock": "0x0",
[nethermind] [I] "toBlock": "0x1",
[nethermind] [I] "topics": [
[nethermind] [I] "0x649bbc62d0e31342afea4e5cd82d4049e7e1ee912fc0889aa790803be39038c5"
[nethermind] [I] ]
[nethermind] [I] }) | {"jsonrpc":"2.0","error":{"code":-32001,"message":"1 could not be found"},"id":1}
[nethermind] [I] 12 Sep 13:59:26 | Request finished HTTP/1.1 POST http://127.0.0.1:8551/ application/json 229 - 200 - application/json 593.4304ms
```
[Antithesis Report](https://ethereum.antithesis.com/report/wsspphKthgdvRjkRwPoWcN3FUf48QuTZ/Pikd1lhN1CwVdtvxiDpRFt3iOxrq1zTqE-ecyndK8N0.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiUGlrZDFsaE4xQ3dWZHR2eGlEcFJGdDNpT3hycTF6VHFFLWVjeW5kSzhOMC5odG1sIiwicmVwb3J0X2lkIjoid3NzcHBoS3RoZ2R2UmprUndQb1djTjNGVWY0OFF1VFoifX0sIm5iZiI6IjIwMjMtMDktMjBUMDg6NDg6NDVaIn1wwE2-B_YBKEWVoCBCkCe00kXhHqJ4z16m_yJnmJ2M_OgofpAjUR3U9OUltlc_iBgJ0NKTGVgE5v2gS4ROoqID )
2. Nethermind (lighthouse) it seems that a request to the execution engine caused an internal error.
```l!
lighthouse-bn Sep 12 14:25:29.485 WARN Execution engine call failed error: ServerMessage { code: -32603, message: "Internal error" }, service: exec
lighthouse-bn Sep 12 14:25:29.488 WARN Error processing HTTP API request method: GET, path: /eth/v2/validator/blocks/142, status: 500 Internal Server Error, elapsed: 3.111099839s
12 Sep 14:25:29 | Error during method execution, request: ID 1, engine_forkchoiceUpdatedV3({
"finalizedBlockHash": "0x9ac8d7e72abc43301454f076d4af573629a4fc753d4125609879f122bf784eb1",
"headBlockHash": "0xd6ace01457e5d479b783866d145864ec2afbb653a06a4a3708ed6c7b01130150",
"safeBlockHash": "0x7f6b04801ae6f0c6eb92fa15c7fa22dc2afce5c0a109ae6d7879d331375ba36a"
},{
"parentBeaconBlockRoot": "0x8dd9f4c4f89e80a699a1fb387923d54955061f0531ff62b77a8eb11f8d8d91fc",
"prevRandao": "0x05476c4e61ec50a12084740e12518c238ad86a27bf85513fd2df7b1c1dda7089",
"suggestedFeeRecipient": "0x00000000219ab540356cbb839cbe05303d7705fa",
"timestamp": "0x650074d6",
"withdrawals": []
}) Nethermind.Merge.Plugin.BlockProduction.EmptyBlockProductionException: Couldn't produce empty block: Setting state for processing block failed
at Nethermind.Merge.Plugin.BlockProduction.PostMergeBlockProducer.PrepareEmptyBlock(BlockHeader parent, PayloadAttributes payloadAttributes) in /git/nethermind/src/Nethermind/Nethermind.Merge.Plugin/BlockProduction/PostMergeBlockProducer.cs:line 75
at Nethermind.Merge.Plugin.BlockProduction.PayloadPreparationService.ProduceEmptyBlock(String payloadId, BlockHeader parentHeader, PayloadAttributes payloadAttributes) in /git/nethermind/src/Nethermind/Nethermind.Merge.Plugin/BlockProduction/PayloadPreparationService.cs:line 92
at Nethermind.Merge.Plugin.BlockProduction.PayloadPreparationService.StartPreparingPayload(BlockHeader parentHeader, PayloadAttributes payloadAttributes) in /git/nethermind/src/Nethermind/Nethermind.Merge.Plugin/BlockProduction/PayloadPreparationService.cs:line 77
at Nethermind.Merge.Plugin.Handlers.ForkchoiceUpdatedHandler.Handle(ForkchoiceStateV1 forkchoiceState, PayloadAttributes payloadAttributes) in /git/nethermind/src/Nethermind/Nethermind.Merge.Plugin/Handlers/ForkchoiceUpdatedHandler.cs:line 74
at Nethermind.Merge.Plugin.EngineRpcModule.ForkchoiceUpdated(ForkchoiceStateV1 forkchoiceState, PayloadAttributes payloadAttributes, Int32 version) in /git/nethermind/src/Nethermind/Nethermind.Merge.Plugin/EngineRpcModule.Paris.cs:line 70
at Nethermind.JsonRpc.JsonRpcService.ExecuteAsync(JsonRpcRequest request, String methodName, ValueTuple`2 method, JsonRpcContext context) in /git/nethermind/src/Nethermind/Nethermind.JsonRpc/JsonRpcService.cs:line 235
at Nethermind.JsonRpc.JsonRpcService.ExecuteRequestAsync(JsonRpcRequest rpcRequest, JsonRpcContext context) in /git/nethermind/src/Nethermind/Nethermind.JsonRpc/JsonRpcService.cs:line 115
at Nethermind.JsonRpc.JsonRpcService.SendRequestAsync(JsonRpcRequest rpcRequest, JsonRpcContext context) in /git/nethermind/src/Nethermind/Nethermind.JsonRpc/JsonRpcService.cs:line 105
12 Sep 14:25:29 | Error when handling ID 1, engine_forkchoiceUpdatedV3({
"finalizedBlockHash": "0x9ac8d7e72abc43301454f076d4af573629a4fc753d4125609879f122bf784eb1",
"headBlockHash": "0xd6ace01457e5d479b783866d145864ec2afbb653a06a4a3708ed6c7b01130150",
"safeBlockHash": "0x7f6b04801ae6f0c6eb92fa15c7fa22dc2afce5c0a109ae6d7879d331375ba36a"
},{
"parentBeaconBlockRoot": "0x8dd9f4c4f89e80a699a1fb387923d54955061f0531ff62b77a8eb11f8d8d91fc",
"prevRandao": "0x05476c4e61ec50a12084740e12518c238ad86a27bf85513fd2df7b1c1dda7089",
"suggestedFeeRecipient": "0x00000000219ab540356cbb839cbe05303d7705fa",
"timestamp": "0x650074d6",
"withdrawals": []
}) | {"jsonrpc":"2.0","error":{"code":-32603,"message":"Internal error","data":"Nethermind.Merge.Plugin.BlockProduction.EmptyBlockProductionException: Couldn't produce empty block: Setting state for processing block failed\n at Nethermind.Merge.Plugin.BlockProduction.PostMergeBlockProducer.PrepareEmptyBlock(BlockHeader parent, PayloadAttributes payloadAttributes) in /git/nethermind/src/Nethermind/Nethermind.Merge.Plugin/BlockProduction/PostMergeBlockProducer.cs:line 75\n at Nethermind.Merge.Plugin.BlockProduction.PayloadPreparationService.ProduceEmptyBlock(String payloadId, BlockHeader parentHeader, PayloadAttributes payloadAttributes) in /git/nethermind/src/Nethermind/Nethermind.Merge.Plugin/BlockProduction/PayloadPreparationService.cs:line 92\n at Nethermind.Merge.Plugin.BlockProduction.PayloadPreparationService.StartPreparingPayload(BlockHeader parentHeader, PayloadAttributes payloadAttributes) in /git/nethermind/src/Nethermind/Nethermind.Merge.Plugin/BlockProduction/PayloadPreparationService.cs:line 77\n at Nethermind.Merge.Plugin.Handlers.ForkchoiceUpdatedHandler.Handle(ForkchoiceStateV1 forkchoiceState, PayloadAttributes payloadAttributes) in /git/nethermind/src/Nethermind/Nethermind.Merge.Plugin/Handlers/ForkchoiceUpdatedHandler.cs:line 74\n at Nethermind.Merge.Plugin.EngineRpcModule.ForkchoiceUpdated(ForkchoiceStateV1 forkchoiceState, PayloadAttributes payloadAttributes, Int32 version) in /git/nethermind/src/Nethermind/Nethermind.Merge.Plugin/EngineRpcModule.Paris.cs:line 70\n at Nethermind.JsonRpc.JsonRpcService.ExecuteAsync(JsonRpcRequest request, String methodName, ValueTuple`2 method, JsonRpcContext context) in /git/nethermind/src/Nethermind/Nethermind.JsonRpc/JsonRpcService.cs:line 235\n at Nethermind.JsonRpc.JsonRpcService.ExecuteRequestAsync(JsonRpcRequest rpcRequest, JsonRpcContext context) in /git/nethermind/src/Nethermind/Nethermind.JsonRpc/JsonRpcService.cs:line 115\n at Nethermind.JsonRpc.JsonRpcService.SendRequestAsync(JsonRpcRequest rpcRequest, JsonRpcContext context) in /git/nethermind/src/Nethermind/Nethermind.JsonRpc/JsonRpcService.cs:line 105"},"id":1}
12 Sep 14:25:29 | Request finished HTTP/1.1 POST http://127.0.0.1:8551/ application/json 623 - 200 - application/json 2387.1613ms
12 Sep 14:25:29 | Request starting HTTP/1.1 POST http://127.0.0.1:8551/ application/json 59
12 Sep 14:25:29 | Request finished HTTP/1.1 POST http://127.0.0.1:8551/ application/json 59 - 200 - application/json 0.0092ms
12 Sep 14:25:29 | Sealed block 94 (0x757734...38fb48), diff: 0, tx count: 123
application/json 12.9007ms
```
[Antithesis Report](https://ethereum.antithesis.com/report/EyH4_RGFpMwWVzVa8Zy-FLasIEtyG6Z0/BhRUaHrkvASJgmy_2Xzbx5wdwTXUx8aapqm3gkxvXzM.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiQmhSVWFIcmt2QVNKZ215XzJYemJ4NXdkd1RYVXg4YWFwcW0zZ2t4dlh6TS5odG1sIiwicmVwb3J0X2lkIjoiRXlINF9SR0ZwTXdXVnpWYThaeS1GTGFzSUV0eUc2WjAifX0sIm5iZiI6IjIwMjMtMDktMjBUMDk6MzY6MTFaIn3Nhsd22h_2MUFia9DC2euxqnxY-YEbGcowLdvXeuOXimNq6yBXix6RPcRVb99cuvKM1YZaj8xk-tPuXUfIjwEK)
### Geth
1. Geth (Prysm)
In this rollout geth falls out of sync around the time when prysm is restarting. Geth does not recover. There is no aparent reason why this occurs. This ends up leading to a syncing problem. If you filter by the `node-watch` service in the antithesis report you will be able to see, which head the clients are on.
1. 1410.721941 prysm-geth is still in sync at slot 114
2. 1428.720609 prysm-geth is unreachable (API) from node-watch
3. 1440.593032 last log from the prysm-geth container before it becomes unavailable.
4. 1665.748958 prysm-geth becomes available again, but returns head slot 0
5. 1758.650934 At the end of the rollout prysm is still reporting head slot 0:
- `node-watch: ('0', '0x7676cb46', ''): ['prysm-geth-0']`
shut down between antithesis time: 1440 - 1630
Interesting logs:
```l!
[ 7451] [ 1420.839147] [service_prysm-geth-0--geth] [I] WARN [09-12|14:20:14.730] Invalid NewPayload params params="{ParentHash:0x2f7c011ee605f641ebcf41257b33e263d408eebea082aed17444bad716a71251 FeeRecipient:0x00000000219ab540356cBB839Cbe05303d7705Fa StateRoot:0xe78d2c1962b7aece09c13e28837a7048aab807753a2ce6580c2de63d49062792 ReceiptsRoot:0x100f3336862d22706bbe26d67e5abf90f8f25ec5a22c4446835b6beaa6b59536 LogsBloom:[0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] Random:0x5244eaa71b45e798e8cce9eca987f79336a57aedeb9191c6b546876e77fffc51 Number:81 GasLimit:25728210 GasUsed:105000 Timestamp:1694528414 ExtraData:[] BaseFeePerGas:+75635 BlockHash:0x828b09c0c290c18e82520e0ef6df54821d8bd633093433b47c29d341614e198d Transactions:[[248 109 129 150 132 1 93 66 13 130 82 8 148 97 119 132 61 179 19 138 230 150 121 165 75 149 207 52 94 215 89 69 13 135 56 215 234 76 104 0 0 128 130 10 150 160 17 13 170 102 71 9 145 148 204 196 57 55 128 201 148 121 24 239 50 252 43 144 60 10 103 235 135 254 221 76 121 181 160 4 151 13 196 79 162 148 203 126 229 71 13 232 111 117 59 36 86 253 80 223 214 192 215 37 69 80 43 169 21 169 63] [248 109 129 151 132 1 93 66 13 130 82 8 148 104 119 4 219 7 233 2 233 168 179 117 64 49 209 104 212 110 61 88 110 135 56 215 234 76 104 0 0 128 130 10 149 160 33 246 142 206 148 16 90 159 217 235 225 189 20 20 154 206 224 177 27 26 131 221 57 22 206 98 26 208 118 9 177 95 160 115 80 246 217 53 53 138 173 194 172 134 105 34 163 219 71 25 111 172 104 54 25 42 202 49 173 133 135 115 51 85 158] [248 109 129 152 132 1 93 66 13 130 82 8 148 21 230 165 162 225 49 221 84 103 250 31 243 172 209 4 244 94 229 148 11 135 56 215 234 76 104 0 0 128 130 10 149 160 195 192 181 196 191 178 219 25 49 132 34 5 188 27 200 105 151 152 151 99 117 206 146 120 188 253 183 215 160 163 136 116 160 75 179 159 125 112 43 248 201 225 27 63 230 7 61 158 162 117 59 7 79 241 167 190 167 210 19 71 186 166 141 118 19] [248 109 129 153 132 1 93 66 13 130 82 8 148 128 196 199 18 89 103 19 154 202 169 49 238 152 74 157 180 16 14 15 59 135 56 215 234 76 104 0 0 128 130 10 149 160 50 214 32 94 25 32 2 177 101 143 80 118 36 152 178 50 17 243 134 65 50 147 158 141 186 65 252 202 221 46 233 154 160 91 188 248 6 123 126 61 15 227 37 85 201 172 163 12 36 242 41 68 46 25 208 243 161 68 109 176 117 120 54 52 22] [248 109 129 154 132 1 93 66 13 130 82 8 148 208 138 99 36 79 205 40 176 174 197 7 80 82 205 206 49 186 4 254 173 135 56 215 234 76 104 0 0 128 130 10 149 160 162 88 101 17 112 237 185 155 127 65 212 72 78 241 28 7 164 147 69 124 86 208 242 51 7 165 119 29 21 100 153 37 160 89 44 139 245 210 246 186 80 124 198 157 148 90 207 56 101 99 212 85 235 239 127 7 119 87 13 123 4 154 214 133 229]] Withdrawals:[] BlobGasUsed:0xc001cef5d0 ExcessBlobGas:0xc001cef5d8}" error="blockhash mismatch, want 828b09c0c290c18e82520e0ef6df54821d8bd633093433b47c29d341614e198d, got a31f296ce19d8bc5752af758e779f663b254c762c2814c39d8ed23bc95c7cacf"
time="2023-09-12 14:24:50" level=warning msg="Execution client is not syncing" prefix=powchain
time="2023-09-12 14:25:58" level=debug msg="Can not trigger event" epoch=14 error="invalid range for non-skipped slot" event=tick highestExpectedSlot=96 noRequiredPeersErrRetries=0 prefix=initial-sync start=448
```
[Antithesis Report](https://ethereum.antithesis.com/report/SsekPJZjfLHEpVRdJctabjKBRsi59rbC/NFtQfEQbs1_qnswIsE-GCaPj9LAzcDEwJWWdKlTzz3g.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTIxVDE0OjQ1OjUzWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJORnRRZkVRYnMxX3Fuc3dJc0UtR0NhUGo5TEF6Y0RFd0pXV2RLbFR6ejNnLmh0bWwiLCJyZXBvcnRfaWQiOiJTc2VrUEpaamZMSEVwVlJkSmN0YWJqS0JSc2k1OXJiQyJ9fX1dNd1TlKK2D7fg1bsG7qLe1lxx2l7k7bkx_253zBLEPOPInFQWWUcC2pSEX9l-ip4f4rle2u8Evy43qHRNQhMD)
Sign in with Wallet
Connect another wallet