# Antithesis dencun-devnet-8 report
This is a summary of the kinds of errors that we found running experiments for devnet-8. Some of the errors might have already been fixed in later releases, please ignore these. If you are interested in a report of one or more of these errors please let use know. We are happy to provide more context.
The repository and branch of the clients used:
```dockerfile
# Consensus Clients
ARG LIGHTHOUSE_REPO="https://github.com/sigp/lighthouse"
ARG LIGHTHOUSE_BRANCH="ce824e00a3566e7e94e77600d97aab5be3b9a99c"
ARG PRYSM_REPO="https://github.com/prysmaticlabs/prysm.git"
ARG PRYSM_BRANCH="c728e0923fc451c151c7ffe373fad3b57fab018e"
ARG NIMBUS_ETH2_REPO="https://github.com/status-im/nimbus-eth2.git"
ARG NIMBUS_ETH2_BRANCH="8993d57"
ARG TEKU_REPO="https://github.com/ConsenSys/teku.git"
ARG TEKU_BRANCH="b49165a76c6aaa56fad01eaa2a80c18b4084c81f"
# Execution Clients
ARG BESU_REPO="https://github.com/hyperledger/besu.git"
ARG BESU_BRANCH="c16a3274b30a96678388dd616006d5919085200e"
ARG GETH_REPO="https://github.com/lightclient/go-ethereum.git"
ARG GETH_BRANCH="79f3c2d9c96bd82320d44851074da76587e41887"
ARG NETHERMIND_REPO="https://github.com/NethermindEth/nethermind.git"
ARG NETHERMIND_BRANCH="84d41bd174fd6c4f8350c5c6f0d4b746acee1c51"
```
The client pairs we launched each with 10 validators attached:
* 10.0.20.10 prysm-geth
* 10.0.20.20 teku-geth
* 10.0.20.50 nimbus-geth
* 10.0.20.80 lighthouse-besu
* 10.0.20.110 prysm-nethermind
* 10.0.20.130 lighthouse-nethermind
* 10.0.20.150 nimbus-nethermind
## Consensus
### Lighthouse
1. *Lighthouse-besu validator* - Crit Error whilst producing block
We wanted to highlight this error message and bring awareness to the `UNHANDLED_ERROR` part. This might not be an issue as the error is caught.
```l!
Sep 12 14:54:41.788 CRIT Error whilst producing block info:
proposal did not use a builder, block_slot: Slot(288),
message: Recoverable("Some endpoints failed, num_failed: 1 http://127.0.0.1:5000/ => RequestFailed(Recoverable(\"Error from beacon node when producing block: ServerMessage(ErrorMessage { code: 500, message: \\\"UNHANDLED_ERROR: GetPayloadFailed(EngineError(Api { error: HttpClient(url: http://127.0.0.1:8551/, kind: timeout, detail: operation timed out) }))\\\", stacktraces: [] })\"))"), service: block
```
[Antithesis Report](https://ethereum.antithesis.com/report/w5_8yp9HjjISsF1lHqwi4P_1aVhi97S8/kSiQRWlaNIeNp_9gYqIvcCHrL95aPAv-FjaOQP4hg4Y.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0Ijoia1NpUVJXbGFOSWVOcF85Z1lxSXZjQ0hyTDk1YVBBdi1GamFPUVA0aGc0WS5odG1sIiwicmVwb3J0X2lkIjoidzVfOHlwOUhqaklTc0YxbEhxd2k0UF8xYVZoaTk3UzgifX0sIm5iZiI6IjIwMjMtMDktMjBUMDg6MzA6MjRaIn1xQ_72blMLO3Sf_HNfXRrSjxkBbDnMF1v4ivn0ulumPBDGQqy6xdrUMC_sxeRFaORlkHa-qmEuSPYfONEj7bQC)
2. *Lighthouse-besu validator* - Similar to the error above although here we have a Error message where the node broadcasts an object to the network without being fully importing it to the database. What is interesting is that the ErrorMessage also has a code of 202. HTTP status codes in the range of 200 normally indicate a form of success.
```l!
Sep 12 14:27:14.877 CRIT Error whilst producing block info: proposal did not use a builder, block_slot: Slot(151), message: Irrecoverable("Some endpoints failed, num_failed: 2 http://127.0.0.1:5000/ => Unavailable(Offline), http://127.0.0.1:5000/ => RequestFailed(Irrecoverable(\"Error from beacon node when publishing block: ServerMessage(ErrorMessage { code: 202, message: \\\"ACCEPTED: the object was broadcast to the network without being fully imported to the local database: ExecutionPayloadError(RejectedByExecutionEngine { status: Invalid { latest_valid_hash: None, validation_error: Some(\\\\\\\"Computed block hash 0xaefd107931c2f7678a27b1a7e15b22d15fa6762e0b586ef779f65b50c62b6780 does not match block hash parameter 0xc245f967525eda628926b08c6283843e3c26f7539705bb10cb8f9e83115fe887\\\\\\\") } })\\\", stacktraces: [] })\"))"), service: block
```
[Antithesis Report](https://ethereum.antithesis.com/report/Xo9xUSjKnBhoCTzvxjjQdG_c-_RSnTb6/ysmx51mXEuI6KAuDaXjibaPf84Y6Lt7nvgZ4T4wolwM.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTIwVDA4OjMzOjI0WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJ5c214NTFtWEV1STZLQXVEYVhqaWJhUGY4NFk2THQ3bnZnWjRUNHdvbHdNLmh0bWwiLCJyZXBvcnRfaWQiOiJYbzl4VVNqS25CaG9DVHp2eGpqUWRHX2MtX1JTblRiNiJ9fX2vkVpKVAl5akya_70qrHUuhOceyTM8k9GUygcogMu82sG3zHl9F-iz1ejfiRuaUzoCbfi5Z_fwvb6Cvq9-uUkG)
3. *Lighthouse-besu lighthouse*. I think this API endpoint is deprecated. I think it is being used by ethdo on our end. However, I don't think it should return an Internal Server Error.
```l!
Sep 12 14:54:41.784 WARN Error processing HTTP API request method: GET, path: /eth/v2/validator/blocks/288, status: 500 Internal Server Error, elapsed: 3.324625075s
```
[Antithesis Report](https://ethereum.antithesis.com/report/u9l3JE7gnX4WA8tmRD1Wx8HlQBZ8pj9j/OAqhwtJQLDEsGLS5-ARJNE7ndiFJBgHto22bzSoubww.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiT0FxaHd0SlFMREVzR0xTNS1BUkpORTduZGlGSkJnSHRvMjJielNvdWJ3dy5odG1sIiwicmVwb3J0X2lkIjoidTlsM0pFN2duWDRXQTh0bVJEMVd4OEhsUUJaOHBqOWoifX0sIm5iZiI6IjIwMjMtMDktMTNUMTQ6NDY6NDJaIn1Fop7Kk6wRKG0v9fz-y3IldcrXS1NNuCJovPiZzJns2MFQ80-PSA1TRLo92gdgxR2jAunLiH_iSRwjQRSSI6EI)
4. *Lighthouse-besu lighthouse*
```l!
Sep 12 14:39:14.798 WARN Error processing HTTP API request method: POST, path: /eth/v1/beacon/pool/sync_committees, status: 400 Bad Request, elapsed: 565.35µs
```
[Antithesis Report](https://ethereum.antithesis.com/report/NPjffuVnxo9t20-H7LZHs1jfWQ8IvV5b/yNJmyky-alSwnk_UZ1HkfqzIun9PrZ8YHLioBVQJtG4.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTEzVDE0OjQwOjU0WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJ5TkpteWt5LWFsU3dua19VWjFIa2Zxekl1bjlQclo4WUhMaW9CVlFKdEc0Lmh0bWwiLCJyZXBvcnRfaWQiOiJOUGpmZnVWbnhvOXQyMC1IN0xaSHMxamZXUThJdlY1YiJ9fX3TutsHmDntYYfdOfp2bbIH1dFw2xmkZnJ5XqN5_BDAunP8lsHucDwZ42c3BH0JGP7Vm_SywI3MoCovaWB_UcgP)
[Antithesis Report](https://ethereum.antithesis.com/report/6g2uIvEaFrpWVsdFQANZFLSD0Fe_Q1UR/8YqWqi4zajmTTHriJNFKUlU9gWPXmeoaeWMbaOjFwZU.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTEzVDE0OjI5OjUzWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiI4WXFXcWk0emFqbVRUSHJpSk5GS1VsVTlnV1BYbWVvYWVXTWJhT2pGd1pVLmh0bWwiLCJyZXBvcnRfaWQiOiI2ZzJ1SXZFYUZycFdWc2RGUUFOWkZMU0QwRmVfUTFVUiJ9fX0M0FGfVQTUKZF19OF5QX9jPD6eGxxn__LQoqKqY_bpf0kxjkl86LUH_YFx11FROqzAd1BVH_AsacnpcFilylgA)
5. *Lighthouse-besu lighthouse*
```l!
Sep 12 14:21:28.960 ERRO Failure verifying signed contribution and proof, contribution_slot: 121, subcommittee_index: 2, aggregator_index: 37, request_index: 0, error: PastSlot { message_slot: Slot(121), earliest_permissible_slot: Slot(122) }
```
[Antithesis Report](https://ethereum.antithesis.com/report/T1rbbXu8SZtI63pyeRoY-TyFAJ18Ta7H/Y-VDuYvL6tcx_YVNAOAozAFwvu8Rc0ZQpVp2Oxmfbs0.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiWS1WRHVZdkw2dGN4X1lWTkFPQW96QUZ3dnU4UmMwWlFwVnAyT3htZmJzMC5odG1sIiwicmVwb3J0X2lkIjoiVDFyYmJYdThTWnRJNjNweWVSb1ktVHlGQUoxOFRhN0gifX0sIm5iZiI6IjIwMjMtMDktMTNUMTU6Mzg6MDBaIn1QKyZAVmuTwqvNWm05vvVBahnRZb6ixlHA8Q8_ItI_3t8PJ-8LbKfzJ56TZFYGth0czSUpubIwiX5mbQygvuAB)
[Antithesis Report \(bug\)](https://ethereum.antithesis.com/report/FdOdUW8Y66r6tYB9OKYro6SJ7TgELARc/END4eKNZM9c_Ym1bcHqrgC-WAp8XcpvO4pNF4YvPieU.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTEzVDE1OjM1OjM0WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJFTkQ0ZUtOWk05Y19ZbTFiY0hxcmdDLVdBcDhYY3B2TzRwTkY0WXZQaWVVLmh0bWwiLCJyZXBvcnRfaWQiOiJGZE9kVVc4WTY2cjZ0WUI5T0tZcm82U0o3VGdFTEFSYyJ9fX2NaAqpmMnHK463cXT0u5Fl2YiD-MwcnPoozrcUGU0TX7kIzuNp9qgX7eo-J-o0-25Ufs4RAiKp6TIq4tRbEIgE)
6. *Lighthouse-besu lighthouse*
```l!
Sep 12 14:23:50.522 ERRO Failure verifying sync committee signature for gossip, validator_index: 37, slot: 133, request_index: 7, error: PastSlot { message_slot: Slot(133), earliest_permissible_slot: Slot(134) }
```
[Antithesis Report](https://ethereum.antithesis.com/report/AscCI1fbNHM2vUN8jrTwPwt4fOa3BBN2/PxQj8ffq8rwrP-6PaXc_jxYo8_t_fvL4x3gdqjbgd24.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiUHhRajhmZnE4cndyUC02UGFYY19qeFlvOF90X2Z2TDR4M2dkcWpiZ2QyNC5odG1sIiwicmVwb3J0X2lkIjoiQXNjQ0kxZmJOSE0ydlVOOGpyVHdQd3Q0Zk9hM0JCTjIifX0sIm5iZiI6IjIwMjMtMDktMTNUMTU6NDI6MDRaIn020FonubTkG8mUfwcm1iRGbJ6kA6nfC_buG3__iJeN114e1O8P8dQYLDm3_aNk1FC2Osb7ei5DRWi7yQB_9kYN)
7. *Lighthouse-besu lighthouse*
```l!
Sep 12 14:22:26.400 WARN Invalid execution payload method: new_payload, slot: 127, proposer_index: 37, graffiti: lighthouse-besu-0, root: 0xf78098f15106e4a8c524630e21c77b03d153f33b207f82c2acd8c2d5c032d3ff, execution_block_hash: 0xa3511f0718a4707a0381c8138dfa123b3fbecd2d3501592407eb0dbda54b6a12, latest_valid_hash: None, validation_error: Some("Computed block hash 0xb64cb7247b495cbb8ab6c9eba4045c9306f581b9d1bd1e4542cd7d5c84d4cfc7 does not match block hash parameter 0xa3511f0718a4707a0381c8138dfa123b3fbecd2d3501592407eb0dbda54b6a12"), service: beacon
```
[Antithesis Report](https://ethereum.antithesis.com/report/kzfk3hcwNYybIAGB8VuQT1M1mk07vsfd/mQ6hvsMVQj3kUjfahD0W0Z9Wxux_DbtqYGUQsBaeyOI.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTEzVDE1OjQ2OjEyWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJtUTZodnNNVlFqM2tVamZhaEQwVzBaOVd4dXhfRGJ0cVlHVVFzQmFleU9JLmh0bWwiLCJyZXBvcnRfaWQiOiJremZrM2hjd05ZeWJJQUdCOFZ1UVQxTTFtazA3dnNmZCJ9fX0CYU43h3oIwpUvh77oNZUtMWl0VYBIjtJ-olnN5HcqnNinq929n-s6-XxRUVqS8GsF7daU9-43VkqPEcuowrEC)
8. *Lighthouse-geth lighthouse*
```l!
Aug 24 09:12:49.858 WARN Invalid execution payload method: new_payload, slot: 279, proposer_index: 55, graffiti: prysm-nethermind-0, root: 0x9e15c29710f751e4708afdbd16b509c1365d9e552b770a99992bb48b14314748, execution_block_hash: 0x79bb79ee29888799e9d525d0932ad0556b72cfb55e3e2ecbc051d0f05f234bcc, latest_valid_hash: Some(0xc371ef989d464f2a47853ea3f7cbef8718d3cefa3d9ac3f1adf7421fcda4a968), validation_error: Some("invalid merkle root (remote: 81d375c5998a60e18fb0498947499156b3b11df92995001438b3095c56cf106b local: 5627f996eae216cc78c05220bb8fe96b31bd58376df05d8ced66f23f693f4a90) dberr: %!w(<nil>)"), service: beacon
```
[Antithesis Report](https://ethereum.antithesis.com/report/cm8-_iuCBEc25ziCyFOiB35io4VIYVoF/ypnCDrv2pqhC2GnwQ91CfjkGF3udZOwjnDgGmMxLbrc.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTA0VDA5OjA1OjQ1WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJ5cG5DRHJ2MnBxaEMyR253UTkxQ2Zqa0dGM3VkWk93am5EZ0dtTXhMYnJjLmh0bWwiLCJyZXBvcnRfaWQiOiJjbTgtX2l1Q0JFYzI1emlDeUZPaUIzNWlvNFZJWVZvRiJ9fX2twrPyqd3eNScgnYlAS66Bchpw1saedZtad2OwMQzDqGf_FJberE60YuWBKIIvtn_-T0S2jSXk9GkeVSiR3fcN)
### Prysm
1. *Prysm-geth prysm* - Why is there an undfined execution engine error if the request times out?
```l!
time="2023-09-12 14:44:10" level=error msg="Could not handle p2p pubsub" error="could not notify the engine of the new payload: timeout from http.Client: received an undefined execution engine error" prefix=sync topic="/eth2/1af88f24/beacon_block/ssz_snappy"
```
[Antithesis Report](https://ethereum.antithesis.com/report/29ZYxWxizmVXb8pQYNWFjV5KX6_lu6_H/iQgQi8nWnbkT9QsUJfifdB6nCkJZWOXStRxZqyruWPM.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTIwVDA3OjQzOjE4WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJpUWdRaThuV25ia1Q5UXNVSmZpZmRCNm5Da0paV09YU3RSeFpxeXJ1V1BNLmh0bWwiLCJyZXBvcnRfaWQiOiIyOVpZeFd4aXptVlhiOHBRWU5XRmpWNUtYNl9sdTZfSCJ9fX3KdFRc-R0OUzVBbCXPr_-vZ69sp42uNOO1EGLldv-30vhlrkXR0_mpiJsnep1OiVUcB_u2YdynOscDksm0XhEJ)
2. *Prysm-geth prysm* - We've noticed that prysm does not seem to recognize 0x00 as the initial finalized root. Why is that?
```l!
time="2023-09-12 14:09:50" level=error msg="could not process new slot" error="0x0000000000000000000000000000000000000000000000000000000000000000: unknown finalized root" prefix=blockchain
```
[Antithesis Report](https://ethereum.antithesis.com/report/wqMndCz1wssCu8E6k0KjA598u3FTdcn6/BA_QLdgv5HAMhENTkMHnKFXHq36QmfJ0TsrsGvBHIVk.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiQkFfUUxkZ3Y1SEFNaEVOVGtNSG5LRlhIcTM2UW1mSjBUc3JzR3ZCSElWay5odG1sIiwicmVwb3J0X2lkIjoid3FNbmRDejF3c3NDdThFNmswS2pBNTk4dTNGVGRjbjYifX0sIm5iZiI6IjIwMjMtMDktMTlUMDg6MDQ6MzRaIn07RzjgC4xdKV_vlcSb6jg1eDlR0-IgW2liuhIYsDES6wzsJjrjK_Tbs6qI196NzLkpiWdRwTgmNgiD5W4SDyYD)
3. *Prysm-geth prysm*
```l!
time="2023-09-12 14:47:51" level=debug msg="Could not process block" error="could not notify the engine of the new payload: received an INVALID payload from execution engine" prefix=sync slot=254
```
[Antithesis Report](https://ethereum.antithesis.com/report/WoRaHPqgRBa9KY3bSicffti0hGi4bco8/R0QsvyxUS7lC8JpzQJMjlIcu1BCHqouafnw2G27LFcQ.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTIwVDA3OjUyOjQwWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJSMFFzdnl4VVM3bEM4SnB6UUpNamxJY3UxQkNIcW91YWZudzJHMjdMRmNRLmh0bWwiLCJyZXBvcnRfaWQiOiJXb1JhSFBxZ1JCYTlLWTNiU2ljZmZ0aTBoR2k0YmNvOCJ9fX2Oe-3UmvhUTQG5hIBiZH_H1elLnndRadZKFIpoKcsZvQs52BxdV8Sqp10gMGb8yYI80ImNCwng-NUR6HTG7ogE)
4. *Prysm-geth prysm*
```l!
time="2023-09-12 14:43:31" level=error msg="Received nil payload ID on VALID engine response" blockHash=0x7b64ff651be72915a68089132c4e6c9746244bd55ef90a2573b6a09e50422d59 prefix=blockchain slot=231
``````
5. *Prysm-besu prysm* - I got this locally while killing nodes and bringing them back up
```l!
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x22f2d9b]
goroutine 5531 [running]:
github.com/prysmaticlabs/prysm/v4/proto/engine/v1.(*ExecutionBlock).UnmarshalJSON(0xc002b3b080, {0xc006378d00, 0x835, 0x900})
proto/engine/v1/json_marshal_unmarshal.go:118 +0x6fb
encoding/json.(*decodeState).object(0xc001efd680, {0x59c1c0?, 0xc002b3b080?, 0xc005d48020?})
GOROOT/src/encoding/json/decode.go:613 +0x6ef
encoding/json.(*decodeState).value(0xc001efd680, {0x59c1c0?, 0xc002b3b080?, 0x7fdb47d97fe8?})
GOROOT/src/encoding/json/decode.go:374 +0x45
encoding/json.(*decodeState).unmarshal(0xc001efd680, {0x59c1c0?, 0xc002b3b080?})
GOROOT/src/encoding/json/decode.go:181 +0x168
encoding/json.Unmarshal({0xc006378d00, 0x835, 0x900}, {0x59c1c0, 0xc002b3b080})
GOROOT/src/encoding/json/decode.go:108 +0x125
github.com/ethereum/go-ethereum/rpc.(*Client).CallContext(0xc003e595f0, {0xe06418, 0xc00233cbd0}, {0x59c1c0, 0xc002b3b080}, {0x8d8d9b?, 0x14?}, {0xc005acec00, 0x2, 0x2})
external/com_github_ethereum_go_ethereum/rpc/client.go:374 +0x285
github.com/prysmaticlabs/prysm/v4/beacon-chain/execution.(*Service).LatestExecutionBlock(0xc000a0dd40, {0xe06370?, 0xc00012dbd0?})
beacon-chain/execution/engine_client.go:457 +0x168
github.com/prysmaticlabs/prysm/v4/beacon-chain/execution.(*Service).logTtdStatus(0x64f1b1bf?, {0xe06370?, 0xc00012dbd0?}, 0x0?)
beacon-chain/execution/check_transition_config.go:141 +0x45
github.com/prysmaticlabs/prysm/v4/beacon-chain/execution.(*Service).checkTransitionConfiguration(0xc000a0dd40, {0xe06370, 0xc00012dbd0}, 0xc00425b6e0)
beacon-chain/execution/check_transition_config.go:106 +0xc2d
created by github.com/prysmaticlabs/prysm/v4/beacon-chain/execution.(*Service).Start
beacon-chain/execution/service.go:244 +0x3ac
```
### Nimbus
1. *Nimbus-geth nimbus*
```l!
WRN 2023-09-12 14:22:52.928+00:00 Unable to get execution payload. Skipping block proposal topics="beacval" slot=129 validator_index=22
```
[Antithesis Report](https://ethereum.antithesis.com/report/TFXbqALdcRroV-kFNanX7kNi3n1H62TB/e9RHguJLEAMVqfZQ0i0jfnTPoNW7qlksvqWo-2gKZRc.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiZTlSSGd1SkxFQU1WcWZaUTBpMGpmblRQb05XN3Fsa3N2cVdvLTJnS1pSYy5odG1sIiwicmVwb3J0X2lkIjoiVEZYYnFBTGRjUnJvVi1rRk5hblg3a05pM24xSDYyVEIifX0sIm5iZiI6IjIwMjMtMDktMjBUMDg6MTE6NDRaIn1zMJDJo_L5qnmVqdsTLaL2rn8_oAv2YZqu_Pr9INuIXkcsPrJsdEPjwWQvSnnHsmkJi3GCE_QDJZBTbKjtsG4G)
2. *Nimbus-geth nimbus*
```l!
DBG 2023-09-12 14:03:52.995+00:00 Received error response from handler topics="beacnde" status=500 meth=GET peer=10.0.20.203:43564 uri=/eth/v1/beacon/states/32/sync_committees error_status="500 Internal Server Error" error_content_type=application/json error_message="{\"code\":500,\"message\":\"Internal server error\",\"stacktraces\":[\"Could not get validator indices\"]}
```
3. *Nimbus-geth nimbus*
```l!
DBG 2023-09-12 14:00:19.463+00:00 Error response from peer topics="sync" responseCode=InvalidRequest errMsg="invalid finalized root"
```
4. *Nimbus-geth nimbus*
```l!
DBG 2023-09-12 14:20:05.520+00:00 Error processing request topics="networking" peer=16U*wRTVBW responseCode=InvalidRequest errMsg="Failed to decompress snappy payload"
```
[Antithesis Report](https://ethereum.antithesis.com/report/dxi-dP05i_4o_gmm66u-5LCLP8UzyWnL/PZflf8JEyrnWtdCs0kzggX7gpMYzvFPDOXAHx4TVCzg.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTIwVDA4OjA0OjM4WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJQWmZsZjhKRXlybld0ZENzMGt6Z2dYN2dwTVl6dkZQRE9YQUh4NFRWQ3pnLmh0bWwiLCJyZXBvcnRfaWQiOiJkeGktZFAwNWlfNG9fZ21tNjZ1LTVMQ0xQOFV6eVduTCJ9fX1acknEZPHF8k_Aux9mOzVqU0sOoMkWgCI1dnKZJMjbRUuMCqjqtQlwrBPxud4LJcwGSzlmQ37PW1zqG3g2rmMA)
5. *Nimbus-geth nimbus*
```l!
ERR 2023-09-12 14:19:42.315+00:00 Failed to obtain execution payload from EL topics="beacval" executionHeadBlock=96cb72a6
```
[Antithesis Report](https://ethereum.antithesis.com/report/K7KWGoq5s5ssG8AMtZbTsgtEqAvB_-u2/R6lp3yPFbX-iW_jBSPRl40NxoXXe0VLzXhpziPUT9Nk.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTIwVDA4OjA2OjQ5WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJSNmxwM3lQRmJYLWlXX2pCU1BSbDQwTnhvWFhlMFZMelhocHppUFVUOU5rLmh0bWwiLCJyZXBvcnRfaWQiOiJLN0tXR29xNXM1c3NHOEFNdFpiVHNndEVxQXZCXy11MiJ9fX18V1tRSjPXQRYA-f0-IFaecQV6ucmH5II5D-0vazKpetEoYVQDiig92LvRREqyNWzn1JZFGC5me1mFpfIJuhYD)
6. *Nimbus-geth nimbus*
```l!
WRN 2023-09-12 14:37:16.932+00:00 Failed to exchange configuration with the configured EL end-points topics="elmon"
```
[Antithesis Report](https://ethereum.antithesis.com/report/N6x7bYMqGAji5DslH00nTQMwaWaZd2VB/vS14LoYbyMXc-ydetO11iw6YOtpPiHfvjTSugEMQMvo.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTIwVDA4OjA5OjAzWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJ2UzE0TG9ZYnlNWGMteWRldE8xMWl3NllPdHBQaUhmdmpUU3VnRU1RTXZvLmh0bWwiLCJyZXBvcnRfaWQiOiJONng3YllNcUdBamk1RHNsSDAwblRRTXdhV2FaZDJWQiJ9fX2XC8xG_bpH_gc3_tjedWPoyfjRS3VOJRn5sJ44K28Cri-iImS1rSS9DUr-v3OeGGTIt62zJwWzySYOm7Tr0wMB)
7. *Nimbus-geth nimbus*
```l!
WRN 2023-09-12 14:22:52.928+00:00 Unable to get execution payload. Skipping block proposal topics="beacval" slot=129 validator_index=22
```
[Antithesis Report](https://ethereum.antithesis.com/report/TFXbqALdcRroV-kFNanX7kNi3n1H62TB/e9RHguJLEAMVqfZQ0i0jfnTPoNW7qlksvqWo-2gKZRc.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiZTlSSGd1SkxFQU1WcWZaUTBpMGpmblRQb05XN3Fsa3N2cVdvLTJnS1pSYy5odG1sIiwicmVwb3J0X2lkIjoiVEZYYnFBTGRjUnJvVi1rRk5hblg3a05pM24xSDYyVEIifX0sIm5iZiI6IjIwMjMtMDktMjBUMDg6MTE6NDRaIn1zMJDJo_L5qnmVqdsTLaL2rn8_oAv2YZqu_Pr9INuIXkcsPrJsdEPjwWQvSnnHsmkJi3GCE_QDJZBTbKjtsG4G)
## Execution
### Geth
1. *Nimbus-geth geth* - We are seeing block hash missmatches where there are blobs involved in the block.
```l!
1 ... 8]] Withdrawals:[] BlobGasUsed:0xc0027f2b20 ExcessBlobGas:0xc0027f2b28}" error="blockhash mismatch, want 35fa8a19c1689e0a1d6676cbfc93cbc3eda3aec78fd116abf917f265f1d22101, got e391f2b6bb64b9dfbd5d444b22ff8e9919e45f154c2de53671c7927bafa47ca6"
```
[Antithesis Report](https://ethereum.antithesis.com/report/eb5TM3Tb0has4PVxmxGSH2AxqAzSYlXv/AxodCt5WgFkPUdDwMoVgwYZ-tSExrXkOzP4RbRmAf7Y.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiQXhvZEN0NVdnRmtQVWREd01vVmd3WVotdFNFeHJYa096UDRSYlJtQWY3WS5odG1sIiwicmVwb3J0X2lkIjoiZWI1VE0zVGIwaGFzNFBWeG14R1NIMkF4cUF6U1lsWHYifX0sIm5iZiI6IjIwMjMtMDktMjBUMDg6Mjc6MzlaIn1kQImrMOrQVydbW4bpP-kPI8UWSD8jM2MyqfQ3gkIHHECT1iT117hU4RG95AxGAJQQHYN5cFXFYpKkYemVbMIG)
### Nethermind
1. *Lighthouse-nethermind nethermind*
```l!
12 Sep 14:44:55 | Error during method execution, request: ID 261, eth_createAccessList({
```
[Antithesis Report](https://ethereum.antithesis.com/report/Flwys4Nw94plbS9-j3cmmGfIyFAlDsr4/BYLrNy6wW6MIe9JWzGy8SKTE-vRP5KWaaBPjcjnOwQ0.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiQllMck55NndXNk1JZTlKV3pHeThTS1RFLXZSUDVLV2FhQlBqY2puT3dRMC5odG1sIiwicmVwb3J0X2lkIjoiRmx3eXM0Tnc5NHBsYlM5LWozY21tR2ZJeUZBbERzcjQifX0sIm5iZiI6IjIwMjMtMDktMjBUMDg6MjQ6MjlaIn3RdYnwcjeKkO9g5RbiR8H4ts9jIF-3ttmdCZp_6ZvXM_iTZO9FwYWR6GWUhpRU4HsvVqaAkZG5mvOszToLXKEH)
2. *Lighthouse-nethermind nethermind*
```l!
}) | {"jsonrpc":"2.0","error":{"code":-32603,"message":"Internal error","data":"System.NullReferenceException: Object reference not set to an instance of an object.\n at Nethermind.Evm.VirtualMachine`1.ExecuteCall[TTracingInstructions](EvmState vmState, Byte[] previousCallResult, ZeroPaddedSpan previousCallOutput, UInt256& previousCallOutputDestination, IReleaseSpec spec) in /git/nethermind/src/Nethermind/Nethermind.Evm/VirtualMachine.cs:line 718\n at Nethermind.Evm.VirtualMachine`1.Run[TTracingActions](EvmState state, IWorldState worldState, ITxTracer txTracer) in /git/nethermind/src/Nethermind/Nethermind.Evm/VirtualMachine.cs:line 241\n at Nethermind.Evm.VirtualMachine.Run[TTracingActions](EvmState state, IWorldState worldState, ITxTracer txTracer) in /git/nethermind/src/Nethermind/Nethermind.Evm/VirtualMachine.cs:line 64\n at Nethermind.Evm.TransactionProcessing.TransactionProcessor.ExecuteEVMCall(Transaction tx, BlockHeader header, IReleaseSpec spec, ITxTracer tracer, ExecutionOptions opts, Int64& gasAvailable, ExecutionEnvironment& env, TransactionSubstate& substate, Int64& spentGas, Byte& statusCode) in /git/nethermind/src/Nethermind/Nethermind.Evm/TransactionProcessing/TransactionProcessor.cs:line 586\n at Nethermind.Evm.TransactionProcessing.TransactionProcessor.Execute(Transaction tx, BlockHeader header, ITxTracer tracer, ExecutionOptions opts) in /git/nethermind/src/Nethermind/Nethermind.Evm/TransactionProcessing/TransactionProcessor.cs:line 100\n at Nethermind.Evm.TransactionProcessing.TransactionProcessor.CallAndRestore(Transaction transaction, BlockHeader block, ITxTracer txTracer) in /git/nethermind/src/Nethermind/Nethermind.Evm/TransactionProcessing/TransactionProcessor.cs:line 77\n at Nethermind.Evm.TransactionProcessing.ReadOnlyTransactionProcessor.CallAndRestore(Transaction transaction, BlockHeader block, ITxTracer txTracer) in /git/nethermind/src/Nethermind/Nethermind.Evm/TransactionProcessing/ReadOnlyTransactionProcessor.cs:line 30\n at Nethermind.Facade.BlockchainBridge.CallAndRestore(BlockHeader blockHeader, Transaction transaction, Boolean treatBlockHeaderAsParentBlock, ITxTracer tracer) in /git/nethermind/src/Nethermind/Nethermind.Facade/BlockchainBridge.cs:line 281\n at Nethermind.Facade.BlockchainBridge.TryCallAndRestore(BlockHeader blockHeader, Transaction transaction, Boolean treatBlockHeaderAsParentBlock, ITxTracer tracer) in /git/nethermind/src/Nethermind/Nethermind.Facade/BlockchainBridge.cs:line 221\n at Nethermind.Facade.BlockchainBridge.CreateAccessList(BlockHeader header, Transaction tx, CancellationToken cancellationToken, Boolean optimize) in /git/nethermind/src/Nethermind/Nethermind.Facade/BlockchainBridge.cs:line 200\n at Nethermind.JsonRpc.Modules.Eth.EthRpcModule.CreateAccessListTxExecutor.ExecuteTx(BlockHeader header, Transaction tx, CancellationToken token) in /git/nethermind/src/Nethermind/Nethermind.JsonRpc/Modules/Eth/EthRpcModule.TransactionExecutor.cs:line 119\n at Nethermind.JsonRpc.Modules.Eth.EthRpcModule.TxExecutor`1.ExecuteTx(TransactionForRpc transactionCall, BlockParameter blockParameter) in /git/nethermind/src/Nethermind/Nethermind.JsonRpc/Modules/Eth/EthRpcModule.TransactionExecutor.cs:line 54\n at Nethermind.JsonRpc.Modules.Eth.EthRpcModule.eth_createAccessList(TransactionForRpc transactionCall, BlockParameter blockParameter, Boolean optimize) in /git/nethermind/src/Nethermind/Nethermind.JsonRpc/Modules/Eth/EthRpcModule.cs:line 356\n at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)\n at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)"},"id":421}
```
[Antithesis Report](https://ethereum.antithesis.com/report/6iyDAm2l_vvdOgxKB3mBxwiJ8llGY--G/smkDIDY_WgnKSBiiOenP1NEfGuvhd6vz_oTzhrnYcrU.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTIwVDA4OjIxOjQ4WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJzbWtESURZX1dnbktTQmlpT2VuUDFORWZHdXZoZDZ2el9vVHpocm5ZY3JVLmh0bWwiLCJyZXBvcnRfaWQiOiI2aXlEQW0ybF92dmRPZ3hLQjNtQnh3aUo4bGxHWS0tRyJ9fX23qEzCeS7YNOKBzlfzgIzyBqt-YVjOiBH8Q6dE4vP7w7CmzeUZe_2L_KeAH3HZRSOAjqIIB5KtoW0_G5NWF0oG)
### Besu
1. *Lighthouse-besu besu*
```l!
2023-09-12 14:32:02.363+00:00 | vert.x-worker-thread-0 | WARN | AbstractEngineNewPayload | Invalid new payload: number: 121, hash: 0x911adef84c69c3bc646ca3db7d5c3d3ff668ad11c15c0d18dcadfe59ebce853d, parentHash: 0xf2fe8151f5c00d70cf321e41a9be20112f3a42b6c7945cd3ade396712c7da85a, latestValidHash: null, status: INVALID, validationError: Computed block hash 0xee0259a1ecfbba906a5ed7544189322aee88f51f3f921c93d6ca962dc59188c3 does not match block hash parameter 0x911adef84c69c3bc646ca3db7d5c3d3ff668ad11c15c0d18dcadfe59ebce853d
```
[Antithesis Report](https://ethereum.antithesis.com/report/WI-teeJ5wGxOC_3Dt7o_lxFpQLf1LTue/OKtgd64He19Nl-Y0DWJxnMgNs7KBEZXZ42EBsCx51Z4.html?auth=v2.public.eyJuYmYiOiIyMDIzLTA5LTIwVDA4OjE1OjUyWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJPS3RnZDY0SGUxOU5sLVkwRFdKeG5NZ05zN0tCRVpYWjQyRUJzQ3g1MVo0Lmh0bWwiLCJyZXBvcnRfaWQiOiJXSS10ZWVKNXdHeE9DXzNEdDdvX2x4RnBRTGYxTFR1ZSJ9fX33_LW4UgPC_oYnKb-Xfslz5UKtPq4r8fnJfc8DpnGgwLnwAz6TXDzW7P0SdUgtRNR7vYB7cYCbo-RoI_1MgFkJ)
2. *Prysm-besu besu* I got this locally while killing nodes and bringing them back up.
```
"2023-09-01 09:57:06.509+00:00 | EthScheduler-Workers-1 | ERROR | Besu | Uncaught exception in thread ""EthScheduler-Workers-1""
java.lang.IllegalArgumentException: Value -1 cannot be represented as an unsigned int (it is negative or too big)
at org.apache.tuweni.bytes.Checks.checkArgument(Utils.java:36)
at org.apache.tuweni.bytes.Bytes.ofUnsignedInt(Bytes.java:413)
at org.apache.tuweni.bytes.Bytes.ofUnsignedInt(Bytes.java:400)
at org.hyperledger.besu.ethereum.rlp.RLPOutput.lambda$writeUnsignedInt$2(RLPOutput.java:222)
at org.hyperledger.besu.ethereum.rlp.RLPOutput.processZeroByte(RLPOutput.java:315)
at org.hyperledger.besu.ethereum.rlp.RLPOutput.writeUnsignedInt(RLPOutput.java:222)
at org.hyperledger.besu.ethereum.eth.encoding.TransactionAnnouncementEncoder.lambda$encodeForEth68$1(TransactionAnnouncementEncoder.java:115)
at org.hyperledger.besu.ethereum.rlp.RLPOutput.writeList(RLPOutput.java:256)
at org.hyperledger.besu.ethereum.eth.encoding.TransactionAnnouncementEncoder.encodeForEth68(TransactionAnnouncementEncoder.java:115)
at org.hyperledger.besu.ethereum.eth.encoding.TransactionAnnouncementEncoder.encodeForEth68(TransactionAnnouncementEncoder.java:89)
at org.hyperledger.besu.ethereum.eth.messages.NewPooledTransactionHashesMessage.create(NewPooledTransactionHashesMessage.java:53)
at org.hyperledger.besu.ethereum.eth.transactions.NewPooledTransactionHashesMessageSender.sendTransactionHashesToPeer(NewPooledTransactionHashesMessageSender.java:60)
at org.hyperledger.besu.ethereum.eth.transactions.TransactionBroadcaster.lambda$sendTransactionHashes$7(TransactionBroadcaster.java:183)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)"
```
Sign in with Wallet
Connect another wallet