# Antithesis Report 07.12.2023 Summary of the errors found running the initial devnet 12 images: - Failed Block Import: Failed to import proposed blocks due to various reasons, including mismatching execution payloads and illegal state modifications. - Validator Indices and Committee Information Issues: Encountered exceptions related to active validator indices, committee information derivation, and potential violations of slashing conditions probably do to a chain-reorg. - Debug Log Reports: Several debug logs revealed breaches of protocol, corrupt inputs, internal errors like null pointer exceptions, and closure of deflators, all of which can jeopardize system stability and security. ## Teku Teku (nethermind) ```l! tech.pegasys.teku.validator.client.duties.synccommittee.ChainHeadBeyondSlotException: Chain head has advanced beyond slot 65 ``` [Antithesis Report](https://ethereum.antithesis.com/report/mxWz72lwzUPmM9TUGamk50T7/i_AOFiW-8oZEHzF8YonVleSMHo4M6AGAm7HJVydcgq0.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiaV9BT0ZpVy04b1pFSHpGOFlvblZsZVNNSG80TTZBR0FtN0hKVnlkY2dxMC5odG1sIiwicmVwb3J0X2lkIjoibXhXejcybHd6VVBtTTlUVUdhbWs1MFQ3In19LCJuYmYiOiIyMDIzLTEyLTA1VDE0OjI3OjAyWiJ9Y8XO01etIq14FDfB6dw6zlSh490ClYY7FhB6QhxqcrtMnJwsZFJN-mrje98-BwJg3AwX8vE8ac-EAqSJulitDw) Teku (Geth) ```l! 2023-12-01 17:53:18.057 ERROR - Failed to import proposed block due to FAILED_EXECUTION_PAYLOAD_EXECUTION: 28223a0e03b3339f5bcc33bcd25daff50bd8cf4bc75144a8803964946f932763 (111) ``` [Antithesis Report](https://ethereum.antithesis.com/report/02IiAw49DrasIjXYgfUY7xWK/oaQ6T2dUHiWrvVL8Z7fhnvGi_r3kFi8Q7pEGAuh7EQs.html?auth=v2.public.eyJuYmYiOiIyMDIzLTEyLTA1VDE0OjM1OjE0WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJvYVE2VDJkVUhpV3J2Vkw4WjdmaG52R2lfcjNrRmk4UTdwRUdBdWg3RVFzLmh0bWwiLCJyZXBvcnRfaWQiOiIwMklpQXc0OURyYXNJalhZZ2ZVWTd4V0sifX19EmO-rb1Go_o2Sw1Ji_kN6ZywVRxczOFww27r8OAmLmB-PkLvhHwCefXmvL_nXl8F1e1y2ol7StofhB09SZt2Bw) Teku (Nethermind) ```l! Caused by: java.lang.IllegalArgumentException: Cannot get active validator indices from an epoch beyond the seed lookahead period. Requested epoch 5 from state in epoch 0 ``` [Antithesis Report](https://ethereum.antithesis.com/report/pH_90Ez7KC9D6g8boXz82pmd/zlc-bn5oo2WHj8-yEn-SPYPobLB4R2eLmgSOZd36IlE.html?auth=v2.public.eyJuYmYiOiIyMDIzLTEyLTA2VDE0OjUwOjQwWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJ6bGMtYm41b28yV0hqOC15RW4tU1BZUG9iTEI0UjJlTG1nU09aZDM2SWxFLmh0bWwiLCJyZXBvcnRfaWQiOiJwSF85MEV6N0tDOUQ2Zzhib1h6ODJwbWQifX190YX-u3Wkd1R1q6GFwSoecXwpQo05smRNGYI49mXzGYncP0MZx0d_djU08DG_dx1BY8r_vWX7nJcwdAmOUCHHAA) Teku (Geth) ```l! Caused by: tech.pegasys.teku.spec.logic.common.statetransition.exceptions.BlockProcessingException: Execution payload randao does not match state randao ``` [Antithesis Report](https://ethereum.antithesis.com/report/U278UtJNk_9vMG6-Uw2jspGG/GN5JW9Blyou3bJqlh18mWDhkDfBQ8CykFfaGaJQvhHM.html?auth=v2.public.eyJuYmYiOiIyMDIzLTEyLTA1VDE1OjI4OjQxWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJHTjVKVzlCbHlvdTNiSnFsaDE4bVdEaGtEZkJROEN5a0ZmYUdhSlF2aEhNLmh0bWwiLCJyZXBvcnRfaWQiOiJVMjc4VXRKTmtfOXZNRzYtVXcyanNwR0cifX19UmO2GGWZ7H6IdMHR5KxRgwOsN5_g9OUCLby4jqKTze0QuPnWmRQjopSin06rcXQd076e-rfhEdcMI31rCCu1CQ) Teku (Geth) ```l! Caused by: java.util.ConcurrentModificationException ``` [Antithesis Report]( https://ethereum.antithesis.com/report/QubAgSRJBwc26mSReUnSM6ST/FqJ9gAdJzEphVh04aXTrviMYOs8xBDRGMGmLxO-IXl0.html?auth=v2.public.eyJuYmYiOiIyMDIzLTEyLTA1VDE1OjI2OjE1WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJGcUo5Z0FkSnpFcGhWaDA0YVhUcnZpTVlPczh4QkRSR01HbUx4Ty1JWGwwLmh0bWwiLCJyZXBvcnRfaWQiOiJRdWJBZ1NSSkJ3YzI2bVNSZVVuU002U1QifX19lyYlFsEbLr9iXazwQaftURS6DZx-amEWRK0ztJuNUqAbBhtiJzqqEUOKlHolfE-k7nxozoWNqZul2VkS-DcUCQ ) Teku (Geth) ```l! java.lang.IllegalStateException: Cannot provide kzg commitments before The Merge ``` [Antithesis Report](https://ethereum.antithesis.com/report/zujrcWzwYksG4IYdlr6QT9md/q8jabU13hc9Qz_qY743K4nZC9GCfHETfXnVImmoW0hk.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoicThqYWJVMTNoYzlRel9xWTc0M0s0blpDOUdDZkhFVGZYblZJbW1vVzBoay5odG1sIiwicmVwb3J0X2lkIjoienVqcmNXendZa3NHNElZZGxyNlFUOW1kIn19LCJuYmYiOiIyMDIzLTEyLTA1VDE1OjIwOjQ3WiJ9ExOmesCwC7CxkBfGkwjw6t-x5vthJa3hzBLca3drw1bKM3I9EiQ1Doy2WY5hzo7TI2RX261TLUDLfTpMSevCDQ) Teku (Geth) ```l! java.lang.IllegalStateException: ExecutionPayloadResult hasn't been cached for slot 131 ``` [Antithesis Report](https://ethereum.antithesis.com/report/bqezO6KoNaakFY_lXkN2pCnI/cVeeZfXs9WlO8gmNFiuaErClv_PsYHhsvEr476alH-Q.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiY1ZlZVpmWHM5V2xPOGdtTkZpdWFFckNsdl9Qc1lIaHN2RXI0NzZhbEgtUS5odG1sIiwicmVwb3J0X2lkIjoiYnFlek82S29OYWFrRllfbFhrTjJwQ25JIn19LCJuYmYiOiIyMDIzLTEyLTA2VDEwOjQzOjE5WiJ9li6TZ7kZ-FHiBRwWVEOE6BX3hcN9oG1jEl8x05jIzXpAH1yDLW-Bo5Ck3nBmmhgEwq-9Q2ZBI1QuH8sOdXB2CQ ) Teku (Nethermind) ```l! java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: Committee information must be derived from a state no older than the previous epoch. State at slot 114 is older than cutoff slot 128 ``` [Antithesis Report](https://ethereum.antithesis.com/report/lLUVKq2QHZimXEEfaNb8Z_os/qmmvQmrh9hwqr9MyCyo1FaWRFZpAJOIDKUohBSoSLrk.html?auth=v2.public.eyJuYmYiOiIyMDIzLTEyLTA2VDEwOjQ2OjU2WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJxbW12UW1yaDlod3FyOU15Q3lvMUZhV1JGWnBBSk9JREtVb2hCU29TTHJrLmh0bWwiLCJyZXBvcnRfaWQiOiJsTFVWS3EyUUhaaW1YRUVmYU5iOFpfb3MifX19tEwVou-r3xbMWsNHNgDFY-FNlxXZ4lVdVncW679qKYOZlSOL3YCsyy8Y5G8AnakyZQfBB1fks9M-FqHcwEnXBw) Teku (Geth) ```l! tech.pegasys.teku.spec.signatures.SlashableConditionException: Refusing to sign block at slot 131 as it may violate a slashing condition ``` [Antithesis Report](https://ethereum.antithesis.com/report/N6F2w6ktFfC7Z0qfYBBT6_bY/-R-gKW_a2dnCU95dcIEAo2g8xh40FbSdU9kqKbqpZYs.html?auth=v2.public.eyJuYmYiOiIyMDIzLTEyLTA2VDEwOjUxOjI3WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiItUi1nS1dfYTJkbkNVOTVkY0lFQW8yZzh4aDQwRmJTZFU5a3FLYnFwWllzLmh0bWwiLCJyZXBvcnRfaWQiOiJONkYydzZrdEZmQzdaMHFmWUJCVDZfYlkifX19u_BFxXgc-nFpfn6JzoL402T3C6NT2Iztt20JHkZzGjEp_vpZCm5aw4Vo-brf3Xaeq0saatBjxRc7Qz2QeDV-Ag ) Teku (Geth) ```l! java.lang.NullPointerException: Deflater has been closed ``` [Antithesis Report](https://ethereum.antithesis.com/report/1qfFmna3tW0oy1lrtgqPVtaV/NnVGBG3Xys_6ci5e2TfQVztoqrhp72klBwCstyP8v1U.html?auth=v2.public.eyJuYmYiOiIyMDIzLTEyLTA2VDEyOjUyOjEzWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJOblZHQkczWHlzXzZjaTVlMlRmUVZ6dG9xcmhwNzJrbEJ3Q3N0eVA4djFVLmh0bWwiLCJyZXBvcnRfaWQiOiIxcWZGbW5hM3RXMG95MWxydGdxUFZ0YVYifX190CLeQnnI6fW-k4yX-IqXOsNoPWzo4OpuYy90hqR6Ppw6hFagqEx5XpkwMNsSxDJcuIQvwk8lVG7ew4Jg8Hy9Dg ) ## Geth Geth (Teku) ```l! DEBUG[12-01|15:44:08.842] Removing p2p peer peercount=4 id=6772f55e6ff00343 duration=593.732ms req=true err="breach of protocol" ``` [Antithesis Report](https://ethereum.antithesis.com/report/zu_0xp-LfWNduLaiznUmD5Bw/5ma5ip2lEBfs4y8oqNhAOYF7tctkkE0g0Gg5_5DFuw8.html?auth=v2.public.eyJuYmYiOiIyMDIzLTEyLTA1VDExOjMwOjExWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiI1bWE1aXAybEVCZnM0eThvcU5oQU9ZRjd0Y3Rra0UwZzBHZzVfNURGdXc4Lmh0bWwiLCJyZXBvcnRfaWQiOiJ6dV8weHAtTGZXTmR1TGFpem5VbUQ1QncifX19lmnNvCDGBunBl-7_2s5bnZzZdPQ-Ci9TES_MZvNl85w7X5L_KgbN-hCvMTzoJTFqmmtOeeGQY8Ce3DPkCnzlCg ) Geth (Lighthouse) ```l! DEBUG[12-01|15:33:46.721] Removing p2p peer peercount=2 id=45409974d97c61f0 duration=25.779ms req=false err="snappy: corrupt input" ``` [Antithesis Report](https://ethereum.antithesis.com/report/oIHmEmNLRApajomeh0AJLDSI/3vNX25bp5Cq2K7td6tzzd9cilfjoFgXkybmrKeNqH7Y.html?auth=v2.public.eyJuYmYiOiIyMDIzLTEyLTA2VDEyOjQwOjA0WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiIzdk5YMjVicDVDcTJLN3RkNnR6emQ5Y2lsZmpvRmdYa3libXJLZU5xSDdZLmh0bWwiLCJyZXBvcnRfaWQiOiJvSUhtRW1OTFJBcGFqb21laDBBSkxEU0kifX19N0uCd8USbYKVA1t3VNOZX7nvzVKqnhvGZjz3yMrLoZ7PrS4EAJqDX0nzgG5fMcNDOF3gAmQrg-02striUcsHDA ) Geth (Teku) ```l! WARN [12-01|15:42:32.126] Served engine_forkchoiceUpdatedV3 conn=127.0.0.1:59846 reqid=57 duration=1m10.511268497s err="Invalid payload attributes" errdata="{Error:invalid timestamp, parent 1701445195 given 1701445159}" ``` [Antithesis Report](https://ethereum.antithesis.com/report/TxwQSF2SvhFoJwSSNr7ShrWZ/kujgJyx9XwzZAvVIcyE2kEmKTOlb0ZDK1TyMLWTF6Q4.html?auth=v2.public.eyJuYmYiOiIyMDIzLTEyLTA2VDE0OjQ1OjE4WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJrdWpnSnl4OVh3elpBdlZJY3lFMmtFbUtUT2xiMFpESzFUeU1MV1RGNlE0Lmh0bWwiLCJyZXBvcnRfaWQiOiJUeHdRU0YyU3ZoRm9Kd1NTTnI3U2hyV1oifX1967c49j9UuP7NGDZzGut3iOKWoV7wZGDVKUv3Q58vl8U5cT-zeXx_lKlvVsOdKE-Zmd8vXcIW_zkF-yHpXHEGBA) Geth (Nimbus) ```l! ERROR[12-01|15:55:41.951] Demoting invalidated transaction hash=d3cd53..b8228d ``` [Antithesis Report](https://ethereum.antithesis.com/report/cpEAkmIsSFyJUPBiKatG6uDh/ut-ux6DdMFp4kjf_qRICvxvfvrHVyiuX0KHnrzdpr9o.html?auth=v2.public.eyJuYmYiOiIyMDIzLTEyLTA2VDEyOjQ4OjU4WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJ1dC11eDZEZE1GcDRramZfcVJJQ3Z4dmZ2ckhWeWl1WDBLSG5yemRwcjlvLmh0bWwiLCJyZXBvcnRfaWQiOiJjcEVBa21Jc1NGeUpVUEJpS2F0RzZ1RGgifX19HKOHhgkBUsT5fauLHf8b_7kjf53SQlAU7Xvc-VQTHQtrGXBlpzTAOCWzxNsrSRB4WB2Lb_Rloof34ZV221JNCw ) ## Nethermind Nethermind (Lighthouse) ```l! "}) | {\"jsonrpc\":\"2.0\",\"error\":{\"code\":-32603,\"message\":\"Internal error\",\"data\":\"Nethermind.Merge.Plugin.BlockProduction.EmptyBlockProductionException: Couldn't produce empty block: Setting state for processing block failed\\n   at Nethermind.Merge.Plugin.BlockProduction.PostMergeBlockProducer.PrepareEmptyBlock(BlockHeader parent, PayloadAttributes payloadAttributes) in /git/nethermind/src/Nethermind/Nethermind.Merge.Plugin/BlockProduction/PostMergeBlockProducer.cs:line 74\\n   at Nethermind.Merge.Plugin.BlockProduction.PayloadPreparationService.ProduceEmptyBlock(String payloadId, BlockHeader parentHeader, PayloadAttributes payloadAttributes) in /git/nethermind/src/Nethermind/Nethermind.Merge.Plugin/BlockProduction/PayloadPreparationService.cs:line 91\\n   at Nethermind.Merge.Plugin.BlockProduction.PayloadPreparationService.StartPreparingPayload(BlockHeader parentHeader, PayloadAttributes payloadAttributes) in /git/nethermind/src/Nethermind/Nethermind.Merge.Plugin/BlockProduction/PayloadPreparationService.cs:line 80\\n   at Nethermind.Merge.Plugin.Handlers.ForkchoiceUpdatedHandler.StartBuildingPayload(Block newHeadBlock, ForkchoiceStateV1 forkchoiceState, PayloadAttributes payloadAttributes) in /git/nethermind/src/Nethermind/Nethermind.Merge.Plugin/Handlers/ForkchoiceUpdatedHandler.cs:line 257\\n   at Nethermind.Merge.Plugin.Handlers.ForkchoiceUpdatedHandler.Handle(ForkchoiceStateV1 forkchoiceState, PayloadAttributes payloadAttributes, Int32 version) in /git/nethermind/src/Nethermind/Nethermind.Merge.Plugin/Handlers/ForkchoiceUpdatedHandler.cs:line 80\\n   at Nethermind.Merge.Plugin.EngineRpcModule.ForkchoiceUpdated(ForkchoiceStateV1 forkchoiceState, PayloadAttributes payloadAttributes, Int32 version) in /git/nethermind/src/Nethermind/Nethermind.Merge.Plugin/EngineRpcModule.Paris.cs:line 47\\n   at Nethermind.JsonRpc.JsonRpcService.ExecuteAsync(JsonRpcRequest request, String methodName, ValueTuple`2 method, JsonRpcContext context) in /git/nethermind/src/Nethermind/Nethermind.JsonRpc/JsonRpcService.cs:line 235\\n   at Nethermind.JsonRpc.JsonRpcService.ExecuteRequestAsync(JsonRpcRequest rpcRequest, JsonRpcContext context) in /git/nethermind/src/Nethermind/Nethermind.JsonRpc/JsonRpcService.cs:line 115\\n   at Nethermind.JsonRpc.JsonRpcService.SendRequestAsync(JsonRpcRequest rpcRequest, JsonRpcContext context) in /git/nethermind/src/Nethermind/Nethermind.JsonRpc/JsonRpcService.cs:line 105\"},\"id\":1} " ``` [Antithesis Report](https://ethereum.antithesis.com/report/HAHMOrlqotx6ns_kRTBfpSjq/yOF8CWOMCT372-LxDQKSO2xoLCpWEVSBR87A2n8KvU4.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoieU9GOENXT01DVDM3Mi1MeERRS1NPMnhvTENwV0VWU0JSODdBMm44S3ZVNC5odG1sIiwicmVwb3J0X2lkIjoiSEFITU9ybHFvdHg2bnNfa1JUQmZwU2pxIn19LCJuYmYiOiIyMDIzLTEyLTA2VDEzOjEwOjM2WiJ9BNmXT2diw-vPDqjLDyOA5CdoOpvwXVn4MPNfVIkgjwPEZdazHmbmSBONiXXwzgAEIKpXs5_2xkvHWCGV5_1_BQ ) ## Besu Besu (Lodestar) ```l! at app//org.hyperledger.besu.ethereum.p2p.discovery.internal.RecursivePeerRefreshState.onNeighboursReceived(RecursivePeerRefreshState.java:197) ``` [Antithesis Report](https://ethereum.antithesis.com/report/TnXi5shl00s-gokWPJTvPk34/VaG2DZgHEMr9NjFi_LS4wQP3jxMyA5Av82pJBzK-1TE.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiVmFHMkRaZ0hFTXI5TmpGaV9MUzR3UVAzanhNeUE1QXY4MnBKQnpLLTFURS5odG1sIiwicmVwb3J0X2lkIjoiVG5YaTVzaGwwMHMtZ29rV1BKVHZQazM0In19LCJuYmYiOiIyMDIzLTEyLTA2VDEzOjAxOjI1WiJ9zaGLMj10nuTtm9eSkN8oxyW7RME4Hqjr4_EUe-8NaJ50VYMt5vcNJjJ_X-BIgkLSqE2RI5Jz6ZIpD8TA0v4qAw) Besu (Lighthouse) ```l! 2023-12-01 15:59:32.548+00:00 | vert.x-eventloop-thread-1 | ERROR | ExecutionEngineJsonRpcMethod | failed to exec consensus method engine_forkchoiceUpdatedV3, error: null ``` [Antithesis Report](https://ethereum.antithesis.com/report/0VLU4QH5Jr53cntrJA6eP4Fl/QTHxNvoZF3-yuq-YwrTM1J5RiSV9PH1ssUOzi-ov5nw.html?auth=v2.public.eyJuYmYiOiIyMDIzLTEyLTA2VDEyOjU4OjQxWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJRVEh4TnZvWkYzLXl1cS1Zd3JUTTFKNVJpU1Y5UEgxc3NVT3ppLW92NW53Lmh0bWwiLCJyZXBvcnRfaWQiOiIwVkxVNFFINUpyNTNjbnRySkE2ZVA0RmwifX19SM7cS5SBIzIWqVafLWnJdzN2186g0oNTJQx_xJeoBHv6NLNw1x0m8dNlt1OKDGlnq7cb0GF_1VkEDHy4TlDxBQ ) Besu (Lighthouse) ```l! 2023-12-01 15:35:28.384+00:00 | vert.x-worker-thread-6 | ERROR | JsonRpcExecutorHandler | engine_exchangeCapabilities - Error streaming JSON-RPC response ``` [Antithesis Report](https://ethereum.antithesis.com/report/cla9Eaqo75NtrE-1TOYwN5Gy/_7FAeuSSNDWXvPEun3HCv6zYtKqOzUT_I_U_rOswpX8.html?auth=v2.public.eyJuYmYiOiIyMDIzLTEyLTA2VDEyOjU1OjAwWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJfN0ZBZXVTU05EV1h2UEV1bjNIQ3Y2ell0S3FPelVUX0lfVV9yT3N3cFg4Lmh0bWwiLCJyZXBvcnRfaWQiOiJjbGE5RWFxbzc1TnRyRS0xVE9Zd041R3kifX19kMtENRTd6rtH16IWTS1shS7qxFs8xqjOQ5zld8GZOQxOe7VE5l2gwyvFZZ-c1GmN2rGZONaLjUY8K9Q3mYBHCQ )