# Antithesis Report 07.11.2023 In this report we ran an experiment with a evil blob producer. Teku provided a evil blob branch, which we used to stress the network. This experiment is a devnet-9 trial run where prysm has some known errors with processing blobs in their pending blob queue. The errors below do not necessitate a issue in the client, but are used as an indication of the kinds of errors we see with an evil client presents. ## Consensus ### Prysm-bn Prysm-bn (Nethermind) ```l! time="2023-10-20 20:41:20" level=error msg="Could not handle p2p pubsub" error="could not validate blob data availability: invalid compressed coordinate: square root doesn't exist" prefix=sync topic="/eth2/2d46aec5/beacon_block/ssz_snappy" ``` [Antithesis Report](https://ethereum.antithesis.com/report/6nbTg8cYYxafxopSyUONG7sylvxt156v/0Qi0Qj97FPhlggmaRhFNio3ylYhdqIL2SYO1ojYpBTI.html?auth=v2.public.eyJuYmYiOiIyMDIzLTExLTA3VDEyOjA0OjI1WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiIwUWkwUWo5N0ZQaGxnZ21hUmhGTmlvM3lsWWhkcUlMMlNZTzFvallwQlRJLmh0bWwiLCJyZXBvcnRfaWQiOiI2bmJUZzhjWVl4YWZ4b3BTeVVPTkc3c3lsdnh0MTU2diJ9fX075X0mfmuFusyRd3G0Dqsh-g5ccvb1hrh8V7V6ReOZUZFE_ELJUoKobkWwCOJp_-iX20bYQxQt7Qzsd0t8N7cG) ### Lighthouse-bn Lighthouse-bn (Besu) In this case the block is received by lighthouse, but head state is not advanced because the block is not part of lighthouse snapshot. This seems to occur when lighthouse goes in and out of sync with the head of the chain. ```l! Oct 20 20:42:50.007 WARN Did not advance head state reason: Err(HeadMissingFromSnapshotCache(0x8877d356146e0e7ce91ba5b168385766b60b582a1202a421a02d2996b7194d58)), service: state_advance ``` Related: ``` [ 574282] [ 2518.984558] [system;capturefs;files;eth/service_lighthouse-besu-0--bn] [I] Oct 20 20:42:43.075 WARN Could not verify blob sidecar for gossip. Ignoring the blob sidecar, index: 0, root: 0xa42b…85a5, slot: 207, error: RepeatBlob { proposer: 18, slot: Slot(207), index: 0 } [ 574570] [ 2522.928867] [system;capturefs;files;eth/service_lighthouse-besu-0--bn] [I] Oct 20 20:42:47.005 INFO Synced slot: 207, block: … empty, epoch: 6, finalized_epoch: 4, finalized_root: 0x239a…853d, exec_hash: 0xa837…93a4 (verified), peers: 5, service: slot_notifier [ 574714] [ 2525.917712] [system;capturefs;files;eth/service_lighthouse-besu-0--bn] [I] Oct 20 20:42:50.007 WARN Did not advance head state reason: Err(HeadMissingFromSnapshotCache(0x8877d356146e0e7ce91ba5b168385766b60b582a1202a421a02d2996b7194d58)), service: state_advance ``` [Antithesis Report](https://ethereum.antithesis.com/report/ctnDykm_FsefQ7jidHVkIvtuj3eNNjo2/IRg5TUZ7VHidSy9ZKI73qex0g1DpVF6PPEKVe8esMKQ.html?auth=v2.public.eyJuYmYiOiIyMDIzLTExLTA3VDEyOjEwOjMzWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJJUmc1VFVaN1ZIaWRTeTlaS0k3M3FleDBnMURwVkY2UFBFS1ZlOGVzTUtRLmh0bWwiLCJyZXBvcnRfaWQiOiJjdG5EeWttX0ZzZWZRN2ppZEhWa0l2dHVqM2VOTmpvMiJ9fX3r3z_Ku8lhhoXikfMVOuakmr67P8wzzHERXMD5AeIIp2GOoO-sqfFnDViSeaUY1Kw9n1eL1jm1YWYcyFWMz0IG) ### Nimbus Nimbus (Geth) ```l! DBG 2023-10-20 20:52:54.941+00:00 Critical error occured while sending response topics="beacnde" meth=GET peer=10.0.20.203:53310 uri=/eth/v1/beacon/states/head/finality_checkpoints code="400 Bad Request" error_msg="Unable to send response" ``` [Antithesis Report](https://ethereum.antithesis.com/report/GGreBUWZH9HkVmicXEA2_F6Wj_8A5pwH/fSDGF26JAtWyeBu4K4MvI1f-Xl-WE2fWZcQLrFDZJss.html?auth=v2.public.eyJuYmYiOiIyMDIzLTExLTA3VDEyOjE5OjMwWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJmU0RHRjI2SkF0V3llQnU0SzRNdkkxZi1YbC1XRTJmV1pjUUxyRkRaSnNzLmh0bWwiLCJyZXBvcnRfaWQiOiJHR3JlQlVXWkg5SGtWbWljWEVBMl9GNldqXzhBNXB3SCJ9fX25gSZIRqEdfoYIeaK9P0yIfVDaqemFDesODhYO-auA4p3PQSPyngjZP3n6yKabb8MPf5YVH92lhThzp3cbDDkB) ## Execution Geth (Prysm) ```l! DEBUG[10-20|20:25:38.379] Message handling failed in `eth` id=5e8c07f9aff14cba conn=staticdial err="disallowed broadcast blob transaction" ``` [Antithesis Report](https://ethereum.antithesis.com/report/NLZMrUdnywUakQMGyC0fSakZBmKEE5nP/BZsZ_GsZBgQgAJ_ukeVwY1BXdP-Wz_yuhcJxBRtjdS4.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiQlpzWl9Hc1pCZ1FnQUpfdWtlVndZMUJYZFAtV3pfeXVoY0p4QlJ0amRTNC5odG1sIiwicmVwb3J0X2lkIjoiTkxaTXJVZG55d1Vha1FNR3lDMGZTYWtaQm1LRUU1blAifX0sIm5iZiI6IjIwMjMtMTEtMDdUMTE6NDk6NDNaIn0qb6VSZG3pO1eJZwrZJPIoRWrvh6AEoaGOpF0M8BIGZrW7N2aVZJ04uNLb4C3xp7BWnjW27xQV0Kj9ghU2vRAO) Geth (Prysm) ```l! DEBUG[10-20|20:43:03.391] Removing p2p peer peercount=5 id=5e8c07f9aff14cba duration=205.898ms req=true err="breach of protocol" ``` [Antithesis Report](https://ethereum.antithesis.com/report/xK3xxHWUbi8dxIxfTnsqm0ymeN-jULdF/V1JWqyhJZEmFiY7iessGjLX6SWrHOydOHumxz2Ghl4o.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiVjFKV3F5aEpaRW1GaVk3aWVzc0dqTFg2U1dySE95ZE9IdW14ejJHaGw0by5odG1sIiwicmVwb3J0X2lkIjoieEszeHhIV1ViaThkeEl4ZlRuc3FtMHltZU4talVMZEYifX0sIm5iZiI6IjIwMjMtMTEtMDdUMTE6NTI6NTJaIn0ssKwplRIA3zahHT-uJhBPF1d0V3SY6KX44QnXz2LHAorpnsduIuvrGE60qvllTZlORRVE27dXVyTs4zlt2tYE) Geth (Nimbus) ```l! DEBUG[10-20|20:30:55.653] Removing p2p peer peercount=5 id=5e8c07f9aff14cba duration=7.413s req=false err="disallowed broadcast blob transaction" ``` [Antithesis Report](https://ethereum.antithesis.com/report/ES5tFoD7096TC7427w92C247dTTZn_9b/E8B7LltjrP7CLC4FABDhA1o7WoWacjLm6Ab_Yp35dtM.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiRThCN0xsdGpyUDdDTEM0RkFCRGhBMW83V29XYWNqTG02QWJfWXAzNWR0TS5odG1sIiwicmVwb3J0X2lkIjoiRVM1dEZvRDcwOTZUQzc0Mjd3OTJDMjQ3ZFRUWm5fOWIifX0sIm5iZiI6IjIwMjMtMTEtMDdUMTI6MzA6NDNaIn3vzfXF4rdxKCRF9XnF7B3P8ICV8TbpBcEBCqZKLANRW-e9nI5GGJ4G95MPkto9MmgVJ2czLcQqqZvw-Ox6ymQH) Geth (Nimbus) ```l! WARN [10-20|20:53:43.897] Failed to reserve account for blob pool tx=68ad94..c2dd45 from=0x1cB96C5809dA5977f99f69c11eE58BaE5711c5f1 err="address already reserved" ``` [Antithesis Report](https://ethereum.antithesis.com/report/rbhykwAM7tKDIf9v3kt6u4gJk8Exy5x9/_ceWbvaQdlgUPeCkWwMP1OunysX9Lz-LmQaY3KiOwKA.html?auth=v2.public.eyJuYmYiOiIyMDIzLTExLTA3VDExOjU4OjU5WiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiJfY2VXYnZhUWRsZ1VQZUNrV3dNUDFPdW55c1g5THotTG1RYVkzS2lPd0tBLmh0bWwiLCJyZXBvcnRfaWQiOiJyYmh5a3dBTTd0S0RJZjl2M2t0NnU0Z0prOEV4eTV4OSJ9fX1poUABavfQHfZxxyAFnvC7SoivvuqM9DmgYuRmyjiWU2CyhQkVO1XXizWrgSn4b7KXTgxh1XEVcVYdEs6KD20M) Besu (Lighthouse) ```l! 2023-10-20 21:00:44.918+00:00 | vert.x-worker-thread-0 | ERROR | AbstractBlockProcessor | failed persisting block ``` [Antithesis Report](https://ethereum.antithesis.com/report/R5we2zr65BgjXRjMzOoJWhUlfs3mwH3x/0Q-dLxyoX8iSE7BHz5ChY2hk05mwYsSzc45iDo4mmhU.html?auth=v2.public.eyJuYmYiOiIyMDIzLTExLTA3VDEyOjI3OjEyWiIsInNjb3BlIjp7IlJlcG9ydFNjb3BlVjEiOnsiYXNzZXQiOiIwUS1kTHh5b1g4aVNFN0JIejVDaFkyaGswNW13WXNTemM0NWlEbzRtbWhVLmh0bWwiLCJyZXBvcnRfaWQiOiJSNXdlMnpyNjVCZ2pYUmpNek9vSldoVWxmczNtd0gzeCJ9fX3gNyd6NSLnZ9xB8KcX__yYOt4l6zc7tUSXyZwgAY9lGV7ViyCRyjfdX0lOJb93td6Tpb_3tZGcQt97MIod6GYN) Besu (Lighthouse) ```l! 2023-10-20 20:30:31.275+00:00 | vert.x-eventloop-thread-1 | ERROR | RoutingContext | Unhandled exception in router ``` [Antithesis Report](https://ethereum.antithesis.com/report/2VQ730_Kk1ugynDxXsjDQXG6AbugZ9Ps/BNnTm_queIHuphevkwbTD20TmacZyTNzkXiJ2tBLN_k.html?auth=v2.public.eyJzY29wZSI6eyJSZXBvcnRTY29wZVYxIjp7ImFzc2V0IjoiQk5uVG1fcXVlSUh1cGhldmt3YlREMjBUbWFjWnlUTnprWGlKMnRCTE5fay5odG1sIiwicmVwb3J0X2lkIjoiMlZRNzMwX0trMXVneW5EeFhzakRRWEc2QWJ1Z1o5UHMifX0sIm5iZiI6IjIwMjMtMTEtMDdUMTI6MjM6MzZaIn2b7EfwDoe_e2_TwoeAFLpHOseX9VX_KuULpyNy6sov-3DlEdnp1lZVzcgL4y0aULEK1DGmy7wlBKUKHzuHP0oP)