HackMD
Shanghai Fork on Goerli: Besu-Focused Recap
TL;DR
Fork was a success. There is no required follow-up for Besu developers and no reason for Besu to delay the Mainnet fork.
Overall Goerli network participation dropped to ~30% and stopped finalising, likely due to nodes with lots of validators not being updated in time, but it starting finalising again after 1.5 hours. We are now back at ~80% participation.
Summary of issues
No outstanding known issues with Besu Goerli nodes.
Besu's Goerli validator EL logs had no errors throughout (we have one node for each CL and 4/5 had some errors in CL logs but all recovered).
Our Web3Signer node (besu+teku) dropped 10K validator for ~1 hour due to missing the Besu version update.
Lodestar dropped 50K validators due to missing an update.
Nimbus had some issues unrelated to the fork.
The Besu nodes' CLs had some errors, mostly related to a spike in BLS changes at the fork.
Lessons learned
- Add Web3signer nodes to the post-release checklist for Besu and Teku (although we don't have a Mainnet web3signer node).
- Enable DEBUG logging on EL + CL pairs across the fork just in case
- Enable metrics for CLs?
Useful links and data
- Tim's summary: https://twitter.com/TimBeiko/status/1635769239272591360
- New dashboard: https://goerli.beaconcha.in/validators/withdrawals
- EthStaker call (not much analysis of the issues so wouldn't recommend)
https://www.youtube.com/watch?v=mu_oKzmvHlc- the important bit :-) https://checkout.poap.xyz/sale/417fc03e-547a-4ff0-9c0c-353cfbd98260
- Upgrade schedule
https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md#upgrade-schedule
| Network | Timestamp | Date & Time (UTC) | Fork Hash | Beacon Chain Epoch |
|---|---|---|---|---|
| Goerli | 1678832736 | 3/14/2023, 10:25:36 PM | 0xf9843abf | 162304 |
- Scheduled epoch: https://prater.beaconcha.in/epoch/162304
- First slot of epoch
162304=5193728but first four slots were missed - Actual first block: https://prater.beaconcha.in/block/8656123 (https://prater.beaconcha.in/slot/5193732)
- EF grafana (requires login from EF) https://grafana.observability.ethpandaops.io/d/MRfYwus7k/nodes?orgId=1&refresh=30s&var-consensus_client=All&var-execution_client=All&var-network=prater&var-filter=ingress_user|!~|synctest.*&viewPanel=19&from=now-6h&to=now
#interop highlights
Barnabas — Today at 08:45
I'm thinking lots of nodes are struggling with the initial bls changes. Ppl tend to run testnets on lower grade hw.
parithosh — Today at 08:45
I think its just that the huge CPU spike from the BLS changes at the fork mess up the goerli machines. I dont think most teams use dedicated infra for goerli. It should stabilize with time.
ralexstokes — Today at 08:45
i did see a lot of "work queue full" messages on my lighthouse node right at the fork
Barnabas — Today at 08:45
and the nodes are being resource exhausted, and thus missing their own proposals and attestations
Right, this might gonna be an issue for a day or two tho, as we going to be missing quite some blocks, and thus process all the bls changes significantly slower.
Barnabas — Today at 08:48
I think some clients store their bls pool in ram, so a huge spike of raw data is pushed to RAM
sproul — Today at 08:51
the changes are quite small, some napkin math tells me 400k changes would only be around 170MB as JSON, even less in-memory
potuz — Today at 08:52
yeah RAM and bandwidth are not an issue
CPU is
the DOS is on signature verification
phil.knows — Today at 09:12
Lodestar's goerli validators have dropped, we're looking into it and trying to push them back up.
phil.knows — Today at 09:27
Our watchtower didn't pull the latest geth stable and was still on 1.11.2, updating now and hope to see them up shortly
We'll see hopefully another 50k ish validators come back online
potuz — Today at 09:22
yeah, we do see a lot of very late Nimbus blocks
tersec — Today at 09:23
yeah, been looking into it more and still no obvious evidence it's strongly fork-related, but it's related to issues we've seen before
parithosh — Today at 09:55
We have finality on goerli 😄
Log snippets surrounding the fork🦉
The errors were shared with the CL teams on #interop
lighthouse-besu
lighthouse:
Mar 14 22:25:36.002 INFO Transitioned to new fork new_fork: Capella, old_fork: Merge, service: network
Mar 14 22:25:36.002 INFO Updating the ENR fork version next_fork_epoch: No other fork, next_fork_version: [3, 0, 16, 32], fork_digest: [98, 137, 65, 239], service: libp2p
Mar 14 22:25:36.016 INFO Network will unsubscribe from old fork gossip topics in a few epochs, remaining_epochs: 2, service: network
Mar 14 22:25:42.001 INFO Synced slot: 5193728, block: … empty, epoch: 162304, finalized_epoch: 162302, finalized_root: 0x5973…9696, exec_hash: 0x1469…8cb7 (verified), peers: 73, service: slot_notifier
Mar 14 22:25:42.001 INFO Ready for Capella service: slot_notifier
Mar 14 22:25:54.000 INFO Synced slot: 5193729, block: … empty, epoch: 162304, finalized_epoch: 162302, finalized_root: 0x5973…9696, exec_hash: 0x1469…8cb7 (verified), peers: 55, service: slot_notifier
Mar 14 22:25:54.001 INFO Ready for Capella service: slot_notifier
Mar 14 22:26:00.101 ERRO Work queue is full queue: gossip_bls_to_execution_change, queue_len: 16384, msg: the system has insufficient resources for load
Mar 14 22:26:00.102 ERRO Work queue is full queue: gossip_bls_to_execution_change, queue_len: 16384, msg: the system has insufficient resources for load
...
...
Mar 14 22:26:13.531 ERRO Work queue is full queue: gossip_bls_to_execution_change, queue_len: 16384, msg: the system has insufficient resources for load
Mar 14 22:26:13.649 INFO New block received root: 0x8c2fced42f67c754b310dd99c401b7894675122faaa7f4872b904872d98c08de, slot: 5193731
Mar 14 22:26:18.000 INFO Synced slot: 5193731, block: … empty, epoch: 162304, finalized_epoch: 162302, finalized_root: 0x5973…9696, exec_hash: 0x1469…8cb7 (verified), peers: 51, service: slot_notifier
Mar 14 22:26:18.001 INFO Ready for Capella service: slot_notifier
Mar 14 22:26:27.523 INFO New block received root: 0x795589c30104e470e22f1f57aff0e4afeee7c61e558065d1a51047c99e29c831, slot: 5193732
Mar 14 22:26:30.002 INFO Synced slot: 5193732, block: … empty, epoch: 162304, finalized_epoch: 162302, finalized_root: 0x5973…9696, exec_hash: 0x1469…8cb7 (verified), peers: 46, service: slot_notifier
Mar 14 22:26:30.003 INFO Ready for Capella service: slot_notifier
Mar 14 22:26:37.496 INFO New block received root: 0x764d67afea2f424d68a4761c7cf67465d251bafd79bf91bc0a0ba25c2150bf87, slot: 5193733
Mar 14 22:26:42.001 INFO Synced slot: 5193733, block: … empty, epoch: 162304, finalized_epoch: 162302, finalized_root: 0x5973…9696, exec_hash: 0x4fed…b334 (verified), peers: 44, service: slot_notifier
Mar 14 22:26:45.000 WARN Did not advance head state reason: Err(HeadMissingFromSnapshotCache(0x795589c30104e470e22f1f57aff0e4afeee7c61e558065d1a51047c99e29c831)), service: state_advance
Mar 14 22:26:54.001 INFO Synced slot: 5193734, block: … empty, epoch: 162304, finalized_epoch: 162302, finalized_root: 0x5973…9696, exec_hash: 0x4fed…b334 (verified), peers: 40, service: slot_notifier
Mar 14 22:26:57.001 WARN Did not advance head state reason: Err(HeadMissingFromSnapshotCache(0x795589c30104e470e22f1f57aff0e4afeee7c61e558065d1a51047c99e29c831)), service: state_advance
Mar 14 22:27:01.764 INFO New block received root: 0xcb7bb6cd72ff87d4b850a14715260de00e82bd17c635af792ba7413de563c3eb, slot: 5193735
Mar 14 22:27:06.000 INFO Synced slot: 5193735, block: … empty, epoch: 162304, finalized_epoch: 162302, finalized_root: 0x5973…9696, exec_hash: 0x4fed…b334 (verified), peers: 40, service: slot_notifier
Mar 14 22:27:09.001 WARN Did not advance head state reason: Err(HeadMissingFromSnapshotCache(0x795589c30104e470e22f1f57aff0e4afeee7c61e558065d1a51047c99e29c831)), service: state_advance
Mar 14 22:27:14.326 INFO New block received root: 0x5df720fdb845e8e45feb7ae7fae4f2eee9ade289f0f215437a1e17d3fa47cd2d, slot: 5193736
Mar 14 22:27:18.000 INFO Synced slot: 5193736, block: 0x5df7…cd2d, epoch: 162304, finalized_epoch: 162302, finalized_root: 0x5973…9696, exec_hash: 0x2a86…1d29 (verified), peers: 39, service: slot_notifier
besu:
{"@timestamp":"2023-03-14T22:25:14,693","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,121 / 168 tx / base fee 2.89 gwei / 29,984,638 (99.9%) gas / (0x1f120a6ce50aa0f9b4b61f92cc7f62626b7d67d973570d84f586d0e3e05e5f34) in 1.771s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:25:25,813","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,122 / 40 tx / base fee 3.25 gwei / 7,422,801 (24.7%) gas / (0x1469313981cb69286cbfbda5c4e9e02018ca6b091939e99cb888ff3fc5ae8cb7) in 0.370s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:26:30,226","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,123 / 183 tx / 0 ws / base fee 3.05 gwei / 29,992,912 (100.0%) gas / (0x4fed6603f749a4bd18a5e3cd21922327a18b20167d084e9e25a36e0a3ffcb334) in 1.637s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:26:31,070","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x4fed6603f749a4bd18a5e3cd21922327a18b20167d084e9e25a36e0a3ffcb334, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:27:17,087","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,124 / 106 tx / 0 ws / base fee 3.43 gwei / 20,874,337 (69.6%) gas / (0x2a869bf7e4590d37846f2e1ec2a9929ddb781b311892bea3b269742db38f1d29) in 1.691s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:27:25,907","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,125 / 108 tx / 0 ws / base fee 3.60 gwei / 18,193,591 (60.6%) gas / (0x75dc5b98ff2a307e59ac01f18809483094f951ec4422c238ed58da474b9506ab) in 0.970s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:28:02,509","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,126 / 114 tx / 0 ws / base fee 3.69 gwei / 17,497,088 (58.3%) gas / (0x1dac75e651ca983c92cffdc72b328327c0ac781113070608df712b68fe091774) in 0.811s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:28:02,625","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x1dac75e651ca983c92cffdc72b328327c0ac781113070608df712b68fe091774, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:28:50,684","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,127 / 138 tx / 0 ws / base fee 3.77 gwei / 29,017,050 (96.7%) gas / (0x70e6500a270910d7141f8cd910591f4b32856e41ac80bf3457e25b9e68ebec50) in 0.865s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:29:27,204","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,128 / 113 tx / 0 ws / base fee 4.21 gwei / 29,707,641 (99.0%) gas / (0x5ec336fe83f99429cbb442e35386ffe206223c706002a2aee7d2564985fd3c3d) in 1.341s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:29:27,338","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x5ec336fe83f99429cbb442e35386ffe206223c706002a2aee7d2564985fd3c3d, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:29:37,300","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,129 / 42 tx / 0 ws / base fee 4.73 gwei / 9,572,260 (31.9%) gas / (0x3c7de036ee08a2e21e65e8368387729755bfc0559bedad9d31d604136c5fe07c) in 0.400s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:29:49,686","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,130 / 37 tx / 0 ws / base fee 4.51 gwei / 6,312,062 (21.0%) gas / (0xbfbe948f0a7b65af308cfd9688c7525d872fe7f9b039f86ebb05d0f9761152aa) in 0.672s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:01,187","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,131 / 34 tx / 0 ws / base fee 4.19 gwei / 6,498,664 (21.7%) gas / (0xfabe30ece41a1ac73266bc931f639cef6140453d79d886849fa1a79180830053) in 0.283s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:34,164","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,132 / 88 tx / 0 ws / base fee 3.89 gwei / 18,806,012 (62.7%) gas / (0xf9580ba0c25e6f4a39c55bc645cfffb5595717e34f5e8565f4f62e88d8eeecce) in 1.762s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:34,266","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0xf9580ba0c25e6f4a39c55bc645cfffb5595717e34f5e8565f4f62e88d8eeecce, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:30:37,945","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,133 / 31 tx / 0 ws / base fee 4.01 gwei / 3,497,579 (11.7%) gas / (0x6974758c64fc440ce9fd9d20df2584ccbd6d8f0e3c89a85be17c8cd2f3536d44) in 0.503s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:53,115","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,133 / 73 tx / 0 ws / base fee 4.01 gwei / 12,718,436 (42.4%) gas / (0x444621c5b40a8f8cc213120d30d429ec4d392d7b570b8098e366b51cd58df9ed) in 0.469s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:31:15,399","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,134 / 132 tx / 0 ws / base fee 3.63 gwei / 25,184,038 (83.9%) gas / (0xca28843497204658716de4df75d5c36c34a53c570e3639c3ec4cbee77827c11d) in 1.076s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:31:38,282","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,135 / 69 tx / 16 ws / base fee 3.94 gwei / 11,461,885 (38.2%) gas / (0x7c907a20f0ff1de3b40fff5c362561dde9606507cfe9e3617846bb20f2db0599) in 0.770s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:31:38,379","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x7c907a20f0ff1de3b40fff5c362561dde9606507cfe9e3617846bb20f2db0599, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:32:03,317","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,136 / 61 tx / 16 ws / base fee 3.82 gwei / 11,860,079 (39.5%) gas / (0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d) in 0.634s. Peers: 24","throwable":""}
lodestar-besu
lodestar:
Mar-14 22:24:42.010[] info: Synced - slot: 5193723 - head: 5193723 0x224a…8375 - execution: valid(0x4cef…384b) - finalized: 0xd17c…42f8:162301 - peers: 52
Mar-14 22:24:54.112[] info: Synced - slot: 5193724 (skipped 1) - head: 5193723 0x224a…8375 - execution: valid(0x4cef…384b) - finalized: 0xd17c…42f8:162301 - peers: 50
Mar-14 22:25:07.494[] info: Synced - slot: 5193725 (skipped 2) - head: 5193723 0x224a…8375 - execution: valid(0x4cef…384b) - finalized: 0xd17c…42f8:162301 - peers: 50
Mar-14 22:25:17.999[] info: Synced - slot: 5193726 - head: 5193726 0xfca1…7c97 - execution: valid(0x1f12…5f34) - finalized: 0xd17c…42f8:162301 - peers: 50
Mar-14 22:25:30.003[] info: Synced - slot: 5193727 - head: 5193727 0x6a7a…af3b - execution: valid(0x1469…8cb7) - finalized: 0xd17c…42f8:162301 - peers: 50
Mar-14 22:25:37.228[network] error: Error onGossipsubMessage code=QUEUE_ERROR_QUEUE_MAX_LENGTH
Error: QUEUE_ERROR_QUEUE_MAX_LENGTH
at JobItemQueue.push (file:///usr/app/packages/beacon-node/src/util/queue/itemQueue.ts:53:15)
at Object.gossipValidatorFnWithQueue [as bls_to_execution_change] (file:///usr/app/packages/beacon-node/src/network/gossip/validation/index.ts:42:31)
at Eth2Gossipsub.onGossipsubMessage (file:///usr/app/packages/beacon-node/src/network/gossip/gossipsub.ts:401:40)
at Eth2Gossipsub.[nodejs.internal.kHybridDispatch] (node:internal/event_target:735:20)
at Eth2Gossipsub.dispatchEvent (node:internal/event_target:677:26)
at Eth2Gossipsub.dispatchEvent (file:///usr/app/node_modules/@libp2p/interfaces/src/events.ts:64:26)
at Eth2Gossipsub.handleReceivedMessage (file:///usr/app/node_modules/@chainsafe/libp2p-gossipsub/src/index.ts:1104:19)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
...
...
ATTESTATION_ERROR_INVALID_SIGNATURE
at Object.submitPoolAttestations (file:///usr/app/packages/beacon-node/src/api/impl/beacon/pool/index.ts:75:15)
at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:44:23)
Mar-14 22:26:28.769[chain] info:
/^----^\
| ◡ ◡ |
| \/ |
/ \
| |;;;|
| |;;;|
| \;;| \
\/\/ | ~|~ |-| |) /? /\ \/\/ \ \| / /
---------------------------------------(((--(((------------\ \-----,
-- ___ ---- __ -- ___ ---- __ --- ____ ---- _____ -- \
-- __ -- _____ --- __ -- __ -- _____ --- __ ---- ___ ---- _/
------------------------------------/ /---------------\ \---------\`
\ \ / /
//\ //
\ \\
/
Mar-14 22:26:28.769[chain] info: Activating withdrawals epoch=162304
Mar-14 22:26:30.000[] info: Synced - slot: 5193732 - head: 5193732 0x7955…c831 - execution: valid(0x4fed…b334) - finalized: 0x5973…9696:162302 - peers: 43
Mar-14 22:26:32.126[rest] error: Req req-3s9f getAggregatedAttestation error No attestation for slot=5193732 dataRoot=0x894b063a72343068f1c5b302cf5921dd49c7ce3a6a501b0943f7d6dd4e1e1272
Error: No attestation for slot=5193732 dataRoot=0x894b063a72343068f1c5b302cf5921dd49c7ce3a6a501b0943f7d6dd4e1e1272
at AttestationPool.getAggregate (file:///usr/app/packages/beacon-node/src/chain/opPools/attestationPool.ts:126:13)
at Object.getAggregatedAttestation (file:///usr/app/packages/beacon-node/src/api/impl/validator/index.ts:495:37)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at Object.handler (file:///usr/app/packages/api/src/utils/server/genericJsonServer.ts:44:23)
besu:
{"@timestamp":"2023-03-14T22:25:13,569","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,121 / 168 tx / base fee 2.89 gwei / 29,984,638 (99.9%) gas / (0x1f120a6ce50aa0f9b4b61f92cc7f62626b7d67d973570d84f586d0e3e05e5f34) in 0.508s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:25:13,955","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x1f120a6ce50aa0f9b4b61f92cc7f62626b7d67d973570d84f586d0e3e05e5f34, finalized: 0x4593144bddce24d12c8f8a845665b5d2a78b0685e9e2e3b9ff38894c9727b6b4, safeBlockHash: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff","throwable":""}
{"@timestamp":"2023-03-14T22:25:25,721","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,122 / 40 tx / base fee 3.25 gwei / 7,422,801 (24.7%) gas / (0x1469313981cb69286cbfbda5c4e9e02018ca6b091939e99cb888ff3fc5ae8cb7) in 0.164s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:26:14,604","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,123 / 169 tx / 0 ws / base fee 3.05 gwei / 29,983,935 (99.9%) gas / (0x037a6fc94d8f00597aaa370a830866bbcae141c141d12f94d9992b33398374e4) in 0.851s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:26:28,722","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,123 / 183 tx / 0 ws / base fee 3.05 gwei / 29,992,912 (100.0%) gas / (0x4fed6603f749a4bd18a5e3cd21922327a18b20167d084e9e25a36e0a3ffcb334) in 0.591s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:26:29,173","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x4fed6603f749a4bd18a5e3cd21922327a18b20167d084e9e25a36e0a3ffcb334, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:26:38,802","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,124 / 81 tx / 0 ws / base fee 3.43 gwei / 16,883,969 (56.3%) gas / (0x9fa304882c45848e0c8afe2481d46bd527eee8c9775251de3da263f023f71a46) in 0.350s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:27:02,838","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,124 / 170 tx / 0 ws / base fee 3.43 gwei / 29,983,113 (99.9%) gas / (0x7abefd9f829a42036d8dae1cee29503c421f07984ee424f68307cd92d5848073) in 0.486s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:27:15,570","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,124 / 106 tx / 0 ws / base fee 3.43 gwei / 20,874,337 (69.6%) gas / (0x2a869bf7e4590d37846f2e1ec2a9929ddb781b311892bea3b269742db38f1d29) in 0.261s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:27:24,690","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,125 / 108 tx / 0 ws / base fee 3.60 gwei / 18,193,591 (60.6%) gas / (0x75dc5b98ff2a307e59ac01f18809483094f951ec4422c238ed58da474b9506ab) in 0.320s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:27:50,953","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,126 / 90 tx / 0 ws / base fee 3.69 gwei / 10,967,695 (36.6%) gas / (0xdc7294122cfb975ab5c453b018f43bc433492a9847d5d0adbabf853c8b8b76cb) in 0.252s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:28:00,676","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,126 / 114 tx / 0 ws / base fee 3.69 gwei / 17,497,088 (58.3%) gas / (0x1dac75e651ca983c92cffdc72b328327c0ac781113070608df712b68fe091774) in 0.231s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:28:00,973","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x1dac75e651ca983c92cffdc72b328327c0ac781113070608df712b68fe091774, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:28:50,521","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,127 / 138 tx / 0 ws / base fee 3.77 gwei / 29,017,050 (96.7%) gas / (0x70e6500a270910d7141f8cd910591f4b32856e41ac80bf3457e25b9e68ebec50) in 0.350s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:29:26,140","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,128 / 113 tx / 0 ws / base fee 4.21 gwei / 29,707,641 (99.0%) gas / (0x5ec336fe83f99429cbb442e35386ffe206223c706002a2aee7d2564985fd3c3d) in 0.420s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:29:26,528","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x5ec336fe83f99429cbb442e35386ffe206223c706002a2aee7d2564985fd3c3d, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:29:36,535","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,129 / 42 tx / 0 ws / base fee 4.73 gwei / 9,572,260 (31.9%) gas / (0x3c7de036ee08a2e21e65e8368387729755bfc0559bedad9d31d604136c5fe07c) in 0.157s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:29:49,466","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,130 / 37 tx / 0 ws / base fee 4.51 gwei / 6,312,062 (21.0%) gas / (0xbfbe948f0a7b65af308cfd9688c7525d872fe7f9b039f86ebb05d0f9761152aa) in 0.152s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:00,844","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,131 / 34 tx / 0 ws / base fee 4.19 gwei / 6,498,664 (21.7%) gas / (0xfabe30ece41a1ac73266bc931f639cef6140453d79d886849fa1a79180830053) in 0.130s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:32,605","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,132 / 88 tx / 0 ws / base fee 3.89 gwei / 18,806,012 (62.7%) gas / (0xf9580ba0c25e6f4a39c55bc645cfffb5595717e34f5e8565f4f62e88d8eeecce) in 0.460s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:32,847","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0xf9580ba0c25e6f4a39c55bc645cfffb5595717e34f5e8565f4f62e88d8eeecce, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:30:37,260","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,133 / 31 tx / 0 ws / base fee 4.01 gwei / 3,497,579 (11.7%) gas / (0x6974758c64fc440ce9fd9d20df2584ccbd6d8f0e3c89a85be17c8cd2f3536d44) in 0.119s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:51,513","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,133 / 73 tx / 0 ws / base fee 4.01 gwei / 12,718,436 (42.4%) gas / (0x444621c5b40a8f8cc213120d30d429ec4d392d7b570b8098e366b51cd58df9ed) in 0.205s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:31:14,590","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,134 / 132 tx / 0 ws / base fee 3.63 gwei / 25,184,038 (83.9%) gas / (0xca28843497204658716de4df75d5c36c34a53c570e3639c3ec4cbee77827c11d) in 0.353s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:31:37,346","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,135 / 69 tx / 16 ws / base fee 3.94 gwei / 11,461,885 (38.2%) gas / (0x7c907a20f0ff1de3b40fff5c362561dde9606507cfe9e3617846bb20f2db0599) in 0.293s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:31:37,592","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x7c907a20f0ff1de3b40fff5c362561dde9606507cfe9e3617846bb20f2db0599, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:32:02,049","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,136 / 61 tx / 16 ws / base fee 3.82 gwei / 11,860,079 (39.5%) gas / (0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d) in 0.216s. Peers: 25","throwable":""}
nimbus-besu
nimbus:
INF 2023-03-14 22:26:12.000+00:00 Slot start topics="beacnde" slot=5193731 epoch=162304 sync=synced peers=131 head=6a7ac459:5193727 finalized=162301:d17c7610 delay=530us497ns
INF 2023-03-14 22:26:14.077+00:00 Block signature verification failed signature=b86827a8 blockRoot=8c2fced4 blck="(slot: 5193731, proposer_index: 178565, parent_root: \"6a7ac459\", state_root: \"87586685\", eth1data: (deposit_root: 9db3faed38a7795c7de171af56d09671d9c10ae3356d43b1f7099582fa823c83, deposit_count: 255518, block_hash: 03f1cdbcaf55d268bdca6f6a3e9c676982727a4889b9a5e77a93014e94c4169e), graffiti: \"Nimbus/v23.3.0-17c0ee-stateofus\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 128, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 343, block_number: 8656123, fee_recipient: \"0x455e5aa18469bc6ccef49594645666c587a3a71b\")"
...
INF 2023-03-14 22:26:16.082+00:00 Sync committee message sent message="(slot: 5193731, beacon_block_root: \"6a7ac459\", validator_index: 373304, signature: \"b8dddcd0\")" delay=82ms29us293ns
INF 2023-03-14 22:26:16.083+00:00 Sync committee message sent message="(slot: 5193731, beacon_block_root: \"6a7ac459\", validator_index: 389082, signature: \"aac3dfd8\")" delay=82ms906us742ns
INF 2023-03-14 22:26:20.121+00:00 Slot end topics="beacnde" slot=5193731 nextActionWait=3s878ms559us51ns nextAttestationSlot=5193732 nextProposalSlot=-1 syncCommitteeDuties="in 01d03h17m" head=6a7ac459:5193727
INF 2023-03-14 22:26:24.000+00:00 Slot start topics="beacnde" slot=5193732 epoch=162304 sync=synced peers=104 head=6a7ac459:5193727 finalized=162301:d17c7610 delay=645us799ns
INF 2023-03-14 22:26:28.078+00:00 Sync committee message sent message="(slot: 5193732, beacon_block_root: \"6a7ac459\", validator_index: 373304, signature: \"b8dddcd0\")" delay=78ms207us982ns
INF 2023-03-14 22:26:28.079+00:00 Sync committee message sent message="(slot: 5193732, beacon_block_root: \"6a7ac459\", validator_index: 389082, signature: \"aac3dfd8\")" delay=79ms92us682ns
NOT 2023-03-14 22:26:28.701+00:00
. . .
. _ . :
$.. . %_...- :
. /{$$$b' ./$#% < |
: Q. $b. |/$$#% | |
: . `|%.. $b__....._____/$$#% .'^ . |
: `.%x. %%% %% .%% ^ - -- - --+---
| .)%:% $b%%%%%%%%%%%% .% . |
| |%%% ==$$%%%%%X__-=$'x%%% |
| d@%% .:::.`. .'.:::. %%% . . . |
----+-- - - :$%%x ::.::.d ::.:: %%x :
| . :$%%% `::'d' .P `::' %%" . ---- - . . :
| :$%%%x. .C' d' %%";%. : .
| . !%%%% ^ | .P' %Xx . % :
: : Q%%%%% `Q :
: . . . : ."$##% . | . .
. | : :d$##%% ^ .d$%% . | | :
|| :: |##%x ^ . $$$%%% . | || ::
. ||| ::: d##%%% ^ Q$$%%%%% - -- ---+||| :::
||\\ //:: :$####%% . Q$$$#%%% ||\\ //:: .
|\\\\ ////: - --d$##%%% ^ . Q$$$%%% . |\\\\ ////:
.. \\ // .. $$#%%%^ ^ Q$$%%x . .. \\ // ..
`::.. ..:;' .$$$##%% ^ . $$%%%% : `::.. ..:;'
`:: ::' .$$BS22#%%% ^ . ~Q$%%%% . `:: ::'
d b $$$$$ $$$$$ $ 5 $$$b $$$$b d$$$b d b $ d$$$$ d$$$b $b $
$ $ $ $ $mmm$ $ $ $ $ $ $ $ $ $ Q. $ $ $`$ $
$ $ $ $ $ $"" $ $ $ $""$b $. $ $ $ $ $ `""$b $. .$ $ `$$
`$$$' $$$$$ $ $ $ $$$P' $ `$ `$$$' `$$$' $$$$$ `$$$P `$$$' $ :$
INF 2023-03-14 22:26:32.134+00:00 Slot end topics="beacnde" slot=5193732 nextActionWait=3s865ms609us791ns nextAttestationSlot=5193733 nextProposalSlot=-1 syncCommitteeDuties=current head=795589c3:5193732
INF 2023-03-14 22:26:32.140+00:00 Contribution sent contribution="(slot: 5193732, beacon_block_root: \"6a7ac459\", subcommittee_index: 1, aggregation_bits: \"0b00100000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000001\")" aggregator_index=373304 selection_proof=95616554 signature=8d6b03dd delay=139ms46us91ns
INF 2023-03-14 22:26:36.001+00:00 Slot start topics="beacnde" slot=5193733 epoch=162304 sync=synced peers=96 head=795589c3:5193732 finalized=162302:59738387 delay=1ms105us931ns
INF 2023-03-14 22:26:37.593+00:00 Block signature verification failed signature=99f66c5e blockRoot=764d67af blck="(slot: 5193733, proposer_index: 303695, parent_root: \"795589c3\", state_root: \"b9a1f0f7\", eth1data: (deposit_root: 9db3faed38a7795c7de171af56d09671d9c10ae3356d43b1f7099582fa823c83, deposit_count: 255518, block_hash: 03f1cdbcaf55d268bdca6f6a3e9c676982727a4889b9a5e77a93014e94c4169e), graffiti: \"Nimbus/v23.3.0-17c0ee-stateofus\", proposer_slashings_len: 0, attester_slashings_len: 0, attestations_len: 128, deposits_len: 0, voluntary_exits_len: 0, sync_committee_participants: 45, block_number: 8656124, fee_recipient: \"0x455e5aa18469bc6ccef49594645666c587a3a71b\")"
...
INF 2023-03-14 22:26:44.119+00:00 Slot end topics="beacnde" slot=5193733 nextActionWait=3s880ms438us843ns nextAttestationSlot=5193734 nextProposalSlot=-1 syncCommitteeDuties=current head=795589c3:5193732
besu:
{"@timestamp":"2023-03-14T22:25:13,678","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,121 / 168 tx / base fee 2.89 gwei / 29,984,638 (99.9%) gas / (0x1f120a6ce50aa0f9b4b61f92cc7f62626b7d67d973570d84f586d0e3e05e5f34) in 0.462s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:25:25,498","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,122 / 40 tx / base fee 3.25 gwei / 7,422,801 (24.7%) gas / (0x1469313981cb69286cbfbda5c4e9e02018ca6b091939e99cb888ff3fc5ae8cb7) in 0.111s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:25:26,174","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x1469313981cb69286cbfbda5c4e9e02018ca6b091939e99cb888ff3fc5ae8cb7, finalized: 0x4593144bddce24d12c8f8a845665b5d2a78b0685e9e2e3b9ff38894c9727b6b4, safeBlockHash: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff","throwable":""}
{"@timestamp":"2023-03-14T22:26:13,925","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,123 / 169 tx / 0 ws / base fee 3.05 gwei / 29,983,935 (99.9%) gas / (0x037a6fc94d8f00597aaa370a830866bbcae141c141d12f94d9992b33398374e4) in 0.539s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:26:28,412","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,123 / 183 tx / 0 ws / base fee 3.05 gwei / 29,992,912 (100.0%) gas / (0x4fed6603f749a4bd18a5e3cd21922327a18b20167d084e9e25a36e0a3ffcb334) in 0.293s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:26:28,899","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x4fed6603f749a4bd18a5e3cd21922327a18b20167d084e9e25a36e0a3ffcb334, finalized: 0x4593144bddce24d12c8f8a845665b5d2a78b0685e9e2e3b9ff38894c9727b6b4, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:26:37,503","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,124 / 81 tx / 0 ws / base fee 3.43 gwei / 16,883,969 (56.3%) gas / (0x9fa304882c45848e0c8afe2481d46bd527eee8c9775251de3da263f023f71a46) in 0.239s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:27:01,519","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,124 / 170 tx / 0 ws / base fee 3.43 gwei / 29,983,113 (99.9%) gas / (0x7abefd9f829a42036d8dae1cee29503c421f07984ee424f68307cd92d5848073) in 0.388s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:27:14,122","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,124 / 106 tx / 0 ws / base fee 3.43 gwei / 20,874,337 (69.6%) gas / (0x2a869bf7e4590d37846f2e1ec2a9929ddb781b311892bea3b269742db38f1d29) in 0.186s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:27:24,888","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,125 / 108 tx / 0 ws / base fee 3.60 gwei / 18,193,591 (60.6%) gas / (0x75dc5b98ff2a307e59ac01f18809483094f951ec4422c238ed58da474b9506ab) in 0.218s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:27:50,388","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,126 / 90 tx / 0 ws / base fee 3.69 gwei / 10,967,695 (36.6%) gas / (0xdc7294122cfb975ab5c453b018f43bc433492a9847d5d0adbabf853c8b8b76cb) in 0.151s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:28:01,210","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,126 / 114 tx / 0 ws / base fee 3.69 gwei / 17,497,088 (58.3%) gas / (0x1dac75e651ca983c92cffdc72b328327c0ac781113070608df712b68fe091774) in 0.179s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:28:01,530","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x1dac75e651ca983c92cffdc72b328327c0ac781113070608df712b68fe091774, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:28:49,717","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,127 / 138 tx / 0 ws / base fee 3.77 gwei / 29,017,050 (96.7%) gas / (0x70e6500a270910d7141f8cd910591f4b32856e41ac80bf3457e25b9e68ebec50) in 0.291s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:29:25,685","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,128 / 113 tx / 0 ws / base fee 4.21 gwei / 29,707,641 (99.0%) gas / (0x5ec336fe83f99429cbb442e35386ffe206223c706002a2aee7d2564985fd3c3d) in 0.348s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:29:26,034","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x5ec336fe83f99429cbb442e35386ffe206223c706002a2aee7d2564985fd3c3d, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:29:37,144","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,129 / 42 tx / 0 ws / base fee 4.73 gwei / 9,572,260 (31.9%) gas / (0x3c7de036ee08a2e21e65e8368387729755bfc0559bedad9d31d604136c5fe07c) in 0.121s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:29:49,150","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,130 / 37 tx / 0 ws / base fee 4.51 gwei / 6,312,062 (21.0%) gas / (0xbfbe948f0a7b65af308cfd9688c7525d872fe7f9b039f86ebb05d0f9761152aa) in 0.116s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:00,757","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,131 / 34 tx / 0 ws / base fee 4.19 gwei / 6,498,664 (21.7%) gas / (0xfabe30ece41a1ac73266bc931f639cef6140453d79d886849fa1a79180830053) in 0.082s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:32,424","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,132 / 88 tx / 0 ws / base fee 3.89 gwei / 18,806,012 (62.7%) gas / (0xf9580ba0c25e6f4a39c55bc645cfffb5595717e34f5e8565f4f62e88d8eeecce) in 0.304s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:32,730","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0xf9580ba0c25e6f4a39c55bc645cfffb5595717e34f5e8565f4f62e88d8eeecce, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:30:37,139","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,133 / 31 tx / 0 ws / base fee 4.01 gwei / 3,497,579 (11.7%) gas / (0x6974758c64fc440ce9fd9d20df2584ccbd6d8f0e3c89a85be17c8cd2f3536d44) in 0.078s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:51,337","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,133 / 73 tx / 0 ws / base fee 4.01 gwei / 12,718,436 (42.4%) gas / (0x444621c5b40a8f8cc213120d30d429ec4d392d7b570b8098e366b51cd58df9ed) in 0.118s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:31:14,355","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,134 / 132 tx / 0 ws / base fee 3.63 gwei / 25,184,038 (83.9%) gas / (0xca28843497204658716de4df75d5c36c34a53c570e3639c3ec4cbee77827c11d) in 0.324s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:31:37,161","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,135 / 69 tx / 16 ws / base fee 3.94 gwei / 11,461,885 (38.2%) gas / (0x7c907a20f0ff1de3b40fff5c362561dde9606507cfe9e3617846bb20f2db0599) in 0.200s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:31:37,449","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x7c907a20f0ff1de3b40fff5c362561dde9606507cfe9e3617846bb20f2db0599, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:32:01,486","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,136 / 61 tx / 16 ws / base fee 3.82 gwei / 11,860,079 (39.5%) gas / (0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d) in 0.188s. Peers: 25","throwable":""}
prysm-besu
prysm:
time="2023-03-14 22:25:13" level=info msg="Synced new block" block=0xfca16416... epoch=162303 finalizedEpoch=162301 finalizedRoot=0xd17c7610... prefix=blockchain slot=5193726
time="2023-03-14 22:25:13" level=info msg="Finished applying state transition" attestations=128 payloadHash=0x1f120a6ce50a prefix=blockchain slot=5193726 syncBitsCount=362 txCount=168
time="2023-03-14 22:25:21" level=info msg="Peer summary" activePeers=55 inbound=45 outbound=10 prefix=p2p
time="2023-03-14 22:25:26" level=info msg="Synced new block" block=0x6a7ac459... epoch=162303 finalizedEpoch=162301 finalizedRoot=0xd17c7610... prefix=blockchain slot=5193727
time="2023-03-14 22:25:26" level=info msg="Finished applying state transition" attestations=128 payloadHash=0x1469313981cb prefix=blockchain slot=5193727 syncBitsCount=359 txCount=40
time="2023-03-14 22:25:36" level=warning msg="Attestation subnets with digest 0xc2ce3aa8 are no longer valid, unsubscribing from all of them." prefix=sync
time="2023-03-14 22:25:36" level=warning msg="Sync subnets with digest 0xc2ce3aa8 are no longer valid, unsubscribing from all of them." prefix=sync
time="2023-03-14 22:26:14" level=error msg="Could not handle p2p pubsub" error="could not process block: could not execute state transition: could not process block: process_sync_aggregate failed: could not verify sync committee signature: invalid sync committee signature" prefix=sync topic="/eth2/628941ef/beacon_block/ssz_snappy"
time="2023-03-14 22:26:21" level=info msg="Peer summary" activePeers=43 inbound=37 outbound=6 prefix=p2p
time="2023-03-14 22:26:30" level=info msg="Synced new block" block=0x795589c3... epoch=162304 finalizedEpoch=162302 finalizedRoot=0x59738387... prefix=blockchain slot=5193732
time="2023-03-14 22:26:30" level=info msg="Finished applying state transition" attestations=128 payloadHash=0x4fed6603f749 prefix=blockchain slot=5193732 syncBitsCount=200 txCount=183
time="2023-03-14 22:26:37" level=error msg="Could not handle p2p pubsub" error="could not process block: signature in block failed to verify" prefix=sync topic="/eth2/628941ef/beacon_block/ssz_snappy"
time="2023-03-14 22:27:01" level=error msg="Could not handle p2p pubsub" error="could not process block: signature in block failed to verify" prefix=sync topic="/eth2/628941ef/beacon_block/ssz_snappy"
time="2023-03-14 22:27:15" level=info msg="Synced new block" block=0x5df720fd... epoch=162304 finalizedEpoch=162302 finalizedRoot=0x59738387... prefix=blockchain slot=5193736
time="2023-03-14 22:27:15" level=info msg="Finished applying state transition" attestations=128 payloadHash=0x2a869bf7e459 prefix=blockchain slot=5193736 syncBitsCount=206 txCount=106
time="2023-03-14 22:27:21" level=info msg="Peer summary" activePeers=46 inbound=35 outbound=11 prefix=p2p
time="2023-03-14 22:27:25" level=info msg="Synced new block" block=0x3e888f1b... epoch=162304 finalizedEpoch=162302 finalizedRoot=0x59738387... prefix=blockchain slot=5193737
time="2023-03-14 22:27:25" level=info msg="Finished applying state transition" attestations=128 payloadHash=0x75dc5b98ff2a prefix=blockchain slot=5193737 syncBitsCount=215 txCount=108
time="2023-03-14 22:27:50" level=error msg="Could not handle p2p pubsub" error="could not process block: signature in block failed to verify" prefix=sync topic="/eth2/628941ef/beacon_block/ssz_snappy"
time="2023-03-14 22:28:01" level=info msg="Synced new block" block=0x44ebce5b... epoch=162304 finalizedEpoch=162302 finalizedRoot=0x59738387... prefix=blockchain slot=5193740
time="2023-03-14 22:28:01" level=info msg="Finished applying state transition" attestations=128 payloadHash=0x1dac75e651ca prefix=blockchain slot=5193740 syncBitsCount=242 txCount=114
time="2023-03-14 22:28:21" level=info msg="Peer summary" activePeers=46 inbound=36 outbound=10 prefix=p2p
time="2023-03-14 22:28:49" level=info msg="Synced new block" block=0xad2aef54... epoch=162304 finalizedEpoch=162302 finalizedRoot=0x59738387... prefix=blockchain slot=5193744
time="2023-03-14 22:28:49" level=info msg="Finished applying state transition" attestations=128 payloadHash=0x70e6500a2709 prefix=blockchain slot=5193744 syncBitsCount=283 txCount=138
besu:
{"@timestamp":"2023-03-14T22:25:13,239","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,121 / 168 tx / base fee 2.89 gwei / 29,984,638 (99.9%) gas / (0x1f120a6ce50aa0f9b4b61f92cc7f62626b7d67d973570d84f586d0e3e05e5f34) in 0.538s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:25:13,440","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x1f120a6ce50aa0f9b4b61f92cc7f62626b7d67d973570d84f586d0e3e05e5f34, finalized: 0x4593144bddce24d12c8f8a845665b5d2a78b0685e9e2e3b9ff38894c9727b6b4, safeBlockHash: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff","throwable":""}
{"@timestamp":"2023-03-14T22:25:25,616","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,122 / 40 tx / base fee 3.25 gwei / 7,422,801 (24.7%) gas / (0x1469313981cb69286cbfbda5c4e9e02018ca6b091939e99cb888ff3fc5ae8cb7) in 0.122s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:26:29,002","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,123 / 183 tx / 0 ws / base fee 3.05 gwei / 29,992,912 (100.0%) gas / (0x4fed6603f749a4bd18a5e3cd21922327a18b20167d084e9e25a36e0a3ffcb334) in 0.527s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:26:29,166","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x4fed6603f749a4bd18a5e3cd21922327a18b20167d084e9e25a36e0a3ffcb334, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:27:14,946","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,124 / 106 tx / 0 ws / base fee 3.43 gwei / 20,874,337 (69.6%) gas / (0x2a869bf7e4590d37846f2e1ec2a9929ddb781b311892bea3b269742db38f1d29) in 0.330s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:27:24,928","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,125 / 108 tx / 0 ws / base fee 3.60 gwei / 18,193,591 (60.6%) gas / (0x75dc5b98ff2a307e59ac01f18809483094f951ec4422c238ed58da474b9506ab) in 0.328s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:28:01,564","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,126 / 114 tx / 0 ws / base fee 3.69 gwei / 17,497,088 (58.3%) gas / (0x1dac75e651ca983c92cffdc72b328327c0ac781113070608df712b68fe091774) in 0.276s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:28:01,666","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x1dac75e651ca983c92cffdc72b328327c0ac781113070608df712b68fe091774, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:28:49,498","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,127 / 138 tx / 0 ws / base fee 3.77 gwei / 29,017,050 (96.7%) gas / (0x70e6500a270910d7141f8cd910591f4b32856e41ac80bf3457e25b9e68ebec50) in 0.416s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:29:25,628","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,128 / 113 tx / 0 ws / base fee 4.21 gwei / 29,707,641 (99.0%) gas / (0x5ec336fe83f99429cbb442e35386ffe206223c706002a2aee7d2564985fd3c3d) in 0.446s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:29:25,786","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x5ec336fe83f99429cbb442e35386ffe206223c706002a2aee7d2564985fd3c3d, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:29:36,728","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,129 / 42 tx / 0 ws / base fee 4.73 gwei / 9,572,260 (31.9%) gas / (0x3c7de036ee08a2e21e65e8368387729755bfc0559bedad9d31d604136c5fe07c) in 0.151s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:29:49,033","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,130 / 37 tx / 0 ws / base fee 4.51 gwei / 6,312,062 (21.0%) gas / (0xbfbe948f0a7b65af308cfd9688c7525d872fe7f9b039f86ebb05d0f9761152aa) in 0.273s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:00,523","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,131 / 34 tx / 0 ws / base fee 4.19 gwei / 6,498,664 (21.7%) gas / (0xfabe30ece41a1ac73266bc931f639cef6140453d79d886849fa1a79180830053) in 0.115s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:32,485","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,132 / 88 tx / 0 ws / base fee 3.89 gwei / 18,806,012 (62.7%) gas / (0xf9580ba0c25e6f4a39c55bc645cfffb5595717e34f5e8565f4f62e88d8eeecce) in 0.405s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:32,615","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0xf9580ba0c25e6f4a39c55bc645cfffb5595717e34f5e8565f4f62e88d8eeecce, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:30:37,176","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,133 / 31 tx / 0 ws / base fee 4.01 gwei / 3,497,579 (11.7%) gas / (0x6974758c64fc440ce9fd9d20df2584ccbd6d8f0e3c89a85be17c8cd2f3536d44) in 0.123s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:51,655","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,133 / 73 tx / 0 ws / base fee 4.01 gwei / 12,718,436 (42.4%) gas / (0x444621c5b40a8f8cc213120d30d429ec4d392d7b570b8098e366b51cd58df9ed) in 0.230s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:31:14,156","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,134 / 132 tx / 0 ws / base fee 3.63 gwei / 25,184,038 (83.9%) gas / (0xca28843497204658716de4df75d5c36c34a53c570e3639c3ec4cbee77827c11d) in 0.390s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:31:37,195","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,135 / 69 tx / 16 ws / base fee 3.94 gwei / 11,461,885 (38.2%) gas / (0x7c907a20f0ff1de3b40fff5c362561dde9606507cfe9e3617846bb20f2db0599) in 0.265s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:31:37,309","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x7c907a20f0ff1de3b40fff5c362561dde9606507cfe9e3617846bb20f2db0599, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:32:02,098","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,136 / 61 tx / 16 ws / base fee 3.82 gwei / 11,860,079 (39.5%) gas / (0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d) in 0.221s. Peers: 25","throwable":""}
teku-besu
teku:
2023-03-14 22:25:28.001 INFO - Slot Event *** Slot: 5193727, Block: 6a7ac459f234540c9975e4eb98049dd3bde33141246916b9f446b7f98e17af3b, Justified: 162302, Finalized: 162301, Peers: 82
2023-03-14 22:25:32.034 INFO - Validator *** Published aggregate Count: 1, Slot: 5193727, Root: 6a7ac459f234540c9975e4eb98049dd3bde33141246916b9f446b7f98e17af3b
2023-03-14 22:25:36.016 INFO - Milestone *** Epoch: 162304, Activating network upgrade: CAPELLA
2023-03-14 22:25:36.016 INFO - Epoch Event *** Epoch: 162304, Justified checkpoint: 162303, Finalized checkpoint: 162302, Finalized root: 59738387e8be4fdbaac9ca07ae39c214414ad145bd81e7fe6b3b6ecf6c639696
2023-03-14 22:25:37.796 INFO - Attestation performance: epoch 162302, expected 594, produced 594, included 594 (100%), distance 1 / 1.30 / 6, correct target 575 (97%), correct head 575 (97%)
2023-03-14 22:25:37.832 INFO - Sync committee performance: epoch 162303, expected 32, produced 32, correct 32, included 22 (69%)
2023-03-14 22:25:40.046 INFO - Slot Event *** Slot: 5193728, Block: ... empty, Justified: 162303, Finalized: 162302, Peers: 82
2023-03-14 22:25:40.127 INFO - Validator *** Published attestation Count: 30, Slot: 5193728, Root: 6a7ac459f234540c9975e4eb98049dd3bde33141246916b9f446b7f98e17af3b
2023-03-14 22:25:44.072 ERROR - Validator *** Produced invalid aggregate for slot 5193728: Rejecting aggregate with invalid batch signature
2023-03-14 22:25:44.074 ERROR - Validator *** Produced invalid aggregate for slot 5193728: Rejecting aggregate with invalid batch signature
2023-03-14 22:25:44.077 ERROR - Validator *** Produced invalid aggregate for slot 5193728: Rejecting aggregate with invalid batch signature
2023-03-14 22:25:44.080 ERROR - Validator *** Produced invalid aggregate for slot 5193728: Rejecting aggregate with invalid batch signature
2023-03-14 22:25:44.083 ERROR - Validator *** Failed to produce aggregate Slot: 5193728 Validator: 88040ea, ab67356, 92b4615, a37e059
tech.pegasys.teku.validator.client.duties.RestApiReportedException: Rejecting aggregate with invalid batch signature
2023-03-14 22:25:52.155 INFO - Slot Event *** Slot: 5193729, Block: ... empty, Justified: 162303, Finalized: 162302, Peers: 76
2023-03-14 22:25:54.120 INFO - Validator *** Published attestation Count: 17, Slot: 5193729, Root: 6a7ac459f234540c9975e4eb98049dd3bde33141246916b9f446b7f98e17af3b
2023-03-14 22:25:58.753 ERROR - Validator *** Produced invalid aggregate for slot 5193729: Rejecting aggregate with invalid batch signature
2023-03-14 22:25:58.754 ERROR - Validator *** Failed to produce aggregate Slot: 5193729 Validator: a349d56
tech.pegasys.teku.validator.client.duties.RestApiReportedException: Rejecting aggregate with invalid batch signature
2023-03-14 22:26:04.091 INFO - Slot Event *** Slot: 5193730, Block: ... empty, Justified: 162303, Finalized: 162302, Peers: 75
2023-03-14 22:26:04.690 WARN - Discarding gossip message for topic /eth2/628941ef/beacon_attestation_38/ssz_snappy because the signature verification queue is full
2023-03-14 22:26:04.692 WARN - Discarding gossip message for topic /eth2/628941ef/beacon_attestation_41/ssz_snappy because the signature verification queue is full
...
...
2023-03-14 22:26:09.823 WARN - Discarding gossip message for topic /eth2/628941ef/beacon_attestation_33/ssz_snappy because the signature verification queue is full
2023-03-14 22:26:09.823 WARN - Discarding gossip message for topic /eth2/628941ef/beacon_attestation_15/ssz_snappy because the signature verification queue is full
2023-03-14 22:26:16.137 INFO - Slot Event *** Slot: 5193731, Block: ... empty, Justified: 162303, Finalized: 162302, Peers: 74
2023-03-14 22:26:19.394 WARN - Late Block Import *** Block: 8c2fced42f67c754b310dd99c401b7894675122faaa7f4872b904872d98c08de (5193731) proposer 178565 arrival 1242ms, pre-state_retrieved +97ms, completed +6055ms
2023-03-14 22:26:20.000 WARN - Validator *** Skipped aggregation for committee 27 at slot 5193731 because there was nothing to aggregate
2023-03-14 22:26:20.866 INFO - Validator *** Published attestation Count: 17, Slot: 5193730, Root: 6a7ac459f234540c9975e4eb98049dd3bde33141246916b9f446b7f98e17af3b
2023-03-14 22:26:28.156 INFO - Slot Event *** Slot: 5193732, Block: ... empty, Justified: 162303, Finalized: 162302, Peers: 70
2023-03-14 22:26:29.096 WARN - Late Block Import *** Block: 795589c30104e470e22f1f57aff0e4afeee7c61e558065d1a51047c99e29c831 (5193732) proposer 225583 arrival 3519ms, pre-state_retrieved +7ms, processed +1260ms, execution_payload_result_received +0ms, begin_importing +0ms, transaction_prepared +280ms, transaction_committed +8ms, completed +22ms
2023-03-14 22:26:31.046 INFO - Validator *** Published attestation Count: 20, Slot: 5193731, Root: 6a7ac459f234540c9975e4eb98049dd3bde33141246916b9f446b7f98e17af3b
2023-03-14 22:26:32.000 WARN - Validator *** Skipped aggregation for committee 55 at slot 5193732 because there was nothing to aggregate
2023-03-14 22:26:32.000 WARN - Validator *** Skipped aggregation for committee 41 at slot 5193732 because there was nothing to aggregate
2023-03-14 22:26:32.000 WARN - Validator *** Skipped aggregation for committee 58 at slot 5193732 because there was nothing to aggregate
2023-03-14 22:26:40.111 INFO - Slot Event *** Slot: 5193733, Block: ... empty, Justified: 162303, Finalized: 162302, Peers: 66
2023-03-14 22:26:40.761 INFO - Validator *** Published attestation Count: 23, Slot: 5193732, Root: 6a7ac459f234540c9975e4eb98049dd3bde33141246916b9f446b7f98e17af3b
2023-03-14 22:26:44.001 WARN - Validator *** Skipped aggregation for committee 22 at slot 5193733 because there was nothing to aggregate
2023-03-14 22:26:48.540 INFO - Validator *** Published attestation Count: 15, Slot: 5193733, Root: 795589c30104e470e22f1f57aff0e4afeee7c61e558065d1a51047c99e29c831
2023-03-14 22:26:52.147 INFO - Slot Event *** Slot: 5193734, Block: ... empty, Justified: 162303, Finalized: 162302, Peers: 67
2023-03-14 22:26:54.096 INFO - Validator *** Published attestation Count: 17, Slot: 5193734, Root: 795589c30104e470e22f1f57aff0e4afeee7c61e558065d1a51047c99e29c831
2023-03-14 22:27:04.052 INFO - Slot Event *** Slot: 5193735, Block: ... empty, Justified: 162303, Finalized: 162302, Peers: 66
2023-03-14 22:27:04.101 INFO - Validator *** Published attestation Count: 18, Slot: 5193735, Root: 795589c30104e470e22f1f57aff0e4afeee7c61e558065d1a51047c99e29c831
2023-03-14 22:27:08.024 INFO - Validator *** Published aggregate Count: 1, Slot: 5193735, Root: 795589c30104e470e22f1f57aff0e4afeee7c61e558065d1a51047c99e29c831
2023-03-14 22:27:12.001 INFO - Updating number of persistent subnet subscriptions from 63 to 64
2023-03-14 22:27:15.258 INFO - Validator *** Published attestation Count: 20, Slot: 5193736, Root: 5df720fdb845e8e45feb7ae7fae4f2eee9ade289f0f215437a1e17d3fa47cd2d
besu:
{"@timestamp":"2023-03-14T22:25:13,237","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,121 / 168 tx / base fee 2.89 gwei / 29,984,638 (99.9%) gas / (0x1f120a6ce50aa0f9b4b61f92cc7f62626b7d67d973570d84f586d0e3e05e5f34) in 0.519s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:25:25,559","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,122 / 40 tx / base fee 3.25 gwei / 7,422,801 (24.7%) gas / (0x1469313981cb69286cbfbda5c4e9e02018ca6b091939e99cb888ff3fc5ae8cb7) in 0.172s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:25:40,050","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x1469313981cb69286cbfbda5c4e9e02018ca6b091939e99cb888ff3fc5ae8cb7, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:26:14,424","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,123 / 169 tx / 0 ws / base fee 3.05 gwei / 29,983,935 (99.9%) gas / (0x037a6fc94d8f00597aaa370a830866bbcae141c141d12f94d9992b33398374e4) in 1.011s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:26:28,076","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,123 / 183 tx / 0 ws / base fee 3.05 gwei / 29,992,912 (100.0%) gas / (0x4fed6603f749a4bd18a5e3cd21922327a18b20167d084e9e25a36e0a3ffcb334) in 0.520s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:26:37,752","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,124 / 81 tx / 0 ws / base fee 3.43 gwei / 16,883,969 (56.3%) gas / (0x9fa304882c45848e0c8afe2481d46bd527eee8c9775251de3da263f023f71a46) in 0.491s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:27:01,858","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,124 / 170 tx / 0 ws / base fee 3.43 gwei / 29,983,113 (99.9%) gas / (0x7abefd9f829a42036d8dae1cee29503c421f07984ee424f68307cd92d5848073) in 0.675s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:27:04,055","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x4fed6603f749a4bd18a5e3cd21922327a18b20167d084e9e25a36e0a3ffcb334, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:27:14,557","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,124 / 106 tx / 0 ws / base fee 3.43 gwei / 20,874,337 (69.6%) gas / (0x2a869bf7e4590d37846f2e1ec2a9929ddb781b311892bea3b269742db38f1d29) in 0.382s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:27:24,814","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,125 / 108 tx / 0 ws / base fee 3.60 gwei / 18,193,591 (60.6%) gas / (0x75dc5b98ff2a307e59ac01f18809483094f951ec4422c238ed58da474b9506ab) in 0.334s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:27:50,781","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,126 / 90 tx / 0 ws / base fee 3.69 gwei / 10,967,695 (36.6%) gas / (0xdc7294122cfb975ab5c453b018f43bc433492a9847d5d0adbabf853c8b8b76cb) in 0.347s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:28:01,322","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,126 / 114 tx / 0 ws / base fee 3.69 gwei / 17,497,088 (58.3%) gas / (0x1dac75e651ca983c92cffdc72b328327c0ac781113070608df712b68fe091774) in 0.241s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:28:40,167","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x1dac75e651ca983c92cffdc72b328327c0ac781113070608df712b68fe091774, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:28:49,289","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,127 / 138 tx / 0 ws / base fee 3.77 gwei / 29,017,050 (96.7%) gas / (0x70e6500a270910d7141f8cd910591f4b32856e41ac80bf3457e25b9e68ebec50) in 0.450s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:29:25,749","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,128 / 113 tx / 0 ws / base fee 4.21 gwei / 29,707,641 (99.0%) gas / (0x5ec336fe83f99429cbb442e35386ffe206223c706002a2aee7d2564985fd3c3d) in 0.635s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:29:36,667","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,129 / 42 tx / 0 ws / base fee 4.73 gwei / 9,572,260 (31.9%) gas / (0x3c7de036ee08a2e21e65e8368387729755bfc0559bedad9d31d604136c5fe07c) in 0.227s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:29:48,762","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,130 / 37 tx / 0 ws / base fee 4.51 gwei / 6,312,062 (21.0%) gas / (0xbfbe948f0a7b65af308cfd9688c7525d872fe7f9b039f86ebb05d0f9761152aa) in 0.218s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:29:48,866","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0xbfbe948f0a7b65af308cfd9688c7525d872fe7f9b039f86ebb05d0f9761152aa, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:30:00,449","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,131 / 34 tx / 0 ws / base fee 4.19 gwei / 6,498,664 (21.7%) gas / (0xfabe30ece41a1ac73266bc931f639cef6140453d79d886849fa1a79180830053) in 0.175s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:32,763","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,132 / 88 tx / 0 ws / base fee 3.89 gwei / 18,806,012 (62.7%) gas / (0xf9580ba0c25e6f4a39c55bc645cfffb5595717e34f5e8565f4f62e88d8eeecce) in 0.666s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:37,074","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,133 / 31 tx / 0 ws / base fee 4.01 gwei / 3,497,579 (11.7%) gas / (0x6974758c64fc440ce9fd9d20df2584ccbd6d8f0e3c89a85be17c8cd2f3536d44) in 0.161s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:51,152","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,133 / 73 tx / 0 ws / base fee 4.01 gwei / 12,718,436 (42.4%) gas / (0x444621c5b40a8f8cc213120d30d429ec4d392d7b570b8098e366b51cd58df9ed) in 0.177s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:30:52,064","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineForkchoiceUpdated","message":"VALID for fork-choice-update: head: 0x444621c5b40a8f8cc213120d30d429ec4d392d7b570b8098e366b51cd58df9ed, finalized: 0x7e19765eb1d4b88849287086ce9d0bfe7e88c2e713b85ea42e32a75277593aff, safeBlockHash: 0x7e8cc14b64da5b3d015c85adbba97ce7ad0375a706aa953e1e5f64b988f650d6","throwable":""}
{"@timestamp":"2023-03-14T22:31:14,176","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,134 / 132 tx / 0 ws / base fee 3.63 gwei / 25,184,038 (83.9%) gas / (0xca28843497204658716de4df75d5c36c34a53c570e3639c3ec4cbee77827c11d) in 0.572s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:31:37,097","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,135 / 69 tx / 16 ws / base fee 3.94 gwei / 11,461,885 (38.2%) gas / (0x7c907a20f0ff1de3b40fff5c362561dde9606507cfe9e3617846bb20f2db0599) in 0.342s. Peers: 25","throwable":""}
{"@timestamp":"2023-03-14T22:32:02,101","level":"INFO","thread":"vert.x-worker-thread-0","class":"AbstractEngineNewPayload","message":"Imported #8,656,136 / 61 tx / 16 ws / base fee 3.82 gwei / 11,860,079 (39.5%) gas / (0xfef2e95d2bed7eccc3d1ed9d9e58116d31018467cc95013ea7b683d3ea3c496d) in 0.269s. Peers: 25","throwable":""}
