## Low participation in Hive issue ### Anomalies overview 1. Hive counts participation in another manner, not like us. Hive participation data for 1 epoch: ``` ["7","7","7","7","7","7","7","7", "7","0","7","7","7","7","7","7", "7","7","7","7","7","7","7","7", "7","7","7","7","0","7","7","7", "7","0","7","7","7","7","7","7", "7","7","7","7","7","7","7","7", "7","7","7","0","7","7","7","7", "7","7","7","7","7","7","7","7"] ``` While our data shows 2 non-7 misses in participation of the same epoch: 9 and 33. Hive counts 2 more gaps. - Clarified: Hive always misses 1 slot attestations in calculation, so maximum participation is 0.97, though requirement is 0.95, so there is some area for client to miss. So when we get 0.97, it's not an our issue of missing 0.03 2. ~~4 seconds delay in 6 seconds slot between aggregation and attestation~~. All ok, Early attestation feature 3. Failed to publish attestation on gossip topic error while no disconnections were detected - Clarified: subscription to the topic was initiated later by another peer, submitted PR https://github.com/ethereum/hive/pull/653 4. Aggregations were not gossiped, clarified, Adrian made fix: https://github.com/ConsenSys/teku/pull/6276 5. There is 1 known case when locally produced aggregation was not used on the next and subsequent slots when producing blocks - investigating --- ### 1 detailed case Participation by Hive view: ``` ["7","7","7","0","7","7","7","7", "7","3","7","7","0","7","7","7", "7","7","7","7","7","7","7","7", "7","7","7","7","3","7","7","7", "7","7","7","7","0","7","7","7", "0","7","7","7","7","7","7","7", "7","7","7","7","7","7","7","7", "7","7","7","7","7","7","7","7"] ``` Aggregate inclusions in block by slot: ``` slot: vc-2ca2,vc-bc61 64: ,2 65: 0, 66: 2, 67: ,4 (found from 64) 68: ,2 69: ,2 70: ,2 71: ,2 72: ,2 73: ,2 74: ,2 75: 2, 76: 2, 77: ,2 78: ,2 79: ,2 80: 2, 81: 2, 82: 2, 83: 2, 84: ,2 85: 2, 86: 2, 87: 2, 88: 2, 89: 2, 90: ,2 91: ,0 (lost forever) 92: ,2 93: ,2 94: ,2 95: ,2 96: 0, 97: ,4 (found from 95) ``` #### 64th slot attestation **vc-bc61** (connected to **bn-d7fc**): ``` 2022-09-29 18:44:45.147 INFO - Signing attestation in slot 64 with data AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}} for validator a5a3303 2022-09-29 18:44:45.161 INFO - Signing attestation in slot 64 with data AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}} for validator 89ff903 2022-09-29 18:44:45.194 INFO - Validator *** Published attestation Count: 2, Slot: 64, Root: c97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125 2022-09-29 18:44:49.006 INFO - Attestation aggregate bits: SszBitlist{size=2, 11} 2022-09-29 18:44:49.086 INFO - Validator *** Published aggregate Count: 1, Slot: 64, Root: c97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125 ``` later ``` 2022-09-29 18:45:03.023 INFO - Block attestations: Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=66, index=0, beacon_block_root=0x507fdfe9585c224af9c85164ca51f820a21ab36fa1175907de4b08aec54b4575, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8294877f3531af27106713eb0d233b2b845878e5b140aa398ea1aa53d3b76ee03c9bfd33deb3a11aea7e6424a52ba9c60079fa25614b735fea083c6e3e3bd184e86a0010b2b6162dd0894bb70d75664cb16738a1527e7306755e0a56b79b84ea}},Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x865b80bd99a9d90f180a0010f87c77384c504046bccb9f1f7b3785574c1cc807fb57a571263eddd739f74542c23c82550609d74b6921950dd9e70a7fa799f3ee0a37e986f3da4c166e1c2bcc6b93059a91d8cc72e481131738d12b4a1966f7f0}} 2022-09-29 18:45:03.047 INFO - Validator *** Published block Count: 1, Slot: 67, Root: b82d1bdd0f18423f70a831ea704b456f109e923b194c871691dd2d1a8f3ba9f9, 0 (0%) gas, EL block: f9dbfcac96c758f9bac8e44cc2cef4846ccccb2f64afb8fe906583ae9db88e3e (105) ``` **vc-2ca2** (connected to **bn-defe**): ``` 2022-09-29 18:44:51.007 INFO - Block attestations: 2022-09-29 18:44:51.041 INFO - Validator *** Published block Count: 1, Slot: 65, Root: 77cafc3d58c328ba9d0eb321c9dacae3ea6edde3d6f810be8e1cf9a3fb21dde2, 0 (0%) gas, EL block: a8b2e46eddb0c361bd7da04124db4d3237e5b6b5c37b2030d7901a3e3f99797b (103) ``` **bn-d7fc:** ``` 2022-09-29 18:44:45.192 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@f0daefd3, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 01}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xb8f019837b7f6415d1507b86a7581b4ba3a064ff2c191086c80a307fd06e49ac73c1fdc265adf0d469d9562e298fe0800229c67c51fbe1e10713291cdba86387bfbc2b13413f7c15b3cb7f6477fe2095d19eac6f1846e8650c8b71b805fd66a1}}, maybeAggregate=Optional.empty, hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2412/0x0000000800916908@6efde367), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=1: 28}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xb8f019837b7f6415d1507b86a7581b4ba3a064ff2c191086c80a307fd06e49ac73c1fdc265adf0d469d9562e298fe0800229c67c51fbe1e10713291cdba86387bfbc2b13413f7c15b3cb7f6477fe2095d19eac6f1846e8650c8b71b805fd66a1}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty} 2022-09-29 18:44:45.193 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@f0daefd3, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 10}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xb02ea5b143586d41fb10eba88f4a50430ed17cccfeb4673468eeab8e3188eae1134d1ca10771186c637e62687b1dad9d147e34520100b51c14da9155fac00694102d0a1b6e4834d04cc5bb9897be072e1a53af8bbfd78de460ce61d32c400780}}, maybeAggregate=Optional.empty, hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2412/0x0000000800916908@240e0fdf), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=1: 9}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xb02ea5b143586d41fb10eba88f4a50430ed17cccfeb4673468eeab8e3188eae1134d1ca10771186c637e62687b1dad9d147e34520100b51c14da9155fac00694102d0a1b6e4834d04cc5bb9897be072e1a53af8bbfd78de460ce61d32c400780}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty} 2022-09-29 18:44:49.085 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@f0daefd3, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x865b80bd99a9d90f180a0010f87c77384c504046bccb9f1f7b3785574c1cc807fb57a571263eddd739f74542c23c82550609d74b6921950dd9e70a7fa799f3ee0a37e986f3da4c166e1c2bcc6b93059a91d8cc72e481131738d12b4a1966f7f0}}, maybeAggregate=Optional[SignedAggregateAndProof{message=AggregateAndProof{aggregator_index=28, aggregate=Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x865b80bd99a9d90f180a0010f87c77384c504046bccb9f1f7b3785574c1cc807fb57a571263eddd739f74542c23c82550609d74b6921950dd9e70a7fa799f3ee0a37e986f3da4c166e1c2bcc6b93059a91d8cc72e481131738d12b4a1966f7f0}}, selection_proof=SszByteVector{0xabba52f2908d9667a84f7ea7139f0be5423717db8c8a36148287c2851df14fe3b58a0ffa271f3efc065c62ba1a40531c1858516209ee2159812e05ddc7847a3e6bd2e686de632f3a96f7b265ee581c61d106a71182d84abc06164c449feda59d}}, signature=SszByteVector{0xb5f312a9ee25d560cf3be3f0e40d2eccddf030c8b0370b0a322d7e08609a47428327a4e593c878c01b5b0f14f6de8f2d118b15a1f608817f7af44303b3931a8b1fd380065a52d5a71c7314d9ccd34e31ef93fd223299ede036506324472afbf7}}], hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2412/0x0000000800916908@66da7ea9), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=2: 9, 28}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x865b80bd99a9d90f180a0010f87c77384c504046bccb9f1f7b3785574c1cc807fb57a571263eddd739f74542c23c82550609d74b6921950dd9e70a7fa799f3ee0a37e986f3da4c166e1c2bcc6b93059a91d8cc72e481131738d12b4a1966f7f0}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty} ``` **bn-defe**: ``` 2022-09-29 18:44:45.193 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@b67d91ed, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 01}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xb8f019837b7f6415d1507b86a7581b4ba3a064ff2c191086c80a307fd06e49ac73c1fdc265adf0d469d9562e298fe0800229c67c51fbe1e10713291cdba86387bfbc2b13413f7c15b3cb7f6477fe2095d19eac6f1846e8650c8b71b805fd66a1}}, maybeAggregate=Optional.empty, hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2237/0x00000008008ec0c8@253b80fc), gossiped=false, producedLocally=false, isValidIndexedAttestation=false, indexedAttestation=Optional.empty, committeeShufflingSeed=Optional.empty, receivedSubnetId=OptionalInt[0]} 2022-09-29 18:44:45.194 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@b67d91ed, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 10}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xb02ea5b143586d41fb10eba88f4a50430ed17cccfeb4673468eeab8e3188eae1134d1ca10771186c637e62687b1dad9d147e34520100b51c14da9155fac00694102d0a1b6e4834d04cc5bb9897be072e1a53af8bbfd78de460ce61d32c400780}}, maybeAggregate=Optional.empty, hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2237/0x00000008008ec0c8@109ba294), gossiped=false, producedLocally=false, isValidIndexedAttestation=false, indexedAttestation=Optional.empty, committeeShufflingSeed=Optional.empty, receivedSubnetId=OptionalInt[0]} ``` **Summary**: Aggregate attestation was not received by other peer by unknown reason, but was later included in block when the same vc was chosen as the next block producer. I should find the reason why aggegate attestations are lost in such cases. #### 90th slot attestation **vc-2ca2** (connected to **bn-defe**): ``` 2022-09-29 18:47:21.348 INFO - Signing attestation in slot 90 with data AttestationData{slot=90, index=0, beacon_block_root=0x1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}} for validator 8429e93 2022-09-29 18:47:21.374 INFO - Validator *** Published attestation Count: 2, Slot: 90, Root: 1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004 2022-09-29 18:47:25.007 INFO - Attestation aggregate bits: SszBitlist{size=2, 11} 2022-09-29 18:47:25.089 INFO - Validator *** Published aggregate Count: 1, Slot: 90, Root: 1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004 ``` **vc-bc61** (connected to **bn-d7fc**): ``` 2022-09-29 18:47:27.024 INFO - Block attestations: [] 2022-09-29 18:47:27.047 INFO - Validator *** Published block Count: 1, Slot: 91, Root: b332f3a89cd6ca56e343c5e13f8df105e8cd3c16be9a68ab331485d9e0bbc956, 0 (0%) gas, EL block: c56739dea5f94d26cb1739dc8b8a3625d3957d7d2a02380de158ebeb4324d3a9 (129) ``` **bn-defe**: ``` 2022-09-29 18:47:21.374 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@b67d91ed, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 10}, data=AttestationData{slot=90, index=0, beacon_block_root=0x1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xaf56c2a438d7ddd495df71ac77b9235f94b650d6a349321b98f6f59a47a73cee27d1f47a359b72a971e8058e9a2e40ee03aa29aa370a13f6949e6afe36c47924acd8d0d8e2b4452c63884d3b62b57549479310be6037ed26e28d4e0db0dc0583}}, maybeAggregate=Optional.empty, hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2237/0x00000008008ec0c8@de90bf8), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=1: 36}, data=AttestationData{slot=90, index=0, beacon_block_root=0x1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xaf56c2a438d7ddd495df71ac77b9235f94b650d6a349321b98f6f59a47a73cee27d1f47a359b72a971e8058e9a2e40ee03aa29aa370a13f6949e6afe36c47924acd8d0d8e2b4452c63884d3b62b57549479310be6037ed26e28d4e0db0dc0583}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty} 2022-09-29 18:47:21.374 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@b67d91ed, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 01}, data=AttestationData{slot=90, index=0, beacon_block_root=0x1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xa1a097df43dc83675037fafff9ba9d88dfc16efc35deca49889dfef69b9383ce6a61a1fdaa91c973a1805214ca7c99b400b9d3d5ec8aac844a1486bd4d3df377703817190a154db85675cf3de2565117c190bad0493563199b94b3c795a15b87}}, maybeAggregate=Optional.empty, hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2237/0x00000008008ec0c8@17ed901f), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=1: 40}, data=AttestationData{slot=90, index=0, beacon_block_root=0x1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xa1a097df43dc83675037fafff9ba9d88dfc16efc35deca49889dfef69b9383ce6a61a1fdaa91c973a1805214ca7c99b400b9d3d5ec8aac844a1486bd4d3df377703817190a154db85675cf3de2565117c190bad0493563199b94b3c795a15b87}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty} 2022-09-29 18:47:21.374 WARN - Failed to publish attestations(s) for slot 90 because no peers were available on the required gossip topic 2022-09-29 18:47:23.312 INFO - Slot Event *** Slot: 90, Block: 1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, Justified: 0, Finalized: 0, Peers: 1 2022-09-29 18:47:25.089 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@b67d91ed, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=90, index=0, beacon_block_root=0x1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x92d959bfc94228347dd791ed821c037d409a24f0dac740313cadb3566cdba1326f3a49a5eb409a6ec216393aeeea89ee1566f928d6b9dccf615656debbbbba14e4e7aa670ccbd98f9c5eada7724b73e0417f154f396519c04249971295c165c0}}, maybeAggregate=Optional[SignedAggregateAndProof{message=AggregateAndProof{aggregator_index=36, aggregate=Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=90, index=0, beacon_block_root=0x1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x92d959bfc94228347dd791ed821c037d409a24f0dac740313cadb3566cdba1326f3a49a5eb409a6ec216393aeeea89ee1566f928d6b9dccf615656debbbbba14e4e7aa670ccbd98f9c5eada7724b73e0417f154f396519c04249971295c165c0}}, selection_proof=SszByteVector{0xad067c9ee3e82ee31d827a356977cef8eaab644050e72d0319d77a1364dfaed55b5a2d2991a93f2e7a95aa777730a2dc0ddc60eea5ca0dfc99cbb7b65069f9fad11f352fe5f8ef076e697006236da92ec986ee50b851817f0aacbc4a2c9881a4}}, signature=SszByteVector{0xa12b39f1e77857e58fa9a900bf11e7a65c93edc599d52adb6a90e54fb26b9fba13f723ba0dbe627871aff24cb5cbbaff14a6aba43ade89ccbbf3982ab433782b2a3695c79a6a797d0369711953b45516ffd14f75f53d45512f7579975e62577b}}], hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2237/0x00000008008ec0c8@551a953a), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=2: 36, 40}, data=AttestationData{slot=90, index=0, beacon_block_root=0x1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x92d959bfc94228347dd791ed821c037d409a24f0dac740313cadb3566cdba1326f3a49a5eb409a6ec216393aeeea89ee1566f928d6b9dccf615656debbbbba14e4e7aa670ccbd98f9c5eada7724b73e0417f154f396519c04249971295c165c0}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty} ``` **bn-d7fc**: nothing interesting **Summary**: Single attestations was not gossiped with error reason "no peers were available on the required gossip topic". Aggregate attestation was produced but was not later included in block when same vc was chosen as the next block producer. I should find the reason why there were no peers on this topic, why gossip was not replayed later. I should find the reason why being later a block producer vc didn't include this attestation in block. #### 95th slot attestation **vc-bc61** (connected to **bn-d7fc**): ``` 2022-09-29 18:47:51.152 INFO - Signing attestation in slot 95 with data AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}} for validator 8dcd8d9 2022-09-29 18:47:51.162 INFO - Signing attestation in slot 95 with data AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}} for validator 8ffa99a 2022-09-29 18:47:51.206 INFO - Validator *** Published attestation Count: 2, Slot: 95, Root: 3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b 2022-09-29 18:47:55.005 INFO - Attestation aggregate bits: SszBitlist{size=2, 11} 2022-09-29 18:47:55.080 INFO - Validator *** Published aggregate Count: 1, Slot: 95, Root: 3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b ``` later ``` 2022-09-29 18:48:03.017 INFO - Block attestations: Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=96, index=0, beacon_block_root=0x96c59d7bdf7e0009d0954491c563243a89ec144c73ab356c713810f0268120d3, source=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}, target=Checkpoint{epoch=3, root=0x96c59d7bdf7e0009d0954491c563243a89ec144c73ab356c713810f0268120d3}}, signature=SszByteVector{0xb77809f622334b7b1f9b17bfbe16a964ca9e453d450f92c080f73952fade5801d5ff703f745cb085f4cf81a0f2acb47a11f8d72f06dd7ac55f90908a81ccc0782efabf9ee5c82990f514cedaa244a34aef77a6236ec79dbf8b8d8c61c1b9cd34}},Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8f93f39c11345f89a3a72f5656398aa864077daaaacad0a5f3c3cd2f2ba83851d395067a9db6b2033f50107bf3a36cf0114209c5f15888da506395d7de18ca3ff204d36dd4d012a0ee827e99e47b154c1dd6260008cfd0f8fc6a2a1f1ecddbb5}} 2022-09-29 18:48:03.029 INFO - Validator *** Published block Count: 1, Slot: 97, Root: daf49fbc69fb755d1e0f806e8692877c49670e4bd21a6a2a7d0c41217ea4faf6, 0 (0%) gas, EL block: 49129590c64cc96b4493ee392dd07f3eba9c6c2fc1d5d77b8ab3069844aa5834 (135) ``` **vc-2ca2** (connected to **bn-defe**): ``` 2022-09-29 18:47:57.015 INFO - Block attestations: [] 2022-09-29 18:47:57.035 INFO - Validator *** Published block Count: 1, Slot: 96, Root: 96c59d7bdf7e0009d0954491c563243a89ec144c73ab356c713810f0268120d3, 0 (0%) gas, EL block: 840ccd1d13f7b76ce7facb612e00e350978cd895e3b1e1c359409c265d010bc6 (134) ``` **bn-d7fc**: ``` 2022-09-29 18:47:51.201 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@f0daefd3, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 10}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x87fd632118a45446a1fa94960a359e9b210ca63283e866ec80b6259c9df5bfb4d3a2ebe3573c00050db96fb0ea7c5caf0a34d6eaedb7c80f999ba463873ff330bacfa7373518c3fdfd25175f96abb49bd33b1a5caa37a843ef2861c135986dab}}, maybeAggregate=Optional.empty, hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2412/0x0000000800916908@1d22a0c3), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=1: 3}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x87fd632118a45446a1fa94960a359e9b210ca63283e866ec80b6259c9df5bfb4d3a2ebe3573c00050db96fb0ea7c5caf0a34d6eaedb7c80f999ba463873ff330bacfa7373518c3fdfd25175f96abb49bd33b1a5caa37a843ef2861c135986dab}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty} 2022-09-29 18:47:51.201 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@f0daefd3, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 01}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8012aa468fbcb413e7280aa844f416144e4154d4e14a136903e75e41e1914a2ea4375ec1f95005ec1712f6c812dde5c715c0e73def919b5fb0dee5e1496a9c5dbb30fb92b1560d561889546a13ddbda1832d50f27f0d14d0f9331b5c1d74634e}}, maybeAggregate=Optional.empty, hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2412/0x0000000800916908@38866cb7), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=1: 12}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8012aa468fbcb413e7280aa844f416144e4154d4e14a136903e75e41e1914a2ea4375ec1f95005ec1712f6c812dde5c715c0e73def919b5fb0dee5e1496a9c5dbb30fb92b1560d561889546a13ddbda1832d50f27f0d14d0f9331b5c1d74634e}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty} 2022-09-29 18:47:51.202 WARN - Failed to publish attestations(s) for slot 95 because no peers were available on the required gossip topic 2022-09-29 18:47:53.134 INFO - Slot Event *** Slot: 95, Block: 3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, Justified: 0, Finalized: 0, Peers: 1 2022-09-29 18:47:55.080 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@f0daefd3, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8f93f39c11345f89a3a72f5656398aa864077daaaacad0a5f3c3cd2f2ba83851d395067a9db6b2033f50107bf3a36cf0114209c5f15888da506395d7de18ca3ff204d36dd4d012a0ee827e99e47b154c1dd6260008cfd0f8fc6a2a1f1ecddbb5}}, maybeAggregate=Optional[SignedAggregateAndProof{message=AggregateAndProof{aggregator_index=12, aggregate=Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8f93f39c11345f89a3a72f5656398aa864077daaaacad0a5f3c3cd2f2ba83851d395067a9db6b2033f50107bf3a36cf0114209c5f15888da506395d7de18ca3ff204d36dd4d012a0ee827e99e47b154c1dd6260008cfd0f8fc6a2a1f1ecddbb5}}, selection_proof=SszByteVector{0xaaaedcd09ab741d6521af325dcefc019a1e8e917bc60bcf6f8318697eeafe5274330819f1a7781de1e264f2584c6404812776a54bb0f19427a3da6d0c6ae07947b3cba40241623c10af19b0f19e6a0e4e2ff34c141651c496de97f3a444076d1}}, signature=SszByteVector{0xaecf3fe0854d8977335b26b254ec0cffd96dfe2a3fef7b882dc5cbfa8ab855e6c04894074ab2d15687dd665d98f84489116c01f33157bf0859337b441b481fc5d808fc124639f9dee434f4cb0b263988d49998d494cc7c870ff9c9aa0f1c2b25}}], hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2412/0x0000000800916908@53760002), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=2: 3, 12}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8f93f39c11345f89a3a72f5656398aa864077daaaacad0a5f3c3cd2f2ba83851d395067a9db6b2033f50107bf3a36cf0114209c5f15888da506395d7de18ca3ff204d36dd4d012a0ee827e99e47b154c1dd6260008cfd0f8fc6a2a1f1ecddbb5}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty} ``` later ``` 2022-09-29 18:48:03.015 INFO - Processing attestation Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8f93f39c11345f89a3a72f5656398aa864077daaaacad0a5f3c3cd2f2ba83851d395067a9db6b2033f50107bf3a36cf0114209c5f15888da506395d7de18ca3ff204d36dd4d012a0ee827e99e47b154c1dd6260008cfd0f8fc6a2a1f1ecddbb5}} 2022-09-29 18:48:03.015 INFO - Setting flag 3 for index 3 from attestation Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8f93f39c11345f89a3a72f5656398aa864077daaaacad0a5f3c3cd2f2ba83851d395067a9db6b2033f50107bf3a36cf0114209c5f15888da506395d7de18ca3ff204d36dd4d012a0ee827e99e47b154c1dd6260008cfd0f8fc6a2a1f1ecddbb5}} 2022-09-29 18:48:03.015 INFO - Processing attestation Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8f93f39c11345f89a3a72f5656398aa864077daaaacad0a5f3c3cd2f2ba83851d395067a9db6b2033f50107bf3a36cf0114209c5f15888da506395d7de18ca3ff204d36dd4d012a0ee827e99e47b154c1dd6260008cfd0f8fc6a2a1f1ecddbb5}} 2022-09-29 18:48:03.015 INFO - Setting flag 3 for index 12 from attestation Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8f93f39c11345f89a3a72f5656398aa864077daaaacad0a5f3c3cd2f2ba83851d395067a9db6b2033f50107bf3a36cf0114209c5f15888da506395d7de18ca3ff204d36dd4d012a0ee827e99e47b154c1dd6260008cfd0f8fc6a2a1f1ecddbb5}} ``` **bn-defe**: nothing interesting **Summary**: Single attestations was not gossiped with error reason "no peers were available on the required gossip topic". Aggregate attestion was produced and was later included in block when the same vc was chosen as next block producer. I should find the reason why there were no peers on this topic, why gossip was not replayed later.