["7","7","7","7","7","7","7","7",
"7","0","7","7","7","7","7","7",
"7","7","7","7","7","7","7","7",
"7","7","7","7","0","7","7","7",
"7","0","7","7","7","7","7","7",
"7","7","7","7","7","7","7","7",
"7","7","7","0","7","7","7","7",
"7","7","7","7","7","7","7","7"]
While our data shows 2 non-7 misses in participation of the same epoch: 9 and 33. Hive counts 2 more gaps. - Clarified: Hive always misses 1 slot attestations in calculation, so maximum participation is 0.97, though requirement is 0.95, so there is some area for client to miss. So when we get 0.97, it's not an our issue of missing 0.03
2. 4 seconds delay in 6 seconds slot between aggregation and attestation. All ok, Early attestation feature
3. Failed to publish attestation on gossip topic error while no disconnections were detected - Clarified: subscription to the topic was initiated later by another peer, submitted PR https://github.com/ethereum/hive/pull/653
4. Aggregations were not gossiped, clarified, Adrian made fix: https://github.com/ConsenSys/teku/pull/6276
5. There is 1 known case when locally produced aggregation was not used on the next and subsequent slots when producing blocks - investigating
Participation by Hive view:
["7","7","7","0","7","7","7","7",
"7","3","7","7","0","7","7","7",
"7","7","7","7","7","7","7","7",
"7","7","7","7","3","7","7","7",
"7","7","7","7","0","7","7","7",
"0","7","7","7","7","7","7","7",
"7","7","7","7","7","7","7","7",
"7","7","7","7","7","7","7","7"]
Aggregate inclusions in block by slot:
slot: vc-2ca2,vc-bc61
64: ,2
65: 0,
66: 2,
67: ,4 (found from 64)
68: ,2
69: ,2
70: ,2
71: ,2
72: ,2
73: ,2
74: ,2
75: 2,
76: 2,
77: ,2
78: ,2
79: ,2
80: 2,
81: 2,
82: 2,
83: 2,
84: ,2
85: 2,
86: 2,
87: 2,
88: 2,
89: 2,
90: ,2
91: ,0 (lost forever)
92: ,2
93: ,2
94: ,2
95: ,2
96: 0,
97: ,4 (found from 95)
vc-bc61 (connected to bn-d7fc):
2022-09-29 18:44:45.147 INFO - Signing attestation in slot 64 with data AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}} for validator a5a3303
2022-09-29 18:44:45.161 INFO - Signing attestation in slot 64 with data AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}} for validator 89ff903
2022-09-29 18:44:45.194 INFO - [34mValidator *** Published attestation Count: 2, Slot: 64, Root: c97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125[0m
2022-09-29 18:44:49.006 INFO - Attestation aggregate bits: SszBitlist{size=2, 11}
2022-09-29 18:44:49.086 INFO - [34mValidator *** Published aggregate Count: 1, Slot: 64, Root: c97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125[0m
later
2022-09-29 18:45:03.023 INFO - Block attestations: Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=66, index=0, beacon_block_root=0x507fdfe9585c224af9c85164ca51f820a21ab36fa1175907de4b08aec54b4575, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8294877f3531af27106713eb0d233b2b845878e5b140aa398ea1aa53d3b76ee03c9bfd33deb3a11aea7e6424a52ba9c60079fa25614b735fea083c6e3e3bd184e86a0010b2b6162dd0894bb70d75664cb16738a1527e7306755e0a56b79b84ea}},Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x865b80bd99a9d90f180a0010f87c77384c504046bccb9f1f7b3785574c1cc807fb57a571263eddd739f74542c23c82550609d74b6921950dd9e70a7fa799f3ee0a37e986f3da4c166e1c2bcc6b93059a91d8cc72e481131738d12b4a1966f7f0}}
2022-09-29 18:45:03.047 INFO - [34mValidator *** Published block Count: 1, Slot: 67, Root: b82d1bdd0f18423f70a831ea704b456f109e923b194c871691dd2d1a8f3ba9f9, 0 (0%) gas, EL block: f9dbfcac96c758f9bac8e44cc2cef4846ccccb2f64afb8fe906583ae9db88e3e (105)[0m
vc-2ca2 (connected to bn-defe):
2022-09-29 18:44:51.007 INFO - Block attestations:
2022-09-29 18:44:51.041 INFO - [34mValidator *** Published block Count: 1, Slot: 65, Root: 77cafc3d58c328ba9d0eb321c9dacae3ea6edde3d6f810be8e1cf9a3fb21dde2, 0 (0%) gas, EL block: a8b2e46eddb0c361bd7da04124db4d3237e5b6b5c37b2030d7901a3e3f99797b (103)[0m
bn-d7fc:
2022-09-29 18:44:45.192 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@f0daefd3, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 01}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xb8f019837b7f6415d1507b86a7581b4ba3a064ff2c191086c80a307fd06e49ac73c1fdc265adf0d469d9562e298fe0800229c67c51fbe1e10713291cdba86387bfbc2b13413f7c15b3cb7f6477fe2095d19eac6f1846e8650c8b71b805fd66a1}}, maybeAggregate=Optional.empty, hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2412/0x0000000800916908@6efde367), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=1: 28}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xb8f019837b7f6415d1507b86a7581b4ba3a064ff2c191086c80a307fd06e49ac73c1fdc265adf0d469d9562e298fe0800229c67c51fbe1e10713291cdba86387bfbc2b13413f7c15b3cb7f6477fe2095d19eac6f1846e8650c8b71b805fd66a1}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty}
2022-09-29 18:44:45.193 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@f0daefd3, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 10}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xb02ea5b143586d41fb10eba88f4a50430ed17cccfeb4673468eeab8e3188eae1134d1ca10771186c637e62687b1dad9d147e34520100b51c14da9155fac00694102d0a1b6e4834d04cc5bb9897be072e1a53af8bbfd78de460ce61d32c400780}}, maybeAggregate=Optional.empty, hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2412/0x0000000800916908@240e0fdf), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=1: 9}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xb02ea5b143586d41fb10eba88f4a50430ed17cccfeb4673468eeab8e3188eae1134d1ca10771186c637e62687b1dad9d147e34520100b51c14da9155fac00694102d0a1b6e4834d04cc5bb9897be072e1a53af8bbfd78de460ce61d32c400780}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty}
2022-09-29 18:44:49.085 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@f0daefd3, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x865b80bd99a9d90f180a0010f87c77384c504046bccb9f1f7b3785574c1cc807fb57a571263eddd739f74542c23c82550609d74b6921950dd9e70a7fa799f3ee0a37e986f3da4c166e1c2bcc6b93059a91d8cc72e481131738d12b4a1966f7f0}}, maybeAggregate=Optional[SignedAggregateAndProof{message=AggregateAndProof{aggregator_index=28, aggregate=Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x865b80bd99a9d90f180a0010f87c77384c504046bccb9f1f7b3785574c1cc807fb57a571263eddd739f74542c23c82550609d74b6921950dd9e70a7fa799f3ee0a37e986f3da4c166e1c2bcc6b93059a91d8cc72e481131738d12b4a1966f7f0}}, selection_proof=SszByteVector{0xabba52f2908d9667a84f7ea7139f0be5423717db8c8a36148287c2851df14fe3b58a0ffa271f3efc065c62ba1a40531c1858516209ee2159812e05ddc7847a3e6bd2e686de632f3a96f7b265ee581c61d106a71182d84abc06164c449feda59d}}, signature=SszByteVector{0xb5f312a9ee25d560cf3be3f0e40d2eccddf030c8b0370b0a322d7e08609a47428327a4e593c878c01b5b0f14f6de8f2d118b15a1f608817f7af44303b3931a8b1fd380065a52d5a71c7314d9ccd34e31ef93fd223299ede036506324472afbf7}}], hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2412/0x0000000800916908@66da7ea9), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=2: 9, 28}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x865b80bd99a9d90f180a0010f87c77384c504046bccb9f1f7b3785574c1cc807fb57a571263eddd739f74542c23c82550609d74b6921950dd9e70a7fa799f3ee0a37e986f3da4c166e1c2bcc6b93059a91d8cc72e481131738d12b4a1966f7f0}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty}
bn-defe:
2022-09-29 18:44:45.193 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@b67d91ed, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 01}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xb8f019837b7f6415d1507b86a7581b4ba3a064ff2c191086c80a307fd06e49ac73c1fdc265adf0d469d9562e298fe0800229c67c51fbe1e10713291cdba86387bfbc2b13413f7c15b3cb7f6477fe2095d19eac6f1846e8650c8b71b805fd66a1}}, maybeAggregate=Optional.empty, hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2237/0x00000008008ec0c8@253b80fc), gossiped=false, producedLocally=false, isValidIndexedAttestation=false, indexedAttestation=Optional.empty, committeeShufflingSeed=Optional.empty, receivedSubnetId=OptionalInt[0]}
2022-09-29 18:44:45.194 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@b67d91ed, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 10}, data=AttestationData{slot=64, index=0, beacon_block_root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xb02ea5b143586d41fb10eba88f4a50430ed17cccfeb4673468eeab8e3188eae1134d1ca10771186c637e62687b1dad9d147e34520100b51c14da9155fac00694102d0a1b6e4834d04cc5bb9897be072e1a53af8bbfd78de460ce61d32c400780}}, maybeAggregate=Optional.empty, hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2237/0x00000008008ec0c8@109ba294), gossiped=false, producedLocally=false, isValidIndexedAttestation=false, indexedAttestation=Optional.empty, committeeShufflingSeed=Optional.empty, receivedSubnetId=OptionalInt[0]}
Summary: Aggregate attestation was not received by other peer by unknown reason, but was later included in block when the same vc was chosen as the next block producer.
I should find the reason why aggegate attestations are lost in such cases.
vc-2ca2 (connected to bn-defe):
2022-09-29 18:47:21.348 INFO - Signing attestation in slot 90 with data AttestationData{slot=90, index=0, beacon_block_root=0x1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}} for validator 8429e93
2022-09-29 18:47:21.374 INFO - [34mValidator *** Published attestation Count: 2, Slot: 90, Root: 1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004[0m
2022-09-29 18:47:25.007 INFO - Attestation aggregate bits: SszBitlist{size=2, 11}
2022-09-29 18:47:25.089 INFO - [34mValidator *** Published aggregate Count: 1, Slot: 90, Root: 1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004[0m
vc-bc61 (connected to bn-d7fc):
2022-09-29 18:47:27.024 INFO - Block attestations: []
2022-09-29 18:47:27.047 INFO - [34mValidator *** Published block Count: 1, Slot: 91, Root: b332f3a89cd6ca56e343c5e13f8df105e8cd3c16be9a68ab331485d9e0bbc956, 0 (0%) gas, EL block: c56739dea5f94d26cb1739dc8b8a3625d3957d7d2a02380de158ebeb4324d3a9 (129)[0m
bn-defe:
2022-09-29 18:47:21.374 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@b67d91ed, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 10}, data=AttestationData{slot=90, index=0, beacon_block_root=0x1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xaf56c2a438d7ddd495df71ac77b9235f94b650d6a349321b98f6f59a47a73cee27d1f47a359b72a971e8058e9a2e40ee03aa29aa370a13f6949e6afe36c47924acd8d0d8e2b4452c63884d3b62b57549479310be6037ed26e28d4e0db0dc0583}}, maybeAggregate=Optional.empty, hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2237/0x00000008008ec0c8@de90bf8), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=1: 36}, data=AttestationData{slot=90, index=0, beacon_block_root=0x1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xaf56c2a438d7ddd495df71ac77b9235f94b650d6a349321b98f6f59a47a73cee27d1f47a359b72a971e8058e9a2e40ee03aa29aa370a13f6949e6afe36c47924acd8d0d8e2b4452c63884d3b62b57549479310be6037ed26e28d4e0db0dc0583}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty}
2022-09-29 18:47:21.374 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@b67d91ed, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 01}, data=AttestationData{slot=90, index=0, beacon_block_root=0x1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xa1a097df43dc83675037fafff9ba9d88dfc16efc35deca49889dfef69b9383ce6a61a1fdaa91c973a1805214ca7c99b400b9d3d5ec8aac844a1486bd4d3df377703817190a154db85675cf3de2565117c190bad0493563199b94b3c795a15b87}}, maybeAggregate=Optional.empty, hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2237/0x00000008008ec0c8@17ed901f), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=1: 40}, data=AttestationData{slot=90, index=0, beacon_block_root=0x1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0xa1a097df43dc83675037fafff9ba9d88dfc16efc35deca49889dfef69b9383ce6a61a1fdaa91c973a1805214ca7c99b400b9d3d5ec8aac844a1486bd4d3df377703817190a154db85675cf3de2565117c190bad0493563199b94b3c795a15b87}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty}
2022-09-29 18:47:21.374 WARN - Failed to publish attestations(s) for slot 90 because no peers were available on the required gossip topic
2022-09-29 18:47:23.312 INFO - [37mSlot Event *** Slot: 90, Block: 1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, Justified: 0, Finalized: 0, Peers: 1[0m
2022-09-29 18:47:25.089 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@b67d91ed, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=90, index=0, beacon_block_root=0x1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x92d959bfc94228347dd791ed821c037d409a24f0dac740313cadb3566cdba1326f3a49a5eb409a6ec216393aeeea89ee1566f928d6b9dccf615656debbbbba14e4e7aa670ccbd98f9c5eada7724b73e0417f154f396519c04249971295c165c0}}, maybeAggregate=Optional[SignedAggregateAndProof{message=AggregateAndProof{aggregator_index=36, aggregate=Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=90, index=0, beacon_block_root=0x1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x92d959bfc94228347dd791ed821c037d409a24f0dac740313cadb3566cdba1326f3a49a5eb409a6ec216393aeeea89ee1566f928d6b9dccf615656debbbbba14e4e7aa670ccbd98f9c5eada7724b73e0417f154f396519c04249971295c165c0}}, selection_proof=SszByteVector{0xad067c9ee3e82ee31d827a356977cef8eaab644050e72d0319d77a1364dfaed55b5a2d2991a93f2e7a95aa777730a2dc0ddc60eea5ca0dfc99cbb7b65069f9fad11f352fe5f8ef076e697006236da92ec986ee50b851817f0aacbc4a2c9881a4}}, signature=SszByteVector{0xa12b39f1e77857e58fa9a900bf11e7a65c93edc599d52adb6a90e54fb26b9fba13f723ba0dbe627871aff24cb5cbbaff14a6aba43ade89ccbbf3982ab433782b2a3695c79a6a797d0369711953b45516ffd14f75f53d45512f7579975e62577b}}], hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2237/0x00000008008ec0c8@551a953a), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=2: 36, 40}, data=AttestationData{slot=90, index=0, beacon_block_root=0x1b24042514e7c3f1a1e0383d5d50a59d5a2dbf126482da9da404935ff46fc004, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x92d959bfc94228347dd791ed821c037d409a24f0dac740313cadb3566cdba1326f3a49a5eb409a6ec216393aeeea89ee1566f928d6b9dccf615656debbbbba14e4e7aa670ccbd98f9c5eada7724b73e0417f154f396519c04249971295c165c0}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty}
bn-d7fc:
nothing interesting
Summary: Single attestations was not gossiped with error reason "no peers were available on the required gossip topic". Aggregate attestation was produced but was not later included in block when same vc was chosen as the next block producer.
I should find the reason why there were no peers on this topic, why gossip was not replayed later. I should find the reason why being later a block producer vc didn't include this attestation in block.
vc-bc61 (connected to bn-d7fc):
2022-09-29 18:47:51.152 INFO - Signing attestation in slot 95 with data AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}} for validator 8dcd8d9
2022-09-29 18:47:51.162 INFO - Signing attestation in slot 95 with data AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}} for validator 8ffa99a
2022-09-29 18:47:51.206 INFO - [34mValidator *** Published attestation Count: 2, Slot: 95, Root: 3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b[0m
2022-09-29 18:47:55.005 INFO - Attestation aggregate bits: SszBitlist{size=2, 11}
2022-09-29 18:47:55.080 INFO - [34mValidator *** Published aggregate Count: 1, Slot: 95, Root: 3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b[0m
later
2022-09-29 18:48:03.017 INFO - Block attestations: Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=96, index=0, beacon_block_root=0x96c59d7bdf7e0009d0954491c563243a89ec144c73ab356c713810f0268120d3, source=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}, target=Checkpoint{epoch=3, root=0x96c59d7bdf7e0009d0954491c563243a89ec144c73ab356c713810f0268120d3}}, signature=SszByteVector{0xb77809f622334b7b1f9b17bfbe16a964ca9e453d450f92c080f73952fade5801d5ff703f745cb085f4cf81a0f2acb47a11f8d72f06dd7ac55f90908a81ccc0782efabf9ee5c82990f514cedaa244a34aef77a6236ec79dbf8b8d8c61c1b9cd34}},Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8f93f39c11345f89a3a72f5656398aa864077daaaacad0a5f3c3cd2f2ba83851d395067a9db6b2033f50107bf3a36cf0114209c5f15888da506395d7de18ca3ff204d36dd4d012a0ee827e99e47b154c1dd6260008cfd0f8fc6a2a1f1ecddbb5}}
2022-09-29 18:48:03.029 INFO - [34mValidator *** Published block Count: 1, Slot: 97, Root: daf49fbc69fb755d1e0f806e8692877c49670e4bd21a6a2a7d0c41217ea4faf6, 0 (0%) gas, EL block: 49129590c64cc96b4493ee392dd07f3eba9c6c2fc1d5d77b8ab3069844aa5834 (135)[0m
vc-2ca2 (connected to bn-defe):
2022-09-29 18:47:57.015 INFO - Block attestations: []
2022-09-29 18:47:57.035 INFO - [34mValidator *** Published block Count: 1, Slot: 96, Root: 96c59d7bdf7e0009d0954491c563243a89ec144c73ab356c713810f0268120d3, 0 (0%) gas, EL block: 840ccd1d13f7b76ce7facb612e00e350978cd895e3b1e1c359409c265d010bc6 (134)[0m
bn-d7fc:
2022-09-29 18:47:51.201 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@f0daefd3, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 10}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x87fd632118a45446a1fa94960a359e9b210ca63283e866ec80b6259c9df5bfb4d3a2ebe3573c00050db96fb0ea7c5caf0a34d6eaedb7c80f999ba463873ff330bacfa7373518c3fdfd25175f96abb49bd33b1a5caa37a843ef2861c135986dab}}, maybeAggregate=Optional.empty, hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2412/0x0000000800916908@1d22a0c3), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=1: 3}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x87fd632118a45446a1fa94960a359e9b210ca63283e866ec80b6259c9df5bfb4d3a2ebe3573c00050db96fb0ea7c5caf0a34d6eaedb7c80f999ba463873ff330bacfa7373518c3fdfd25175f96abb49bd33b1a5caa37a843ef2861c135986dab}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty}
2022-09-29 18:47:51.201 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@f0daefd3, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 01}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8012aa468fbcb413e7280aa844f416144e4154d4e14a136903e75e41e1914a2ea4375ec1f95005ec1712f6c812dde5c715c0e73def919b5fb0dee5e1496a9c5dbb30fb92b1560d561889546a13ddbda1832d50f27f0d14d0f9331b5c1d74634e}}, maybeAggregate=Optional.empty, hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2412/0x0000000800916908@38866cb7), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=1: 12}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8012aa468fbcb413e7280aa844f416144e4154d4e14a136903e75e41e1914a2ea4375ec1f95005ec1712f6c812dde5c715c0e73def919b5fb0dee5e1496a9c5dbb30fb92b1560d561889546a13ddbda1832d50f27f0d14d0f9331b5c1d74634e}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty}
2022-09-29 18:47:51.202 WARN - Failed to publish attestations(s) for slot 95 because no peers were available on the required gossip topic
2022-09-29 18:47:53.134 INFO - [37mSlot Event *** Slot: 95, Block: 3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, Justified: 0, Finalized: 0, Peers: 1[0m
2022-09-29 18:47:55.080 INFO - onAttestation: ValidateableAttestation{spec=tech.pegasys.teku.spec.Spec@f0daefd3, attestation=Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8f93f39c11345f89a3a72f5656398aa864077daaaacad0a5f3c3cd2f2ba83851d395067a9db6b2033f50107bf3a36cf0114209c5f15888da506395d7de18ca3ff204d36dd4d012a0ee827e99e47b154c1dd6260008cfd0f8fc6a2a1f1ecddbb5}}, maybeAggregate=Optional[SignedAggregateAndProof{message=AggregateAndProof{aggregator_index=12, aggregate=Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8f93f39c11345f89a3a72f5656398aa864077daaaacad0a5f3c3cd2f2ba83851d395067a9db6b2033f50107bf3a36cf0114209c5f15888da506395d7de18ca3ff204d36dd4d012a0ee827e99e47b154c1dd6260008cfd0f8fc6a2a1f1ecddbb5}}, selection_proof=SszByteVector{0xaaaedcd09ab741d6521af325dcefc019a1e8e917bc60bcf6f8318697eeafe5274330819f1a7781de1e264f2584c6404812776a54bb0f19427a3da6d0c6ae07947b3cba40241623c10af19b0f19e6a0e4e2ff34c141651c496de97f3a444076d1}}, signature=SszByteVector{0xaecf3fe0854d8977335b26b254ec0cffd96dfe2a3fef7b882dc5cbfa8ab855e6c04894074ab2d15687dd665d98f84489116c01f33157bf0859337b441b481fc5d808fc124639f9dee434f4cb0b263988d49998d494cc7c870ff9c9aa0f1c2b25}}], hashTreeRoot=Suppliers.memoize(tech.pegasys.teku.spec.datastructures.attestation.ValidateableAttestation$$Lambda$2412/0x0000000800916908@53760002), gossiped=false, producedLocally=true, isValidIndexedAttestation=true, indexedAttestation=Optional[IndexedAttestation{attesting_indices=SszList{size=2: 3, 12}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8f93f39c11345f89a3a72f5656398aa864077daaaacad0a5f3c3cd2f2ba83851d395067a9db6b2033f50107bf3a36cf0114209c5f15888da506395d7de18ca3ff204d36dd4d012a0ee827e99e47b154c1dd6260008cfd0f8fc6a2a1f1ecddbb5}}], committeeShufflingSeed=Optional[0xc9117116154b22040e8a5228bb7ac0eb67968e5ad97f598649806f0c9c9eb3d7], receivedSubnetId=OptionalInt.empty}
later
2022-09-29 18:48:03.015 INFO - Processing attestation Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8f93f39c11345f89a3a72f5656398aa864077daaaacad0a5f3c3cd2f2ba83851d395067a9db6b2033f50107bf3a36cf0114209c5f15888da506395d7de18ca3ff204d36dd4d012a0ee827e99e47b154c1dd6260008cfd0f8fc6a2a1f1ecddbb5}}
2022-09-29 18:48:03.015 INFO - Setting flag 3 for index 3 from attestation Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8f93f39c11345f89a3a72f5656398aa864077daaaacad0a5f3c3cd2f2ba83851d395067a9db6b2033f50107bf3a36cf0114209c5f15888da506395d7de18ca3ff204d36dd4d012a0ee827e99e47b154c1dd6260008cfd0f8fc6a2a1f1ecddbb5}}
2022-09-29 18:48:03.015 INFO - Processing attestation Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8f93f39c11345f89a3a72f5656398aa864077daaaacad0a5f3c3cd2f2ba83851d395067a9db6b2033f50107bf3a36cf0114209c5f15888da506395d7de18ca3ff204d36dd4d012a0ee827e99e47b154c1dd6260008cfd0f8fc6a2a1f1ecddbb5}}
2022-09-29 18:48:03.015 INFO - Setting flag 3 for index 12 from attestation Attestation{aggregation_bits=SszBitlist{size=2, 11}, data=AttestationData{slot=95, index=0, beacon_block_root=0x3018bdffaf4dd0274ee876153152005866520e143b670aff30fe80746b4a791b, source=Checkpoint{epoch=0, root=0x0000000000000000000000000000000000000000000000000000000000000000}, target=Checkpoint{epoch=2, root=0xc97c30a280902b88cb40a4dcda8e5a56efcf1b62ba679fcfebdf11e3fb706125}}, signature=SszByteVector{0x8f93f39c11345f89a3a72f5656398aa864077daaaacad0a5f3c3cd2f2ba83851d395067a9db6b2033f50107bf3a36cf0114209c5f15888da506395d7de18ca3ff204d36dd4d012a0ee827e99e47b154c1dd6260008cfd0f8fc6a2a1f1ecddbb5}}
bn-defe:
nothing interesting
Summary: Single attestations was not gossiped with error reason "no peers were available on the required gossip topic". Aggregate attestion was produced and was later included in block when the same vc was chosen as next block producer.
I should find the reason why there were no peers on this topic, why gossip was not replayed later.