# Observations in Attestation Validation in Prysm

In this report, we list some observations in attestation propagation in gossipsub.

We have instrumented go-libp2p-pubsub with [metrics](https://github.com/bharath-123/go-libp2p-pubsub/tree/bm-metrics).

Some of the key metrics we have instrumented are:
1. The time taken by a peer to forward a message it has received to its mesh/fanout peer set
2. The time taken by a peer to push a message it has received to the RPC queues of the peers it wants to forward the message to
3. The time taken for async validation of the message

At a high level, when a message is received by a peer, the message first goes through the async validation pipeline. If that is successful, it is pushed to the corresponding outgoing RPC queues of each of its mesh/fanout peers for that topic. Messages are popped out of the RPC queue and forwarded to the peer's network stream.

Below we have a visual that overlays the above metrics at the p90.


The above visual covers the async validation time, RPC queue push time, and message forward time for messages in the `/eth2/<fork_version>/beacon_attestation_2/ssz_snappy` subnet.

We observe that the majority of the time in forwarding an attestation appears to be spent in async validation.

We profiled [Prysm's async validation code](https://github.com/OffchainLabs/prysm/blob/0aa248e6631912142690929984ac924f31c08ae1/beacon-chain/sync/validate_beacon_attestation.go#L37) for beacon attestations and noticed that most of the time seems to be taken by the [`validateUnaggregatedAttWithState` method](https://github.com/OffchainLabs/prysm/blob/0aa248e6631912142690929984ac924f31c08ae1/beacon-chain/sync/validate_beacon_attestation.go#L315).


Upon profiling the `validateUnaggregatedAttWithState` method, we noticed that the majority of the time is spent in the [`validateWithBatchVerifier` method](https://github.com/OffchainLabs/prysm/blob/0aa248e6631912142690929984ac924f31c08ae1/beacon-chain/sync/batch_verifier.go#L84).


`validateWithBatchVerifier` pushes a batch of attestation signatures to a [BLS signature verifier routine](https://github.com/OffchainLabs/prysm/blob/0aa248e6631912142690929984ac924f31c08ae1/beacon-chain/sync/batch_verifier.go#L31), which accumulates a batch of BLS signatures to be verified. It triggers batch verification when the batch of BLS signatures reaches the max batch limit. It also triggers batch verification every 50ms if the batch size doesn't reach its limit.

The default limit for the [BLS batch verifier appears to be 1000 in Prysm](https://github.com/OffchainLabs/prysm/blob/0aa248e6631912142690929984ac924f31c08ae1/cmd/beacon-chain/flags/base.go#L345). Upon instrumenting the batch sizes as a histogram, we observe that the batch size rarely goes beyond 50.


This could imply that BLS signatures are being verified every 50ms since the batch size never reaches the limit, which could be a potential bottleneck.

We attempted to reduce the batch verification interval to 10ms from 50ms and observed a significant drop in attestation validation time.


We observe that the time to execute the attestation async validator reduces significantly at the p90 after applying this change.


We also observe an almost 33% reduction in message publish time at the p90 after applying this change, which could be attributed to the reduced async validation time.

Should the batch size be tuned appropriately to achieve optimal attestation validation performance?

We observed that the Prysm team initially had a max batch limit of 50, which they later changed to 1000 ([link](https://github.com/OffchainLabs/prysm/blob/0aa248e6631912142690929984ac924f31c08ae1/CHANGELOG.md#L157)). We are unsure why this change was made without considering the gossipsub traffic.

Upon reviewing the analysis to increase the max batch limit from 50 to 1000 in https://github.com/OffchainLabs/prysm/pull/15467, it appears that the analysis did not take into account the average batch sizes and only considered CPU usage at each point in time. It appears that currently, most attestations spend time in the batch waiting for their signatures to be verified.

Recommendations:

1. We could possible pin workers to N CPUs and max them out for BLS signature verification.
2. We could reduce the default value for the batch verifier limit to a more reasonable value.