# Sync committee subnet sybil/eclipse attack

## Spoiler

The attack dropped Sync Committee participation by about **15%** on Prater
The attack 'warm up' period: about **1 hour**
The attack cost: less than **1.5 USD/hour** on AWS
Public IPs used: **100**
Sybil nodes used: **256**

### Attack potential

With further efforts the attack could potentially drop participation by 90%
The same attack is applicable to attestation subnets

## Short description

From one hand this is a sybil attack, when attacker mints a set of malicious nodes which size exceeds the number of honest nodes by factors (x3-x10 in our case). From the other hand this is an eclipse attack as the main target is to eclipse honest nodes.

While attacking the whole network requires a larger amount of resources the described attack targets a smaller subnet of Ethereum nodes (Sync Committee Subnet) which comprises maximum 128 nodes running sync committee subnet validators selected by the consensus protocol.

The attack goal is
1. Establish as many p2p connections as possible between attacker sybil nodes and victim honest nodes to eclipse victim nodes from each other within the same subnet (while a node still may not be eclipsed from the rest of the honest network). Sybil nodes behave identically to honest nodes until attack phase.
2. At some moment stop propagating subnet Gossip messages which would prevent attesting nodes from publishing/propagating individual contributions to aggregating nodes and thus decrease Sync Committee participation rate in blocks

To skip details and jump to attack results click this [TLDR shortcut](#Attack-phase1)

## Long description

The attacker node utilizes both connection types:
- _Outbound_: the attacker searches for the subnet nodes and makes active connection attempts to them from all available sybil nodes.
- _Inbound_: the attacker advertises his sybil nodes via Discovery system and listens for inbound p2p connections from the victim nodes. Connections from other nodes are dropped.

### Algorithm

#### Sybil part

All sybil nodes start listening on their dedicated endpoints (IP + TCP port). See [Heku](#Heku) topic for more details.

To advertise nodes in Discovery a dedicated DiscV5 system is started for every sybil node. A discovery system should be up and listening on the same IP as its corresponding node. Else the `NodeRecord` would not be disseminated withing discovery network or would be evicted over time unless DiscV5 is not responding to inbound `Ping` requests.

All inbound RPC connections are unboundly accepted unless a peer is not subscribed to any sync committee subnet and therefore is useless for the attack.

#### Victim nodes search

A separate [discovery system](#Discovery) is used to traverse all known Eth2 nodes and select those claiming to be subscribed to `sync_committee_0` gossip topic (via `SYNC_COMMITTEE_SUBNET_ENR_FIELD` custom field in `NodeRecord`)

The above lookup basically located around 90-100 nodes which I suppose were effectively the nodes where the majority of validators were running in the Prater testnet. So usually most of them were subscribed to all `sync_committee_*` subnets

#### Active connections

The node starts initiating connections to all victim nodes from all sybil nodes. It was configured to initiate a connection once per 3 minutes for a single local-remote node pair. On average it resulted in `256 * 100 / (3 * 60) = 142 connections/second`

#### Warm-up period

The node is waiting to gain enough outbound and inbound connections. The maximum number of connections observed was about 7000-8000

Note: after exceeding that number the attacking node becomes unstable due to remaining software issues. Not sure that could be solved by adding more CPU cores. Likely at the moment the easiest way to scale the attack is just spinning up more setups.

#### Monitoring

The attaker continuously gather statistics on Sync Committee participation from 3 sources
1. `contrib`: Individual contributions from `sync_committee_*` subnets
2. `aggr`: Aggregated participations from the `sync_committee_contribution_and_proof` topic
3. `block`: Participation finally included into the blocks

The attack is considered successful when `aggr / contrib` ratio drops significantly during attack phase.

#### Attack phase

The attack phase is started/stopped manually. During attack all sybil nodes just suspend propagating all gossip messages on `sync_committee_*` topics.

The percentage of participation in the aggregated sync committee messages is monitored.

Prior to starting the attack phase it's hard to estimate (though I believe is possible in theory) the expected drop of participation rate. It strongly depends on the peer topology of the victim nodes.

If a node was eclipsed from the subnet then it would be missing from committee participation (Aggregator nodes marked with `A`):


If a node was not completely eclipsed from the sync subnet its messages would still be propagated:


An interconnected subset of nodes could be eclipsed. If there is no subnet aggregator among them messages of those nodes would not be aggregated and thus included to block:


But if there is at least one aggregator node it would aggregate participations of that subset and still publish it to the aggregates topic (Aggregator nodes marked with `A`):


### It's not too easy to connect

Effectiveness of a sybil attack is also greatly affected by low connection rate to honest peers. While the attacker node is always welcome to accept an inbound connection and not drop it as long as possible, honest nodes (especially nodes with many validators) are often full of inbound connections and would drop the connection during the next slot timeframe with high probability.

Below there is some connection success statistics. It's aggregated with respect to number of attestation subnets a node is subscribed to (which is basically correlates to the number of validators a node running). 'Successful connection' means here: a node accepted inbound connection and didn't drop it during one slot.

Basically the chance of successful connection is about **15%**

| Att subnets | Succes rate | Connect attempts | Peer count |
|-------------|-------------|------------------|------------|
| 0 | 0.152 | 76174 | 332 |
| 1 | 0.236 | 279651 | 1335 |
| 2..63 | 0.098 | 326496 | 1243 |
| 64 | 0.132 | 59848 | 230 |
| **Total** | **0.165** | **742169** | **3140** |

([Source data](https://docs.google.com/spreadsheets/d/1REmHpjxaXvUZaSzuabeZQ5yForcKc7tFkNFrWvIqbp0/edit?usp=sharing#gid=1962457285))

To make the attack effectiveness even better an attacker may try to keep the victim nodes inbound connection pool flooded with connection requests. Either apply another attack which works on a lower network level and keeps victim TCP sockets busy with slow negotiated connections.

## Implementation details

### Heku

'Heku' (aka 'Hacker Teku') is the Teku node with some modifications to its components, primarily to its networking layer. Worth mentioning that Heku uses unofficial Teku Java API and doesn't alter the Teku codebase.

Heku makes it easier to get access to internal node components and substitute them with custom implementations. It also has a set of utilities to fetch and analyze networking events.

(Heku [GitHub repo](https://github.com/Nashatyrev/heku) is private at the moment as it contains various attack scenarios. Please contact me if anyone is interested in browsing it.)

#### MuliPeerNode


`MultiPeerNode` is the modified node which is able to pretend being several 'virtual' nodes with a separate `nodeId` and network endpoint (listening port and optionally IP address). That way a node could have more than one connection to a remote peer. Maximum number of connections to a single peer is bounded by the number of local endpoints.

The original Teku codebase expects one connection per remote node. However the Heku may establish several connections to single remote node via distinct local virtual nodes. To address this issue the Teku core code sees *mangled* `PeerId` per connection which is a mix of remote and local `PeerId`s.
From the Heku side the `MultiPeerId` class contains complete peering information: both remote and local PeerId


`MultiPeerNode` also requires a special peer manager implementation (`MultiPeerManager`) which `connect()` method accepts local endpoint besides remote peer address.

`ConnectionManager` implementation is created depending on the attack scenario

##### Discovery

Two kinds of discovery systems were running:
- Dedicated discovery system for every local endpoint with a dedicated listening UDP port for advertising local endpoints for inbound connections
- One Discovery to search for the victim peers (advertised as sync subnet participants). It needs no dedicated listening port as it serves just outbound discovery requests

##### Gossip

Unlike discovery there is just a single Gossip instance. Thanks to `MultiPeerId` mangling Gossip treats every connection (local endpoint <=> remote peer) as a separate remote peer.

Attacker app intercepts Gossip message handling to gather gossip statistics and suspend message forwarding during attack phase

### IP proxies

The original idea was spinning up sybil nodes on just a single box running `MultiPeerNode` with a single public IP address. However it appeared that the Prysm client bans all the nodes colocated on a single IP if there are more than 5 such nodes detected: either found in Discovery or performed connection attempts.

One of the ways to get additional public IPs (which appeared to be the easiest on AWS) was to spin up minimal boxes to use them as proxies for TCP and UDP traffic

- TCP outbound: **SOCKS5**
Used the `Dante` server with the simplest config
- TCP inbound: **iptables**
:::spoiler Configure commands

        iptables -t nat -A PREROUTING -p tcp --dport 9000:10000 -j DNAT --to $DEST_IP
        iptables -t nat -A POSTROUTING -p tcp -d $DEST_IP --dport 9000:10000 -j SNAT --to-source $LOCAL_IP

:::
- UDP outbound: **direct**
To keep it simple the outbound UDP traffic was not relayed via SOCKS5 server.
- UDP inbound: **iptables**
The node advertised with Discovery is required to listen on the advertised IP, else it's `NodeRecord` would not be added (or pruned later) to the Discovery network tables.
:::spoiler Configure commands

    iptables -t nat -A PREROUTING -p udp --dport 9000:10000 -j DNAT --to $DEST_IP
    iptables -t nat -A POSTROUTING -p udp -d $DEST_IP --dport 9000:10000 -j SNAT --to-source $LOCAL_IP

:::
### AWS setup

- 1x 16-core box (`c5a.4xlarge` : **0.73 $/hr**): to run the `MultiPeerNode`
- 100x minimal boxes as IP proxies (`t2.nano` : **0.0066 $/hr**): to run traffic proxies. `Terraform` was used to spin up and setup proxy boxes

The total cost is **1.39 $/hr**

### Some implementation tricks

- optimizations: there was a number of LibP2P optimizations (mostly Gossip) merged upstream to manage enormous number of peers
- Some validations were turned of: BLS, Discovery signatures, Peer `Status` message verification. More details on the latter is [here](#Status-message-DoS)
- Advertised (via Discovery) that sybil node subscribed to _all_ attestation subnets, while in fact not subscribed to any subnet upon connection. This trick was done to make sybil nodes more attractive for connection from victim nodes
- for numerous public IPs use numerous AWS nano instances instead of elastic public IPs for one EC2 instance to avoid uncomfortable questions from AWS security team

Optimizations and validations tweaks allowed to reach throughput up to 2Gbit/sec and more than 7000 simultaneous peer connection for a single Heku node.

## Attack statistics

### Connections

The number of victim peers found via discovery (subscribed to sync subnet `0`) was: **82**
The total number of peers connected (subscribed to any sync subnet) was: **~133**
The total number of connections was: **~4300**, outbound/inbound: **~3100/1200**

Connections by client:
| Client | Peer count | Outbound | Inbound | Total connections |
|------------|------------|----------|---------|-------------------|
| LIGHTHOUSE | 69 | 2528 | 705 | 3233 |
| NIMBUS | 15 | 135 | 203 | 338 |
| PRYSM | 13 | 16 | 50 | 66 |
| TEKU | 36 | 384 | 18 | 402 |
| UNKNOWN | 2 | 89 | 45 | 134 |

Average per peer:
| Client | Outbound | Inbound | Total connections |
|------------|----------|---------|-------------------|
| LIGHTHOUSE | 36.6 | 10.2 | 46.9 |
| NIMBUS | 9.0 | 13.5 | 22.5 |
| PRYSM | 1.2 | 3.8 | 5.1 |
| TEKU | 10.7 | 0.5 | 11.2 |
| UNKNOWN | 44.5 | 22.5 | 67.0 |

(detailed data could be found [here](https://docs.google.com/spreadsheets/d/1iHd0a6jjIIozVA038PNVr8LEaTHRJWntI8Uc9ZRXYeE/edit?usp=sharing#gid=2076524043))

Different clients respond very differently to the attack. Presumably different implementations of `ConnectionManager`, peer scoring, etc matter.

As per statistics the most 'friendly' was Lighthouse and the most unresponsive was Prysm. The statistics was pretty varying from run to run. The actual reasons why a client is affected by the attack this or that way is quite interesting but was out of scope of attack investigation.

#### Traffic

There is no accurate traffic statistics. With 5000-7000 connections and 1 attestation subnet subscription the traffic peak was about **1-1.5 Gbit/sec**

#### LightHouse node full eclipse case

It was reported by [@AgeManning](https://github.com/AgeManning) from Sigma Prime that some Lighthouse nodes were 100% eclipsed by the attacking nodes:


That could also be observed from the attacker side: Some Lighthouse nodes had reached their connection limit of 130 peers

| Client | Conn | Out conn | In Conn |
|------------|------|----------|---------|
| LIGHTHOUSE | 130 | 80 | 50 |
| LIGHTHOUSE | 130 | 75 | 55 |
| LIGHTHOUSE | 130 | 70 | 60 |

### Attack phase

There were 3 attack phases (~16 slots each) when all messages on all `sync_committee_*` topics are stopped being propagated. While the original attack was intended to target only one subnet in the Prater testnet validators are concentrated on a limited set of nodes so the attack effectively affected all the subnets. Thus it was reasonable to attack all subnets for better attack effectiveness.

The graph below shows the Sync Committee metrics changes during attack phases (thick red lines). While the attacker node still receives all individual contributions it stops propagating them during attack phases. The resulting effect is distinctly visible on the aggregated metrics - they drop by 10-15% during attack phases.

[Data source](https://docs.google.com/spreadsheets/d/1iHd0a6jjIIozVA038PNVr8LEaTHRJWntI8Uc9ZRXYeE/edit?usp=sharing)
- <span style="color:red">⬤</span> Attack perdios, when all gossip message on `sync_committee_*` topics are stopped being propagated
- <span style="color:#4285f4">⬤</span> Number of individual Sync Committee contributions on `sync_committee_*` topics
- <span style="color:#e69138">⬤</span> Total number of aggregated Sync Committee participants on `sync_committee_contribution_and_proof` gossip topic
- <span style="color:#9900ff">⬤</span> Maximum number of participants in a single aggregated message across 4 subnets on `sync_committee_contribution_and_proof` gossip topic
- <span style="color:#34a853">⬤</span> Number of participants included into a block

## Conclusions

Small Gossip subnets are pretty vulnerable to sybil/eclipse attacks. Moreover it was demonstrated that such attacks could be quite cheap and even performed with public cloud services.

## Potential recommendations
### Increase the number of peers

In case of high available bandwidth it makes sense to increase the maximum number of connected peers. That should proportionally increase the space for small topics peers and should exponentially increase the resistance to sybil/eclipse attacks.

It was demonstrated that Teku could handle thousands of connections with just a few modifications and optimizations. Other clients might definitely do the same or even better.

### Long term peer scoring

It could make sense to maintain long term nodes scoring and reserve the connection pool for nodes with good and long score history. That will not 100% prevent such attacks but should definitely harden them

### Attestation subnet proposal

We should probably revisit this PR: [ Proposed attnets revamp #2749 ](https://github.com/ethereum/consensus-specs/issues/2749) and probably generalize it to other small subnets (such as Sync Committee and Smapling at some point).

Adopting this proposal should significantly harden sybil/eclipse attacks on small gossip subnets

## Open questions
#### Unreliability
When not too much virtual nodes involved (256 against planned 1K-2K) the success of the attack heavily depends on 'luck' of how many subnet nodes were eclipsed. To make this attack more reliable and effective the setup needs to be scaled.

#### Some clients are less affected

For example Prysm client for some reason doesn't accept too many inbound connections and doesn't initiate too many outbound connections. The reason could be investigated further. Probably the attacking software could be fixed to make Prysm more vulnerable to this attack

#### Not tested during sync committee turn over

Probably the attack would be more effective if performed during the sync committee turn over. However it needed some more software upgrading

#### `Status` message DoS

With many peer connections (usually more than 3000) at some point Heku become unable to handle existing connections. It was noticed `StorageQueryChannel` was flooded with `getLatestFinalizedStateAtSlot` requests which (for unknown reason) were processed very slowly. The issue was tracked down to the root cause: a lot of simultaneous `Status` messages from all connected peers which are validated by the `PeerChainValidator`.

I couldn't reliably reproduce the problem but disabling `PeerChainValidator` eliminated its effect.

There is a chance that Teku still has DoS vulnerability: 1 or more malicious peers may start sending specific `Status` messages at specific periods which would cause Teku DB service become totally unresponsible (I had observed some periods when a simple DB read request was processed for about 10sec and queued DB read request waits for 10mins)