EPF - Sixth Update

In my last update I described the code involved so far in the CL P2P setup, but I also talked briefly about the necessary next steps for the implementation, which were the main points that took place for these last two weeks.

Ethereum Network

As specified in the consensus specs, the Ethereum Node Record (ENR) for an Ethereum consensus client must carry a generic eth2 key with an 16-byte value of the node's current fork digest, next fork version, and next fork epoch to ensure connections are made with peers on the intended Ethereum network.

This is how the struct looks in rust:

pub struct ForkId {
    pub fork_digest: [u8; 4],
    pub next_fork_version: [u8; 4],
    pub next_fork_epoch: u64,
}

And the constructor for the default fork id:

impl ForkId {
    pub fn new() -> Self {
        Self {
            fork_digest: compute_fork_digest(
                BELLATRIX_FORK_VERSION,
                GENESIS_VALIDATORS_ROOT.into(),
            ),
            next_fork_version: BELLATRIX_FORK_VERSION,
            next_fork_epoch: u64::max_value(),
        }
    }
}

We don't plan a future fork so we can take these values:

• next_fork_version: Remains the same as the current fork.
• next_fork_epoch: Max possible epoch.

And the default config values for the Bellatrix fork are:

// https://eth2book.info/bellatrix/part3/containers/state/#table_0

// BELLATRIX_FORK_VERSION = 0x02000000
pub const BELLATRIX_FORK_VERSION: [u8; 4] = [0x02, 0x00, 0x00, 0x00];

// GENESIS_VALIDATORS_ROOT = 0x4b363db94e286120d76eb905340fdd4e54bfe9f06bf33ff6cf5ad27f511bfe95
pub const GENESIS_VALIDATORS_ROOT: [u8; 32] = [
    75, 54, 61, 185, 78, 40, 97, 32, 215, 110, 185, 5, 52, 15, 221, 78, 84, 191, 233, 240, 107,
    243, 63, 246, 207, 90, 210, 127, 81, 27, 254, 149,
];

I took these values from the Eth2 Book , these are valid for the current main Ethereum network.

Now let’s take a look at how the fork_digest is calculated:

// https://eth2book.info/bellatrix/part3/helper/misc/#compute_fork_digest
pub fn compute_fork_digest(current_version: [u8; 4], genesis_validators_root: Hash256) -> [u8; 4] {
    let mut result = [0; 4];
    let root = compute_fork_data_root(current_version, genesis_validators_root);
    result.copy_from_slice(
        root.as_bytes()
            .get(0..4)
            .expect("root hash is at least 4 bytes"),
    );
    result
}

// https://eth2book.info/bellatrix/part3/helper/misc/#compute_fork_data_root
pub fn compute_fork_data_root(
    current_version: [u8; 4],
    genesis_validators_root: Hash256,
) -> Hash256 {
    ForkData {
        current_version,
        genesis_validators_root,
    }
    .tree_hash_root()
}

#[derive(Debug, Clone, PartialEq, Default, Serialize, Deserialize, Encode, Decode, TreeHash)]
pub struct ForkData {
    pub current_version: [u8; 4],
    pub genesis_validators_root: Hash256,
}

pub trait SignedRoot: TreeHash {
    fn signing_root(&self, domain: Hash256) -> Hash256 {
        SigningData {
            object_root: self.tree_hash_root(),
            domain,
        }
        .tree_hash_root()
        .into()
    }
}

#[derive(Debug, PartialEq, Clone, Serialize, Deserialize, Encode, Decode, TreeHash)]
pub struct SigningData {
    pub object_root: Hash256,
    pub domain: Hash256,
}

impl SignedRoot for ForkData {}

The key part in these structs are the helpful macros provided by the libraries I’ll add below, they allow us to encode and serialize the data in the specified way.

use serde_derive::{Deserialize, Serialize};
use ssz_derive::{Decode, Encode};
use tree_hash::{Hash256, TreeHash};
use tree_hash_derive::TreeHash;

• Serde Derive:
• SSZ Derive
• Tree Hash and Tree Hash Derive

Now using this in the ENR creation:

// eth2 field key
pub const ETH2_ENR_KEY: &str = "eth2";

pub fn build_enr(combined_key: &CombinedKey) -> Enr {
    let mut enr_builder = enr::EnrBuilder::new("v4");

    enr_builder.ip("0.0.0.0".parse().unwrap());

    enr_builder.udp4(9000);

    enr_builder.tcp4(9000);

    enr_builder.add_value(ETH2_ENR_KEY, &ForkId::new().as_ssz_bytes());

    enr_builder.build(combined_key).unwrap()
}

Now the ENR created will have the eth2 key with the formatted ForkId.

Data Transform

I started modifying the gossipsub behaviour creation to include the transform:

// build a gossipsub network behaviour
let mut gossipsub = Gossipsub::new_with_transform(
    MessageAuthenticity::Anonymous,
    gossipsub_config,
    None,
    SnappyTransform::new(),
)?;

Then created a simple struct with a constructor:

pub struct SnappyTransform {
    max_size_per_message: usize,
}

impl SnappyTransform {
    pub fn new() -> Self {
        SnappyTransform {
            max_size_per_message: GOSSIP_MAX_SIZE_BELLATRIX,
        }
    }
}

Implemented the gossipsub::DataTransform trait for this struct:

impl DataTransform for SnappyTransform {
    fn inbound_transform(
        &self,
        raw_message: RawGossipsubMessage,
    ) -> Result<GossipsubMessage, std::io::Error> {
        let len = decompress_len(&raw_message.data)?;
        if len > self.max_size_per_message {
            return Err(Error::new(
                ErrorKind::InvalidData,
                "ssz_snappy decoded data > GOSSIP_MAX_SIZE",
            ));
        }

        let mut decoder = Decoder::new();
        let decompressed_data = decoder.decompress_vec(&raw_message.data)?;

        Ok(GossipsubMessage {
            source: raw_message.source,
            data: decompressed_data,
            sequence_number: raw_message.sequence_number,
            topic: raw_message.topic,
        })
    }

    fn outbound_transform(
        &self,
        _topic: &TopicHash,
        data: Vec<u8>,
    ) -> Result<Vec<u8>, std::io::Error> {
        if data.len() > self.max_size_per_message {
            return Err(Error::new(
                ErrorKind::InvalidData,
                "ssz_snappy Encoded data > GOSSIP_MAX_SIZE",
            ));
        }
        let mut encoder = Encoder::new();
        encoder.compress_vec(&data).map_err(Into::into)
    }
}

• inbound_transform: Takes a RawGossipsubMessage received and converts it to a GossipsubMessage
• outbound_transform: Takes the data to be published and transforms it. The transformed data will then be used to create a crate::RawGossipsubMessage to be sent to peers.

And finally update the behavior gossipsub type:

// We create a custom network behaviour that combines Gossipsub and Discv5.
#[derive(NetworkBehaviour)]
struct Behaviour {
    gossipsub: Gossipsub<SnappyTransform, AllowAllSubscriptionFilter>,
    discovery: Discovery,
}

Current Challenges

I added the necessary changes I mentioned in the last update, but of course there are still some things to solve. Currently I can connect to peers and receive messages, but I keet being disconnected.

Local peer id: 16Uiu2HAkySRycoEf4hjCHyHqEU3J54RswvSFkmmAx4mtsdHzmPhH
Swarm: NewListenAddr { listener_id: ListenerId(8721776756759638073), address: "/ip4/127.0.0.1/tcp/9000" }
Swarm: NewListenAddr { listener_id: ListenerId(8721776756759638073), address: "/ip4/10.182.221.108/tcp/9000" }
Swarm: ConnectionEstablished { peer_id: PeerId("16Uiu2HAmVEWyvezosFuFbnpTt7RWfQBGHUWAx76GhdV6R66hak1o"), endpoint: Dialer { address: "/ip4/127.0.0.1/tcp/9002", role_override: Dialer }, num_established: 1, concurrent_dial_errors: Some([]) }
Swarm: ConnectionEstablished { peer_id: PeerId("16Uiu2HAmMJ5wq7Uih3RQ2PzisBCFcBuYg5Qy15BYMh8dQnBVcvVr"), endpoint: Dialer { address: "/ip4/127.0.0.1/tcp/9003", role_override: Dialer }, num_established: 1, concurrent_dial_errors: Some([]) }
Swarm: ConnectionEstablished { peer_id: PeerId("16Uiu2HAkxXY47H2ecnysMRHPzu4Kc2S7AuE8DjhgKNShNUME2382"), endpoint: Dialer { address: "/ip4/127.0.0.1/tcp/9004", role_override: Dialer }, num_established: 1, concurrent_dial_errors: Some([]) }
Gossipsub: Message { propagation_source: PeerId("16Uiu2HAkxXY47H2ecnysMRHPzu4Kc2S7AuE8DjhgKNShNUME2382"), message_id: MessageId(62f8937b482d3174c5017e3ef573786914617c17), message: GossipsubMessage { data: 640000008067ec8433.., source: None, sequence_number: None, topic: TopicHash { hash: "/eth2/224daa27/beacon_block/ssz_snappy" } } }
Gossipsub: Message { propagation_source: PeerId("16Uiu2HAkxXY47H2ecnysMRHPzu4Kc2S7AuE8DjhgKNShNUME2382"), message_id: MessageId(b691f9178a00274025696ee204bd5a94d4ac8688), message: GossipsubMessage { data: 64000000848102bf4f.., source: None, sequence_number: None, topic: TopicHash { hash: "/eth2/224daa27/beacon_block/ssz_snappy" } } }
Gossipsub: Message { propagation_source: PeerId("16Uiu2HAmMJ5wq7Uih3RQ2PzisBCFcBuYg5Qy15BYMh8dQnBVcvVr"), message_id: MessageId(fed1c3b8a90ca2dadc91c360885c4a89fe1e3d19), message: GossipsubMessage { data: 64000000b3235b0fac.., source: None, sequence_number: None, topic: TopicHash { hash: "/eth2/224daa27/beacon_block/ssz_snappy" } } }
Gossipsub: Message { propagation_source: PeerId("16Uiu2HAmMJ5wq7Uih3RQ2PzisBCFcBuYg5Qy15BYMh8dQnBVcvVr"), message_id: MessageId(ffc781e23087d5d66e23e13042830ab60ed3ab90), message: GossipsubMessage { data: 64000000a8ee44128b.., source: None, sequence_number: None, topic: TopicHash { hash: "/eth2/224daa27/beacon_block/ssz_snappy" } } }
Gossipsub: Message { propagation_source: PeerId("16Uiu2HAmMJ5wq7Uih3RQ2PzisBCFcBuYg5Qy15BYMh8dQnBVcvVr"), message_id: MessageId(fa73d2d9ca6a73c40c47417060ec16b739b4f8a5), message: GossipsubMessage { data: 64000000b88551ad1f.., source: None, sequence_number: None, topic: TopicHash { hash: "/eth2/224daa27/beacon_block/ssz_snappy" } } }
ConnectionClosed: Some(IO(Custom { kind: Other, error: A(YamuxError(Closed)) }))
ConnectionClosed: Some(IO(Custom { kind: Other, error: A(YamuxError(Closed)) }))
ConnectionClosed: Some(IO(Custom { kind: Other, error: A(YamuxError(Closed)) }))

I couldn't find anything in the specs that could let me know what was lacking in my codebase, so I reached to @alexstokes for help, and he suspected it could be a scoring issue (yes, peers are scored to keep track of usefulness or maliciousness) and advised me to test with a local testnet to see what was going on.

I went on with that using the lighthouse client, and here is what I found:

DEBG Connection established                  connection: Listener, peer_id: 16Uiu2HAkySRycoEf4hjCHyHqEU3J54RswvSFkmmAx4mtsdHzmPhH, service: libp2p
DEBG RPC Error                               direction: Outgoing, score: 0, peer_id: 16Uiu2HAkySRycoEf4hjCHyHqEU3J54RswvSFkmmAx4mtsdHzmPhH, client: Unknown, err: Peer does not support the protocol, protocol: metadata, service: libp2p
DEBG Peer score adjusted                     score: -10.00, peer_id: 16Uiu2HAkySRycoEf4hjCHyHqEU3J54RswvSFkmmAx4mtsdHzmPhH, msg: handle_rpc_error, service: libp2p
// ..
TRCE Sending Ping                            peer_id: 16Uiu2HAkySRycoEf4hjCHyHqEU3J54RswvSFkmmAx4mtsdHzmPhH, service: libp2p
DEBG RPC Error                               direction: Outgoing, score: -9.999968760147635, peer_id: 16Uiu2HAkySRycoEf4hjCHyHqEU3J54RswvSFkmmAx4mtsdHzmPhH, client: Unknown, err: Peer does not support the protocol, protocol: ping, service: libp2p
DEBG Peer has been banned                    score: -100.00, peer_id: 16Uiu2HAkySRycoEf4hjCHyHqEU3J54RswvSFkmmAx4mtsdHzmPhH, service: libp2p
DEBG Peer Manager disconnecting peer         reason: Bad Score, peer_id: 16Uiu2HAkySRycoEf4hjCHyHqEU3J54RswvSFkmmAx4mtsdHzmPhH, service: libp2p
TRCE Async task completed                    task: libp2p
DEBG Ignoring rpc message of disconnecting peer, peer_id: 16Uiu2HAkySRycoEf4hjCHyHqEU3J54RswvSFkmmAx4mtsdHzmPhH, msg_kind: outbound_err, protocol: goodbye, service: libp2p
DEBG Peer disconnected                       peer_id: 16Uiu2HAkySRycoEf4hjCHyHqEU3J54RswvSFkmmAx4mtsdHzmPhH, service: libp2p

I was being disconnected for:

• Not supporting the metadata protocol
• Not supporting the ping protocol

I was also curious about which other unsupported protocols could make me a bad peer, so I looked in the lighthouse client for it:

RPCError::UnsupportedProtocol => {
    // Not supporting a protocol shouldn't be considered a malicious action, but
    // it is an action that in some cases will make the peer unfit to continue
    // communicating.

    match protocol {
        Protocol::Ping => PeerAction::Fatal,
        Protocol::BlocksByRange => return,
        Protocol::BlocksByRoot => return,
        Protocol::Goodbye => return,
        Protocol::MetaData => PeerAction::LowToleranceError,
        Protocol::Status => PeerAction::LowToleranceError,
    }
}

Clearly I needed the ping protocol working. For my next steps I will implement these protocols. Until then!