Try   HackMD

Development Update 7

It's been a long time since the last update. Well friends, I'm back!
A lot has been happening behind the scenes…

  1. I dove into the DAS-Prototype codebase
  2. Created DAS-Playground
  3. Run DAS-Prototype on my local machine

Friendly reminder: DAS-Prototype is a repository that Eric and Timofey have built to test possible DAS networking solutions. I'm modeling one of these possible solutions.

 

Preface

Peer to Peer Networks

Peer to peer networks are a bunch of individual nodes all agreeing to connect with one another to store and pass along useful data to others in the network. Instead of the typical client/server model, where there are some privileged nodes (servers) controlling information flow, all nodes are on the same footing; they're all clients and servers at the same time!

The p2p paradigm allows for more resilient, decentralized, and trust-minimized networks. Wahoo!

DAS

The scaling solution for Ethereum, called Danksharding, requires new information to be placed on the blockchain. This information, aka "blob" data, is state transition data that rollups (layer 2s) need to post to the base chain (layer 1) for security reasons. The importance of blobs and rollups can be found in Wonderland's Danksharding Notes.

Before blob data can be placed on chain, a proposer must present the blobs to DAS validators. These validators within the network must agree on the blob data being legitimate and available. Once validators agree to this, the block (containing the blobs) is published.

*Other nodes can help spread information throughout the network/verify samples are available, but validators are the gatekeepers for what blobs go on chain!

*Multiple blobs will be placed in a single block with full Danksharding

Project Summary

Our goal is to figure out how individuals interested in DAS can communicate and come to consensus on the availability of blob data.

I'm exploring one possible solution to the DAS networking problem: creating a secure K-DHT overlay atop discv5.

In the best case, a DAS overlay network (where any node can participate) will be used to pass around blob chunks (samples) and attestations to these samples. If/when the main overlay is under attack, DAS nodes will switch networks to this secure overlay where only validators communicate this DAS data.

DAS-Prototype is a testing ground + benchmarking repository for different DAS networking designs. I'll be utilizing this repository to model the secure K-DHT… Thanks, Timofey and Eric!

The full project proposal can be found here.

 

Diving into DAS-Prototype

During the holidays, I cracked open the repository and had a couple of realizations while exploring:

  1. This whole repo revolves around a single data structure: DASNode
    To build a secure K-DHT overlay and integrate/test it within DAS-Prototype, I'll need to understand and modify this data structure.
  2. Modeling p2p networks require the utilization of the asynchronous + concurrent programming paradigms.
  3. Creating an isolated repository where I can tinker with these networking concepts from DAS-Prototype would be extremely beneficial.

DASNode

pub struct DASNode {
    discovery: Arc<Discovery>,
    libp2p: Libp2pService,
    samples: Arc<RwLock<HashMap<NodeId, usize>>>,
    handled_ids: Arc<RwLock<HashMap<Vec<u8>, usize>>>,
    overlay: Arc<OverlayProtocol<DASContentKey, XorMetric, DASValidator, MemoryContentStore>>,
}

The discovery, libp2p and overlay fields within DASNode are protocol data structures that give our node the ability to interact with and handle their respective networks' requests/responses. Each field provides access to the routing tables and other useful functionalities.

The libp2p service is an alternative option to Eric and Timofey's discv5's talkreq + utp based messaging! Both options are being tested out within DAS-Prototype, however Ethereum will likely converge on a single transport option.

The samples field is where a node stores the chunks of blob data it's responsible for serving and the handled_ids field is still a mystery… TBD!

Async

Remember the communication between nodes we talked about a few paragraphs back?

It takes time for any node to talk to another. In order for our nodes to remain efficient (not get stuck waiting for a response, etc.), there needs to be a way for a peer to run some code in the background while the rest of the code can continue executing.

This concept is referred to as asynchronous programming and is generally used within our DASNode when requesting another node for peers or samples.

Concurrency

Concurrent programming is a programming paradigm that allows for multiple sets of asynchronous code to be run congruently. Tasks are the objects that handle individual asynchronous code. Tokio's library gives the user the ability to run lots of tasks in tandem.

Details on the matter can be found inside of the Tokio tutorial.

The two major functionalities within our model that utilize concurrent programming:

  1. Peer to peer networks require each node acts as both a client and server. Each individual peer needs to be able to respond to multiple requests at the same time (efficient servers don't just reply to one request sequentially).

  2. Multiple peer interactions within a network happen simultaneously. To recreate this inside a model, nodes need to be able to run their queries + responses concurrently.

Note:
"Concurrency and parallelism are not the same thing. If you alternate between two tasks, then you are working on both tasks concurrently, but not in parallel."

For some computation to be executed in parallel, you'd need multiple CPUs processing individual instances of code simultaneously!

 

DAS-Playground

Instead of directly jumping in to making PRs to Model-DAS, I've decided to first tinker with/build out DAS p2p ideas in an isolated environment. So… DAS-Playground was created!

So far…

I've created a program that spins up tasks that create nodes with new discv5 servers, but recently realized that I need to create nodes that contain individual tasks for each protocol a DASNode supports (discv5, libp2p and overlays).

Nodes with tasks for servers are good. Tasks that create these nodes are not good.

Repository Goals

  1. Create a fully functional DASNode
  2. Facilitate basic communication and interactions between DASNodes
  3. Create a new field within DASNode that contains a secure Kademlia DHT routing table. Only validator nodes!

Once I've achieved said goals for DAS-Playground, implement them within Model-DAS.

Note:
Brechy's repository, CL-P2P-Setup, is a fantastic resource to help implement some of these fundamental networking concepts. Thanks, Brechy!

 

Running DAS-Prototype

It's worth noting that over the holidays I got Eric and Timofey's repo running on my computer!

Timofey helped fix WSL configuration problems within my setup. There was some OS platform specific behavior within std::fs::DirEntry's metadata() function.

It turns out that Linux's tmpfs (temporary file system), used within WSL, doesn't support the specific metadata field that metadata().created() utilizes

 

Readings + Writings

The Internet: Explained from First Principles is a tremendously helpful article for getting a better understanding of networking as a whole. If you're not a networking beast, I'd highly recommend reading this thing BEFORE diving into the weeds of Ethereum's networking stack… I wish I'd have read this thing a few months back!

I'm working on a few articles that help tie a bunch of concepts that were floating around my head. One summarizes Data Availability Sampling and the other goes in depth on the peer to peer networking solution I'm wanting to model. I hope to have both finished within the next few weeks!

 

To Do

  • Continue work within DAS-Playground
  • Finish reading Timofey's DAS Research doc + writing feedback
  • Complete "Intro to Data Availability Sampling" + "DAS Overlays" writings
  • Iterate on project proposal (Dankrad gave feedback before the holidays)
Last changed by 
 
Echo
0
253

Read more

Secure Kademlia Library

DAS Networking in Ethereum requires lots of small blob samples to be placed in a distributed network where arbitrary nodes can query “the network” to make sure these samples are available.

Aug 14, 2023
Aggregated DAS Content

This is a living document of my favorite Data Availability Sampling resources. 

Jul 5, 2023
Development Update #10

Before we get to the normal Dev Update... Over the past 4 months I dove deep into Danksharding, Ethereum Networking, and then p2p networking in rust + building out a discv5 overlay simulation. A few days back, I read a few &quot;big-picture&quot; Ethereum developer documents: Network Layer and Data Availability. Somewhere along the journey I lost the thread. I was building this Data Availability Sampling p2p network simulation without understanding the &quot;why&quot; behind it all.  

Apr 7, 2023
EPF Final Update

Tackling the DAS Networking problem requires a p2p network that has low-latency, is privacy preserving and is sybil-resistant. The idea behind Model DAS is to measure benchmarks on DAS querying via Secure Kademlia DHT Discv5 overlay network. This Secure K-DHT is a seperate validator-only overlay network that leverages validators&apos; incentive alignment to create a robust DAS network if the primary network were to come under attack. Status Report The original vision for Model DAS didn&apos;t entail creating a new DAS simulation; the idea was just to extend the functionality of another. But seeing that I was new to gerenal networking concepts and the Rust programming language, it quickly became aparent that building a simplified version of DAS Prototype was an escential prerequisite to executing the vision behind Model DAS. Finished

Mar 1, 2023
Read more from Echo
Published on HackMD