Update (12/3/2024): Change to verkle-gen-devnet-4 👋 Intro This guide will demonstrate how to setup nodes on the Verkle testnet (verkle-gen-devnet-4). Please note that the current verkle testnets change very frequently, so some configurations may be outdated. However, the overall setup flow should look pretty much the same. ⚙️ Preparation 1. Clone repository git clone https://github.com/ethpandaops/verkle-devnets/ We will need the configuration and genesis files from the above repository later.

Introduction Yes, we know, running a full node is painful. From NodeReal’s BSC Annual Report 2023, we observed that running a full node requires minimal storage of 1.6TB. 1 As of writing this blog, it already exceeded 2TB. However, only a minority of key-value pairs have been accessed in the recent period. With this in mind, BSC developers have come up with a non-consensus state expiry scheme to minimize the state data that full nodes have to store. Unveiling the smallest full node ever Say goodbye to full node pain: Introducing the smallest full node ever.

🌲 Why Post-Verkle State Expiry? Significance of the project As the blockchain grows, a looming challenge arises: State Bloat. With the increasing influx of new user activities into the network, more states are created, including accounts and contracts. The growing state imposes a storage burden on nodes, compelling them to store the entire state and affecting performance. When individuals like yourself and me can no longer run a node, this leads to centralization. Do you see the flow here? More state leads to higher hardware maintenance costs, reducing the number of people able to run a node, which then leads to centralization. In other words, state bloat puts the entire blockchain at risk if not properly addressed. The underlying idea is to remove redundant data—data that is not accessed for a long time. We can achieve this with state expiry. State expiry allows us to temporarily remove inactive data from the blockchain state, so nodes only have to store the most recent data (i.e., the past 6 months) needed to execute a block. If a particular state is needed in the future, anyone can perform a state revive by submitting a cryptographic proof. Okay, that's the state expiry part. But why Verkle tree? Because the Verkle tree is inevitable. It brings numerous benefits such as stateless clients and faster sync time. Hence, it is worth exploring how state expiry looks in the Verkle tree.

Development Summary The analysis on Ethereum state size and the key values analysis have been generated. Meanwhile, the code for conversion is completed and the conversion progress has been running. It will take approximately 2 days to finish the conversion analysis (assuming there are no bugs in the process). Check out my development branch for Verkle Tree here. Check out my development branch for Geth here. Check out my local Verkle testnet setup here. State Storage Analysis

Development Updates __________.__ .__ .__ __ .__ .__ .___ .___ \__ ___| |__ ____ ____ | | ____ ______ | |__ _____ _____/ |_ | |__ _____ ______ | | _____ ____ __| _/____ __| _/ | | | | \_/ __ \ _/ __ \| | _/ __ \\____ \| | \\__ \ / \ __\ | | \\__ \ / ___/ | | \__ \ / \ / __ _/ __ \ / __ | | | | Y \ ___/ \ ___/| |_\ ___/| |_> | Y \/ __ \| | | | | Y \/ __ \_\___ \ | |__/ __ \| | / /_/ \ ___// /_/ | |____| |___| /\___ \___ |____/\___ | __/|___| (____ |___| |__| |___| (____ /_____/ |____(____ |___| \____ |\___ \____ | |__| All of the components have been built already. Also managed to get a local private Verkle testnet running with the Clique consensus. I'm testing out the state expiry functionality by simulating the hard forks and sending revive transactions. The PoC isn't fully done yet as there are still some bugs here and there, but it's close to being done. Check out my development branch for Verkle Tree here.

Development Updates I've completed most of the components related to the EVM layer which includes hard fork, revive gas cost and new transaction type. The next steps is to complete the state storage components and the integration of EVM, state storage and Verkle Tree. Check out my development branch for Verkle Tree here. Check out my development branch for Geth here. Code Notes State Expiry Hard Forks

Summary I had completed all of the codes for Verkle Tree and starting to work on the EVM layer. Development I've completed the core revive logic and added the necessary unit testings to ensure most cases are covered. I've also started working on the EVM layer which includes hard fork logic, revive gas cost and new transaction type. Check out my development branch for Verkle Tree here. Check out my development branch for Geth here.

Summary I had completed most of the required codes related to leaf node which includes the addition of new state epoch metadata and new Verkle proof format. Development I've finished Verkle Tree modifications and state expiry logic. I've also added some unit tests to test out the new Verkle Tree features. New proof format is completed but unit tests have yet to be included. Check out my development branch here. Code Journal New Leaf Node Optimization

Summary I had completed most of the required codes related to leaf node which includes the addition of new state epoch metadata and new Verkle proof format. Development I've finished most of the development related to state epoch metadata and leaf node modification. I'm almost done with modifying the Verkle proof component. Check out my development branch here. Code Notes Navigating through the Verkle commitment seemed daunting at first, but it wasn't so bad when I got hands on. The following is the notes for the NewLeafNode function in the tree.go file:

Summary I've mostly spent my time on the design specification and creating the rough template on setting up a local private network. Design Specs I've finished writing the design specification for this project, which includes the rough design of a few major components such as state epoch metadata, verkle tree, state storage, EVM and pruning. Check it out here. Local Private Network I've briefly setup a repository to test out the state expiry scheme. I'm using Clique consensus (PoA) as it's easy to setup. I will testing on a few scenarios such as submitting regular ETH transfers and ERC20 transfers and test out the expiry feature.

Summary This post-verkle state expiry scheme suggests adding additional state epoch metadata for each value in the leaf node and including it in the calculation of the commitment. The metadata is used to determine if a value is expired or not. Expired values could not be accessed, and users have to first submit a transaction to revive the values by providing the necessary witnesses. State Epoch Metadata State Epoch Definition The definition of a state epoch refers to a time period measured in blocks. This parameter could be adjusted through a governance proposal and pushed to the network through hard forks. Ideally, an epoch should be around 6 months to 1 year so that values do get expired without compromising the user experience significantly. On the client side, the epoch parameter is represented with 16-bit values (i.e., represents 0 to 65535). This should be sufficient with the assumption that a state epoch should span at least several months type StateEpoch uint16

Summary In week 6, I was mostly writing the design specification document for the EPF project. Particularly, I came out with some ideas to modify the Verkle Tree such that the state expiry scheme works. I got some feedback from Ignacio and finalize my design on this component. Design Specification This document covers the design details of each project components such as verkle tree modification, witness format and EVM. I will be including pseudocode or diagrams so it helps to make the development phase as efficient as possible. Verkle Tree Modification The idea of my state expiry scheme is to expire off the values in leaf nodes (i.e. extension nodes). But I faced a bottleneck where the calculation of the commitments depend on the values. So there's no way to delete the values off and replace them with some placeholder data that allow the same commitment value to be regenerated. Instead of expiring each individual value, another approach is to expire "batches" of values. In Verkle Tree, values are batched into 2. The first 128 values are batched together and the commitment C1 is calculated. The same goes to C2.

🧬 Empowering Ethereum’s Eternal Evolution Summary This project introduces a new state expiry scheme that utilizes the Verkle Tree. All inactive values (i.e., account balance, nonce, contract code, slots, etc.) will be expired and can be revived by submitting a Verkle proof. Motivation On Ethereum, the State consists of the following: Account balances Account nonces

Summary This week, I've been working on my project proposal and presentation. I've finalized my project to be post-verkle state expiry where only the values in the bottom nodes are expired, and can be resurrected using a Verkle proof. I’ve dived into the Verkle code repository to familiarise myself with the inner workings of Verkle tree. Project Proposal & Presentation I've completed my project proposal, here's the link. Here's the presentation slides. Verkle Code Deep Dive I've also started to dive deep into the inner workings of the Verkle Tree on the this branch.

Summary This week, I have completed the code for storage analysis, but found some critical issues to the design so I had to redo it. I've also connected with Stellar's state expiry dev and continued my conversation with the relevant mentors. I've also began to write my project proposal. Storage Analysis I've managed to setup a node on mainnet and let it full sync for 1 day. Here's a quick snippet: As shown, we are able to see how much storage is being used for a certain block range (customizable). But there's a huge flaw with this approach. Geth now uses a hash-based approach. Each MPT node is stored in the database where the key is its hash and the value is RLP-encoded data or raw value bytes. As we know, any underlying changes to the leaf nodes (values) will propagate its changes to their parent nodes, which modifies their hashes. Therefore, my current approach of storing key-value as hash(node)-blocknum will not work because there are guaranteed redundant trie nodes in the database Hence, the analysis result will be flawed.

❌ State Bloat: The Killer of all Blockchains On Ethereum, State consists of the following: Account balances Account nonces Contract code Contract storage At any given time, nodes need to know the current state to process new blocks and transactions. As more users join the network, more states are created (more accounts and contracts). As the state grows, the storage requirement for a node increases because they need to store the full state. When a blockchain's state grows significantly, it hinders the network's performance, making it difficult for many people to run nodes, leading to centralization. This issue is known as the State Bloat problem.

Summary This week, I'm mostly working with the storage analysis miniproject. Most of the code components are completed. Now I'm left with completing the analysis component, then I'll perform some simple testing on the Ethereum testnet before running a full node on the mainnet. Storage Analysis To recap, the goal of this miniproject is to analyze the exact amount of state space used and unused within a certain period of time. By having this analysis, it may help with understanding the effect of prior state expiry scheme as well as coming up with more appropriate ones. The method of performing this analysis is quite straightforward. Essentially, I would run a Geth node via full sync on the Ethereum mainnet. The Geth code is modified such that each node would have an additional metadata and stored in the database. The metadata that I chose is the last block number where the node is accessed. By collecting this information, we can precisely analyze a period of inactivity for the entire state. Daily Updates To ensure consistency, I post daily updates (on weekdays) on what I did for EPF. Check out my daily updates this week:

Summary Compilation of State Bloat Solutions I've spent some time compiling and summarize all available state bloat solutions that I found (not limited to Ethereum only). I also came up with my variant of solution, but it's still a rough idea. Here's the notes. Conversations I've also had some conversations with Ignacio and Guillaume to get some context on state expiry. Here are the summarized points: While weak statelessness doesn't technically solve the state bloat problem, the responsibility of storing data is solely placed on block producers, which shouldn't be as bad as we think. Ideally, state expiry should come with a state revive feature. Imagine if L2 has some bug in the contract where some underlying state of the rollup couldn't be recovered, then it's gonna be doomed for them. Perhaps some kind of enshrined MEV could help compensate block producers for storing state.

Summary Currently, my main interests revolve around Verkle Tree, statelessness and state expiry. Here are some of the project direction: Verkle Tree + State Expiry Implementation POC Helping consensus clients (e.g. Prysm) to implement VKT components Helping execution clients (e.g. Reth) to implement VKT components Aside from gathering and reading the necessary materials, I've also talked to Ignacio (mentor) to get some insights on the current progress of Verkle Tree, as well as how I can potentially make a contribution through EPF. Lastly, I've also been studying cryptography mathematics, so that by the time I reach the implementation stage, I'm able to get onto the codebase as quickly as possible.