Bonsai - HackMD

# Bonsai ## Introduction Bonsai introduces several enhancements to the Besu client. Unlike other clients that store nodes of the world state by hash, Bonsai stores them by location in the trie. This approach enables natural pruning by replacing old nodes at the same position in the trie with the new ones. Unlike hash-based storage, which only adds new nodes to the database and keeps growing, Bonsai's technique ensures that there is only one version of the state at any given time. Consequently, it does not support managing reorganizations (reorgs) as it only maintains a single state. Bonsai addresses this limitation by introducing "trielogs" > You can visit this link to see a diagram illustrating how the world state is structured. https://ethereum.stackexchange.com/a/6413 ## Components of Bonsai ### Trie The trie component of Bonsai stores nodes based on their location in the trie, enabling calculation of the root hash and block validation. By naturally pruning old nodes, the trie keeps the state representation compact and reduces storage requirements. ![](https://hackmd.io/_uploads/SJSqVxUUh.png) Before Bonsai, the trie nodes was saved like that ![](https://hackmd.io/_uploads/BkOeYVUU2.png) > This diagram is a simplified representation of the trie. There are also extension nodes, and each branch can have up to 16 children. The trie is stored in a single column in the database, using a unique key for each element. Here's how the keys are organized for the account and storage tries: 1. **Account Trie**: - Key: Account trie location. - Value: Account data associated with the key. 2. **Storage Trie**: - Key: Concatenation of the account hash and storage location. (We need to concatenate the hash with the location because otherwise, each account could write to the key of another account since they would all have keys like 0x, 0x00, etc. So we have to add the account hash to differentiate the entries in the database.) - Value: Storage data associated with the key. ### Flat Database The flat database is a storage optimization in Bonsai that stores all leaf nodes in a flat format. This design allows for quick and efficient access to accounts and storages, enhancing the performance of SLOAD operations. Instead of traversing the trie structure, the flat database enables direct access to the required data in a single disk read, resulting in faster query processing. The flat database is structured into three columns at the database level, storing data in a key-value format: 1. **Accounts Column**: - Stores account representations. - Key: Account Hash. - Value: Account data, including balance, nonce, and other relevant information. 2. **Storage Column**: - Stores storage slot representations. - Key: Concatenation of Account Hash and Slot Hash. - Value: Data associated with the storage slot. 3. **Code Column**: - Stores code representations. - Key: Account Hash. - Value: Bytecode associated with the account. ### Trielogs Trielogs are a key feature introduced by Bonsai to manage state rollback and rollforward. They represent the differences between two blocks' state by capturing each modification made to an account or storage. Trielogs contain the prior and next versions for each modification, allowing for state changes to be reverted or applied selectively. The trielogs are stored in a specific column in the database, using the block hash as the key. Each trielog contains the modified data along with the previous and current values for accounts, storage slots, and code. - **Key**: Block Hash. - **Value**: Trielog data, including the modified information, along with the previous and next values for accounts, storage slots, and code. ``` Block Hash: 0xabcdef1234567890... Trielog: - Account: - Account address: 0x1234567890abcdef... - Previous Value: { balance: 100, nonce: 0 , ...} - Next Value: { balance: 150, nonce: 1 , ...} - Storage Slot: - Account address: 0x1234567890abcdef... - Slot key: 0x0111... - Previous Value: 0x01 - Next Value: 0x02 - Code: - Account address: 0x1234567890abcdef... - Previous Value: 0xaabbccddeeff... - Next Value: 0x112233445566... ``` > Please note that this example is an illustrative representation of a trielog and may not exactly reflect the internal structure as it exists in the actual code. However, it provides a simplified understanding of how the data is organized within the trielog. ### Accumulator Bonsai introduces an object called the Bonsai accumulator, which accumulates modifications made during a block processing. It stores changes to accounts and storages, and at the end, these modifications are merged and used to update the trie and the flat database. This accumulator system allows for additional optimizations in Besu, such as preloading trie nodes during block processing. For example, if a read is detected in the flat db, a background task can be triggered to pre-fetch the path of that account in the trie. This preloading can reduce the time needed to update the trie at the end. #### Rollback/Rollfoward Since Bonsai only persists the head, it is necessary to have the ability to revert back to a previous state. This is achieved by passing one or more trielogs to the accumulator. The accumulator then processes the modifications recorded in the trielogs, either using the prior data in the case of a rollback or the next data in the case of a rollforward. Once all the modifications have been accumulated, the trie and flat database are updated as if importing a new block. By utilizing the trielogs and the accumulator, the system enables the capability to revert to previous states. This is particularly useful in situations where a rollback is required due to reorganization or other events. The accumulator efficiently processes the modifications recorded in the trielogs, allowing for the restoration of the state to a specific point in time. Applying a trielog for rollback or rollforward does not require any validation as in the case of a regular block import. If the trielog exists, it means that the validation has already been done previously because the trielog is created after the block import and its validation. Therefore, replaying the trielog will only change the values of the state and will be very fast. #### Preload cache The accumulator introduces an optimization mechanism as mentioned below. Its purpose is to detect modifications to storage slots or contracts during the execution of the EVM. To simplify, when reading from the flat database, a thread is launched to put in memory the path of that leaf node in the trie. This allows us to have the majority, if not all, of the trie nodes already in memory when updating the trie to calculate the root hash. As a result, it reduces the time spent waiting for disk access and improves performance. By utilizing the accumulator, the system optimizes the process of updating the trie by preloading and caching the necessary leaf nodes. This reduces the reliance on disk access and improves the overall efficiency of the state management system. ### Worldstate Worldstate Components The worldstate consists of several components, including: 1. Storage: The storage component determines how data is stored and persisted. It can be either persistent, based on a RocksDB snapshot , or using a layered. Each option has its own characteristics and benefits. 2. Root Hash and Block Hash: The root hash and block hash of the state are derived from the storage component. 3. Accumulator: The accumulator is a data structure that maintains and manages a set of modifications made to the worldstate. It allows performing rollback or rollforward operations on the state. For example, an accumulator can record changes to the data over time, enabling selective undoing of changes to revert to a previous state or applying pending modifications to advance to a new state. Each instance of Worldstate has its own accumulator, allowing it to manage rollbacks on different states in a thread-safe manner. 4. Persistence Flag: The persistence flag is a boolean value indicating whether the worldstate is allowed to persist modifications during a rollback or rollforward operation. As Bonsai only persists a single state in the database we need to add various optimizations in Besu to support multiple copies of the state at different height simultaneously. This enables handling scenarios where, for example, an RPC request requires the account balance at block 4 and another request needs it at block 5 concurrently. This is achieved through a combination of rocksDB snapshot and in-memory diff layers. The persisted, snapshot, and layered states work together to manage changes in the state. Periodically, a snapshot of the state is taken, capturing the state at a specific point in time. Each addition of a new block generates a layered diff in memory, representing the changes introduced by that block. Once a sufficient number of blocks have been processed, a new snapshot is created. This approach prevents the accumulation of an infinite list of diff layers and provides a fresh starting point for future modifications. Additionally, when a block is finalized and considered permanent, the state is persisted to maintain the most up-to-date version of the state. To summarize we have different types of world state representation: - Persisted state: This refers to the state representation that is persisted in a database. It represents the state at a specific point in time and is stored persistently for future reference. - Snapshot state: A snapshot state is a copy of the persisted one at a particular moment. It provides a view of the state at that specific point in time. - Layered state: Layered state refers to a snapshot to which an in-memory diff is applied. This diff represents the changes or updates made to the snapshot. The layered state allows for efficient and incremental updates to the snapshot without modifying the original data directly. A layered state is different from a trielog. It contains the differences between two blocks of the entire state, including leaf nodes, branch nodes and extension nodes, while a trielog only has the state before and after only of the leaf. ``` Trielog: - Account: - Account address: 0x1234567890abcdef... - Previous Value: { balance: 100, nonce: 0 , ...} - Next Value: { balance: 150, nonce: 1 , ...} - Storage Slot: - Account address: 0x1234567890abcdef... - Slot key: 0x0111... - Previous Value: 0x01 - Next Value: 0x02 - Code: - Account address: 0x1234567890abcdef... - Previous Value: 0xaabbccddeeff... - Next Value: 0x112233445566... ``` VS ``` Layered state 0x <> Branch node present in the layered 0x01 <> Branch node present in the layered 0x0102 <> Leaf node present in the layered 0x02 <> Branch node is missing so will ask the parent state ``` To say it differently, the trielog provides a view of the modifications made between two blocks at the account and storage levels. On the other hand, the layered state is more a view of how these modifications have impacted the storage of the worldstate (both flat and trie). ![](https://hackmd.io/_uploads/B1ikEKD8n.png) ## Lifecyle of the worldstate To protect the integrity of the head from inconsistencies, we never use the persisted state for simulating transactions, validating blocks, creating blocks, or responding to RPC calls. The persisted state is only used to import a new block and change the head of Besu. It's important to note that we always have a copy of the persisted state in a snapshot. This last one is created right after the import or at the start of Besu.Choosing the right moment to create a snapshot of the persisted one is crucial. We want to ensure that we create a snapshot of a state that is not currently being modified. Creating a snapshot during a block import could result in an invalid snapshot that represents a transitional state between two blocks. To avoid this, it's important to carefully synchronize the snapshot creation process with the block processing workflow. Typically, snapshots are created after the completion of a block processing cycle. This ensures that all modifications made during the block execution are finalized, and the state has reached a stable point. Now, let's say I want to simulate a transaction on the state of block 4 while the head is at block 9. What do we do? Every time we need a world state, we will request an instance of BonsaiWorldstate from the world state provider. If it's for something other than changing the head, the provider will look into its cache to check if there is an available image of world state corresponding to the requested block: ### The worldstate is available in the cache When retrieving a world state, the system follows a copy-on-access approach. If an image of the world state instance exists in the cache, it creates a copy of it and returns the copy. Why do we make a copy? The purpose is to preserve a clean and intact world state image in the cache, which can be utilized to serve other requests if necessary. If the retrieved world state is a snapshot, a copy of the snapshot is returned. Similarly, if it's a layered, a copy of the layered is returned. With this copy, we can perform simulations or any other operations as needed. It's important to note that this copy is never persisted and remains unique to our specific call, eliminating any potential concurrency issues. <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentstyletype="text/css" height="340px" preserveAspectRatio="none" style="width:497px;height:340px;background:#FFFFFF;" version="1.1" viewBox="0 0 497 340" width="497px" zoomAndPan="magnify"><defs></defs><g><rect fill="#FFFFFF" height="129.5313" style="stroke:#181818;stroke-width:1.0;" width="10" x="255.5" y="112.4297"></rect><rect fill="#FFFFFF" height="100.3984" style="stroke:#181818;stroke-width:1.0;" width="10" x="457.5" y="141.5625"></rect><line style="stroke:#181818;stroke-width:0.5;stroke-dasharray:5.0,5.0;" x1="28" x2="28" y1="81.2969" y2="259.9609"></line><line style="stroke:#181818;stroke-width:0.5;stroke-dasharray:5.0,5.0;" x1="260.5" x2="260.5" y1="81.2969" y2="259.9609"></line><line style="stroke:#181818;stroke-width:0.5;stroke-dasharray:5.0,5.0;" x1="462.5" x2="462.5" y1="81.2969" y2="259.9609"></line><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="41" x="5" y="77.9951">Caller</text><ellipse cx="28.5" cy="13.5" fill="#E2E2F0" rx="8" ry="8" style="stroke:#181818;stroke-width:0.5;"></ellipse><path d="M28.5,21.5 L28.5,48.5 M15.5,29.5 L41.5,29.5 M28.5,48.5 L15.5,63.5 M28.5,48.5 L41.5,63.5 " fill="none" style="stroke:#181818;stroke-width:0.5;"></path><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="41" x="5" y="271.9561">Caller</text><ellipse cx="28.5" cy="283.7578" fill="#E2E2F0" rx="8" ry="8" style="stroke:#181818;stroke-width:0.5;"></ellipse><path d="M28.5,291.7578 L28.5,318.7578 M15.5,299.7578 L41.5,299.7578 M28.5,318.7578 L15.5,333.7578 M28.5,318.7578 L41.5,333.7578 " fill="none" style="stroke:#181818;stroke-width:0.5;"></path><rect fill="#E2E2F0" height="30.2969" rx="2.5" ry="2.5" style="stroke:#181818;stroke-width:0.5;" width="156" x="182.5" y="50"></rect><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="142" x="189.5" y="69.9951">WorldStateProvider</text><rect fill="#E2E2F0" height="30.2969" rx="2.5" ry="2.5" style="stroke:#181818;stroke-width:0.5;" width="156" x="182.5" y="258.9609"></rect><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="142" x="189.5" y="278.9561">WorldStateProvider</text><rect fill="#E2E2F0" height="30.2969" rx="2.5" ry="2.5" style="stroke:#181818;stroke-width:0.5;" width="58" x="433.5" y="50"></rect><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="44" x="440.5" y="69.9951">Cache</text><rect fill="#E2E2F0" height="30.2969" rx="2.5" ry="2.5" style="stroke:#181818;stroke-width:0.5;" width="58" x="433.5" y="258.9609"></rect><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="44" x="440.5" y="278.9561">Cache</text><rect fill="#FFFFFF" height="129.5313" style="stroke:#181818;stroke-width:1.0;" width="10" x="255.5" y="112.4297"></rect><rect fill="#FFFFFF" height="100.3984" style="stroke:#181818;stroke-width:1.0;" width="10" x="457.5" y="141.5625"></rect><polygon fill="#181818" points="243.5,108.4297,253.5,112.4297,243.5,116.4297,247.5,112.4297" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;" x1="28.5" x2="249.5" y1="112.4297" y2="112.4297"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="187" x="35.5" y="107.3638">Request WorldState (block 4)</text><polygon fill="#181818" points="445.5,137.5625,455.5,141.5625,445.5,145.5625,449.5,141.5625" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;" x1="265.5" x2="451.5" y1="141.5625" y2="141.5625"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="173" x="272.5" y="136.4966">Check WorldState (block 4)</text><polygon fill="#181818" points="276.5,166.6953,266.5,170.6953,276.5,174.6953,272.5,170.6953" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;stroke-dasharray:2.0,2.0;" x1="270.5" x2="456.5" y1="170.6953" y2="170.6953"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="130" x="282.5" y="165.6294">WorldState (block 4)</text><line style="stroke:#181818;stroke-width:1.0;" x1="265.5" x2="307.5" y1="199.8281" y2="199.8281"></line><line style="stroke:#181818;stroke-width:1.0;" x1="307.5" x2="307.5" y1="199.8281" y2="212.8281"></line><line style="stroke:#181818;stroke-width:1.0;" x1="266.5" x2="307.5" y1="212.8281" y2="212.8281"></line><polygon fill="#181818" points="276.5,208.8281,266.5,212.8281,276.5,216.8281,272.5,212.8281" style="stroke:#181818;stroke-width:1.0;"></polygon><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="167" x="272.5" y="194.7622">Copy WorldState (block 4)</text><polygon fill="#181818" points="39.5,237.9609,29.5,241.9609,39.5,245.9609,35.5,241.9609" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;stroke-dasharray:2.0,2.0;" x1="33.5" x2="259.5" y1="241.9609" y2="241.9609"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="203" x="45.5" y="236.895">Return WorldState (block 4 bis)</text></g></svg> ### The worldstate is not available in the cache If the requested world state is not available in the cache, the world state provider will need to recreate an image of that world state. Here's how it will proceed: 1. It will locate the nearest snapshot/layered in the cache that corresponds to a block close to the requested block. 2. The provider will make a copy of that snapshot/layered and perform rollback or rollforward operations to reach the state of the requested block. 3. Once it has obtained the desired state, it will save this copy for future requests and make another copy to return to the caller. In summary, the world state provider uses the closest available snapshot/layered as a starting point and applies necessary changes to reach the specific block's state. The copied world state is then stored for caching purposes and a separate copy is returned to the caller. <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentstyletype="text/css" height="484px" preserveAspectRatio="none" style="width:549px;height:484px;background:#FFFFFF;" version="1.1" viewBox="0 0 549 484" width="549px" zoomAndPan="magnify"><defs></defs><g><rect fill="#FFFFFF" height="273.0625" style="stroke:#181818;stroke-width:1.0;" width="10" x="255.5" y="112.4297"></rect><rect fill="#FFFFFF" height="243.9297" style="stroke:#181818;stroke-width:1.0;" width="10" x="509.5" y="141.5625"></rect><line style="stroke:#181818;stroke-width:0.5;stroke-dasharray:5.0,5.0;" x1="28" x2="28" y1="81.2969" y2="403.4922"></line><line style="stroke:#181818;stroke-width:0.5;stroke-dasharray:5.0,5.0;" x1="260.5" x2="260.5" y1="81.2969" y2="403.4922"></line><line style="stroke:#181818;stroke-width:0.5;stroke-dasharray:5.0,5.0;" x1="514.5" x2="514.5" y1="81.2969" y2="403.4922"></line><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="41" x="5" y="77.9951">Caller</text><ellipse cx="28.5" cy="13.5" fill="#E2E2F0" rx="8" ry="8" style="stroke:#181818;stroke-width:0.5;"></ellipse><path d="M28.5,21.5 L28.5,48.5 M15.5,29.5 L41.5,29.5 M28.5,48.5 L15.5,63.5 M28.5,48.5 L41.5,63.5 " fill="none" style="stroke:#181818;stroke-width:0.5;"></path><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="41" x="5" y="415.4873">Caller</text><ellipse cx="28.5" cy="427.2891" fill="#E2E2F0" rx="8" ry="8" style="stroke:#181818;stroke-width:0.5;"></ellipse><path d="M28.5,435.2891 L28.5,462.2891 M15.5,443.2891 L41.5,443.2891 M28.5,462.2891 L15.5,477.2891 M28.5,462.2891 L41.5,477.2891 " fill="none" style="stroke:#181818;stroke-width:0.5;"></path><rect fill="#E2E2F0" height="30.2969" rx="2.5" ry="2.5" style="stroke:#181818;stroke-width:0.5;" width="156" x="182.5" y="50"></rect><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="142" x="189.5" y="69.9951">WorldStateProvider</text><rect fill="#E2E2F0" height="30.2969" rx="2.5" ry="2.5" style="stroke:#181818;stroke-width:0.5;" width="156" x="182.5" y="402.4922"></rect><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="142" x="189.5" y="422.4873">WorldStateProvider</text><rect fill="#E2E2F0" height="30.2969" rx="2.5" ry="2.5" style="stroke:#181818;stroke-width:0.5;" width="58" x="485.5" y="50"></rect><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="44" x="492.5" y="69.9951">Cache</text><rect fill="#E2E2F0" height="30.2969" rx="2.5" ry="2.5" style="stroke:#181818;stroke-width:0.5;" width="58" x="485.5" y="402.4922"></rect><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="44" x="492.5" y="422.4873">Cache</text><rect fill="#FFFFFF" height="273.0625" style="stroke:#181818;stroke-width:1.0;" width="10" x="255.5" y="112.4297"></rect><rect fill="#FFFFFF" height="243.9297" style="stroke:#181818;stroke-width:1.0;" width="10" x="509.5" y="141.5625"></rect><polygon fill="#181818" points="243.5,108.4297,253.5,112.4297,243.5,116.4297,247.5,112.4297" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;" x1="28.5" x2="249.5" y1="112.4297" y2="112.4297"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="187" x="35.5" y="107.3638">Request WorldState (block 4)</text><polygon fill="#181818" points="497.5,137.5625,507.5,141.5625,497.5,145.5625,501.5,141.5625" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;" x1="265.5" x2="503.5" y1="141.5625" y2="141.5625"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="173" x="272.5" y="136.4966">Check WorldState (block 4)</text><polygon fill="#181818" points="276.5,166.6953,266.5,170.6953,276.5,174.6953,272.5,170.6953" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;stroke-dasharray:2.0,2.0;" x1="270.5" x2="508.5" y1="170.6953" y2="170.6953"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="157" x="282.5" y="165.6294">WorldState not available</text><polygon fill="#181818" points="497.5,195.8281,507.5,199.8281,497.5,203.8281,501.5,199.8281" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;" x1="265.5" x2="503.5" y1="199.8281" y2="199.8281"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="220" x="272.5" y="194.7622">Check closest worldState (block 4)</text><polygon fill="#181818" points="276.5,224.9609,266.5,228.9609,276.5,232.9609,272.5,228.9609" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;stroke-dasharray:2.0,2.0;" x1="270.5" x2="508.5" y1="228.9609" y2="228.9609"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="130" x="282.5" y="223.895">WorldState (block 9)</text><line style="stroke:#181818;stroke-width:1.0;" x1="265.5" x2="307.5" y1="258.0938" y2="258.0938"></line><line style="stroke:#181818;stroke-width:1.0;" x1="307.5" x2="307.5" y1="258.0938" y2="271.0938"></line><line style="stroke:#181818;stroke-width:1.0;" x1="266.5" x2="307.5" y1="271.0938" y2="271.0938"></line><polygon fill="#181818" points="276.5,267.0938,266.5,271.0938,276.5,275.0938,272.5,271.0938" style="stroke:#181818;stroke-width:1.0;"></polygon><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="143" x="272.5" y="253.0278">Rollback (block 9 -> 4)</text><polygon fill="#181818" points="497.5,296.2266,507.5,300.2266,497.5,304.2266,501.5,300.2266" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;" x1="265.5" x2="503.5" y1="300.2266" y2="300.2266"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="159" x="272.5" y="295.1606">Add WorldState (block 4)</text><polygon fill="#181818" points="276.5,310.2266,266.5,314.2266,276.5,318.2266,272.5,314.2266" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;stroke-dasharray:2.0,2.0;" x1="270.5" x2="508.5" y1="314.2266" y2="314.2266"></line><line style="stroke:#181818;stroke-width:1.0;" x1="265.5" x2="307.5" y1="343.3594" y2="343.3594"></line><line style="stroke:#181818;stroke-width:1.0;" x1="307.5" x2="307.5" y1="343.3594" y2="356.3594"></line><line style="stroke:#181818;stroke-width:1.0;" x1="266.5" x2="307.5" y1="356.3594" y2="356.3594"></line><polygon fill="#181818" points="276.5,352.3594,266.5,356.3594,276.5,360.3594,272.5,356.3594" style="stroke:#181818;stroke-width:1.0;"></polygon><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="165" x="272.5" y="338.2935">copy WorldState (block 4)</text><polygon fill="#181818" points="39.5,381.4922,29.5,385.4922,39.5,389.4922,35.5,385.4922" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;stroke-dasharray:2.0,2.0;" x1="33.5" x2="259.5" y1="385.4922" y2="385.4922"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="203" x="45.5" y="380.4263">Return WorldState (block 4 bis)</text></g></svg> ### Closing the worldtate Whenever we retrieve a snapshot/layered state, it is essential to ensure that it is closed properly to avoid any memory leaks. This means that whenever we create a block or simulate a transaction, we need to close the corresponding world state. To prevent any oversight in closing it, our code enforces strict rules, and we are not allowed to manually open it within an RPC method or similar contexts. The responsibility of opening and closing the world state is delegated to the TransactionSimulator, MainnetBlockValidator, and BlockCreator, which minimizes the chances of future developers forgetting to close it. We strive to maintain this rigor, and certain parts of the code explicitly prevent opening a world state anywhere else to avoid errors (although we acknowledge that mistakes can still happen, and we aim to protect against them as much as possible). All the world states we create need to be closed at some point or another. For example, if we need a world state to validate a block, the provider will provide us a copy of the cache. Once the validator finishes using this copy, it will close it. The world state that is in the cache and allows creating copies will be closed once it is evicted from the cache either because it is too old or because we haven't used it for a long time. Why do we need to close the world states? It's because the snapshots of RocksDB, if left open, can create a memory leak. A layered world state, for example, relies on a snapshot. So if a layered world state is in the cache but the snapshot is evicted, we still need to keep the snapshot open. The layered world state will keep it open, but as soon as the snapshot no longer has any open dependencies (all layered using this one are closed and it is evicted from the cache), it will be closed. ![](https://hackmd.io/_uploads/B1ikEKD8n.png) However, this approach has a downside. Once a simulation or validation is terminated, we cannot access data from the world state anymore. To address this limitation, most methods accept an interface where the caller can provide their code to be executed if they need to access world state information before it is closed. This method is invoked just before the close, allowing the caller to retain control over the world state data. Overall, this strict control over opening and closing world states ensures memory management and minimizes the risk of errors while still providing a mechanism for accessing data when necessary. <svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" contentstyletype="text/css" height="484px" preserveAspectRatio="none" style="width:741px;height:484px;background:#FFFFFF;" version="1.1" viewBox="0 0 741 484" width="741px" zoomAndPan="magnify"><defs></defs><g><rect fill="#FFFFFF" height="282.0625" style="stroke:#181818;stroke-width:1.0;" width="10" x="184.5" y="112.4297"></rect><rect fill="#FFFFFF" height="214.7969" style="stroke:#181818;stroke-width:1.0;" width="10" x="480.5" y="141.5625"></rect><rect fill="#FFFFFF" height="29.1328" style="stroke:#181818;stroke-width:1.0;" width="10" x="652" y="170.6953"></rect><line style="stroke:#181818;stroke-width:0.5;stroke-dasharray:5.0,5.0;" x1="28" x2="28" y1="81.2969" y2="403.4922"></line><line style="stroke:#181818;stroke-width:0.5;stroke-dasharray:5.0,5.0;" x1="189" x2="189" y1="81.2969" y2="403.4922"></line><line style="stroke:#181818;stroke-width:0.5;stroke-dasharray:5.0,5.0;" x1="485" x2="485" y1="81.2969" y2="403.4922"></line><line style="stroke:#181818;stroke-width:0.5;stroke-dasharray:5.0,5.0;" x1="657" x2="657" y1="81.2969" y2="403.4922"></line><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="41" x="5" y="77.9951">Caller</text><ellipse cx="28.5" cy="13.5" fill="#E2E2F0" rx="8" ry="8" style="stroke:#181818;stroke-width:0.5;"></ellipse><path d="M28.5,21.5 L28.5,48.5 M15.5,29.5 L41.5,29.5 M28.5,48.5 L15.5,63.5 M28.5,48.5 L41.5,63.5 " fill="none" style="stroke:#181818;stroke-width:0.5;"></path><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="41" x="5" y="415.4873">Caller</text><ellipse cx="28.5" cy="427.2891" fill="#E2E2F0" rx="8" ry="8" style="stroke:#181818;stroke-width:0.5;"></ellipse><path d="M28.5,435.2891 L28.5,462.2891 M15.5,443.2891 L41.5,443.2891 M28.5,462.2891 L15.5,477.2891 M28.5,462.2891 L41.5,477.2891 " fill="none" style="stroke:#181818;stroke-width:0.5;"></path><rect fill="#E2E2F0" height="30.2969" rx="2.5" ry="2.5" style="stroke:#181818;stroke-width:0.5;" width="105" x="137" y="50"></rect><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="91" x="144" y="69.9951">RPCMethods</text><rect fill="#E2E2F0" height="30.2969" rx="2.5" ry="2.5" style="stroke:#181818;stroke-width:0.5;" width="105" x="137" y="402.4922"></rect><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="91" x="144" y="422.4873">RPCMethods</text><rect fill="#E2E2F0" height="30.2969" rx="2.5" ry="2.5" style="stroke:#181818;stroke-width:0.5;" width="167" x="402" y="50"></rect><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="153" x="409" y="69.9951">TransactionSimulator</text><rect fill="#E2E2F0" height="30.2969" rx="2.5" ry="2.5" style="stroke:#181818;stroke-width:0.5;" width="167" x="402" y="402.4922"></rect><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="153" x="409" y="422.4873">TransactionSimulator</text><rect fill="#E2E2F0" height="30.2969" rx="2.5" ry="2.5" style="stroke:#181818;stroke-width:0.5;" width="156" x="579" y="50"></rect><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="142" x="586" y="69.9951">WorldStateProvider</text><rect fill="#E2E2F0" height="30.2969" rx="2.5" ry="2.5" style="stroke:#181818;stroke-width:0.5;" width="156" x="579" y="402.4922"></rect><text fill="#000000" font-family="sans-serif" font-size="14" lengthAdjust="spacing" textLength="142" x="586" y="422.4873">WorldStateProvider</text><rect fill="#FFFFFF" height="282.0625" style="stroke:#181818;stroke-width:1.0;" width="10" x="184.5" y="112.4297"></rect><rect fill="#FFFFFF" height="214.7969" style="stroke:#181818;stroke-width:1.0;" width="10" x="480.5" y="141.5625"></rect><rect fill="#FFFFFF" height="29.1328" style="stroke:#181818;stroke-width:1.0;" width="10" x="652" y="170.6953"></rect><polygon fill="#181818" points="172.5,108.4297,182.5,112.4297,172.5,116.4297,176.5,112.4297" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;" x1="28.5" x2="178.5" y1="112.4297" y2="112.4297"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="125" x="35.5" y="107.3638">processTransaction</text><polygon fill="#181818" points="468.5,137.5625,478.5,141.5625,468.5,145.5625,472.5,141.5625" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;" x1="194.5" x2="474.5" y1="141.5625" y2="141.5625"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="125" x="201.5" y="136.4966">processTransaction</text><polygon fill="#181818" points="640,166.6953,650,170.6953,640,174.6953,644,170.6953" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;" x1="490.5" x2="646" y1="170.6953" y2="170.6953"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="128" x="497.5" y="165.6294">Request WorldState</text><polygon fill="#181818" points="501.5,195.8281,491.5,199.8281,501.5,203.8281,497.5,199.8281" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;stroke-dasharray:2.0,2.0;" x1="495.5" x2="656" y1="199.8281" y2="199.8281"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="118" x="507.5" y="194.7622">Return WorldState</text><line style="stroke:#181818;stroke-width:1.0;" x1="490.5" x2="532.5" y1="228.9609" y2="228.9609"></line><line style="stroke:#181818;stroke-width:1.0;" x1="532.5" x2="532.5" y1="228.9609" y2="241.9609"></line><line style="stroke:#181818;stroke-width:1.0;" x1="491.5" x2="532.5" y1="241.9609" y2="241.9609"></line><polygon fill="#181818" points="501.5,237.9609,491.5,241.9609,501.5,245.9609,497.5,241.9609" style="stroke:#181818;stroke-width:1.0;"></polygon><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="129" x="497.5" y="223.895">process Transaction</text><polygon fill="#181818" points="205.5,267.0938,195.5,271.0938,205.5,275.0938,201.5,271.0938" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;" x1="199.5" x2="479.5" y1="271.0938" y2="271.0938"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="262" x="211.5" y="266.0278">execude Code Before Closing Worldstate</text><polygon fill="#181818" points="468.5,281.0938,478.5,285.0938,468.5,289.0938,472.5,285.0938" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;stroke-dasharray:2.0,2.0;" x1="194.5" x2="474.5" y1="285.0938" y2="285.0938"></line><line style="stroke:#181818;stroke-width:1.0;" x1="490.5" x2="532.5" y1="314.2266" y2="314.2266"></line><line style="stroke:#181818;stroke-width:1.0;" x1="532.5" x2="532.5" y1="314.2266" y2="327.2266"></line><line style="stroke:#181818;stroke-width:1.0;" x1="491.5" x2="532.5" y1="327.2266" y2="327.2266"></line><polygon fill="#181818" points="501.5,323.2266,491.5,327.2266,501.5,331.2266,497.5,327.2266" style="stroke:#181818;stroke-width:1.0;"></polygon><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="107" x="497.5" y="309.1606">close Worldstate</text><polygon fill="#181818" points="205.5,352.3594,195.5,356.3594,205.5,360.3594,201.5,356.3594" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;" x1="199.5" x2="484.5" y1="356.3594" y2="356.3594"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="158" x="211.5" y="351.2935">return Transaction result</text><polygon fill="#181818" points="39.5,381.4922,29.5,385.4922,39.5,389.4922,35.5,385.4922" style="stroke:#181818;stroke-width:1.0;"></polygon><line style="stroke:#181818;stroke-width:1.0;" x1="33.5" x2="183.5" y1="385.4922" y2="385.4922"></line><text fill="#000000" font-family="sans-serif" font-size="13" lengthAdjust="spacing" textLength="132" x="45.5" y="380.4263">return RPC response</text></g></svg> ## Verkle trie with Bonsai Bonsai is not tied to a specific trie structure. It manages how we store and retrieve trie nodes. It also allows us to go back in time by saving different account and storage modifications of each block. From this perspective, we realize that Bonsai is not necessarily linked to the type of trie we use. Therefore, we can consider transitioning to the Verkle trie without having to rewrite all the optimizations provided by Bonsai or rethink the entire logic. This also shows that the performance work we are currently doing on Bonsai will not be lost. In Bonsai, the accumulator plays a crucial role in managing and tracking the modifications made to the trie. It acts as a temporary storage for these changes before they are committed to the underlying trie structure (PMT or Verkle trie). During and after the transition from the PMT to the Verkle trie, the accumulator remains an integral part of the process. The modifications made to the trie are first accumulated in the accumulator, similar to how it works in the current Bonsai implementation. However, instead of directly committing these modifications to the PMT, they are now pushed and applied to the Verkle trie. The accumulator ensures that the changes made to the trie are appropriately recorded and can be efficiently applied to the Verkle trie when necessary. It helps maintain consistency and facilitates the process of transitioning from one trie structure to another without losing the modifications. To summarize, the accumulator remains involved in the transition process and after, allowing for the temporary storage and subsequent application of modifications to the Verkle trie, replacing the role it previously played with the PMT in the Bonsai framework.