Portal Network Implementers Call Notes

# Portal Network Implementers Call Notes ## May 19th, 2025 - Call #56 ### 1. Team update - Update by [Piper](https://github.com/pipermerriam) - Preparing a co-authored document with [Jason](https://github.com/carver) that outlines Portal’s role in the L1 scaling roadmap - Inviting teams to review and provide feedback prior to the public release - Update on [Trin](https://github.com/ethereum/trin) & [Glados](https://github.com/ethereum/glados) by [Michael](https://github.com/mrferris) - Internal refactoring - Added new docker image for trin-execution - Moved consensus-related constants into ethportal-api - Added trace offer endpoint which support multiple content keys - Deployed e2hs bridges for data bridging - Glados: UTP transfer failures are now visible, aiding network health tracking - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [ScottyPoi](https://github.com/ScottyPoi) - Boot nodes redeployed with some issues to be fixed - Merged updated SSZ container for HistoricalSummariesStateProof, resolving Hive test failures - Update on [Nimbus Portal Client](https://github.com/status-im/nimbus-eth1/tree/master/portal) (formerly Fluffy) by [Kim](https://github.com/kdeme) - Added access from History network to historical summaries - Updated state bridge for more efficient gossiping - Fixed a bug in FindContent responses - Working on rate-limiting to better monitor gossip success rates - Renaming from "Fluffy" to "Nimbus Portal Client" underway across code and infrastructure - Dual support ("f" as Fluffy and "n" as Nimbus Portal) for transition phase in ENRs - Update on [Shisui](https://github.com/zen-eth/shisui) by [Qi Zhou](https://github.com/qizhou) - Performance improvements via refactoring discv5 to pipeline the reloop and dispatch loop asynchronously - Fixed memory leak issues and block boundary check error - Refactored validation logic to reduce redundancy - Implemented support for retrieving ephemeral header from storage (code currently under review) - Update on [Samba](https://github.com/meldsun0/samba) by [Derek](https://github.com/Dsorken) - Implemented the protocol versioning incl. accept codes and utp size prefix - Fixed an issue in history offer logic with large transfers - Integration with Besu - Improved logs and added parameter options for running Samba inside Besu - Made all Portal History JSON-RPC calls available as method calls for Java projects - Basic Besu node now operational with Samba integration - Plan to implement the beaconStore endpoint and Capella validation this week ### 2. Discussion topics #### 2.1 Beacon Spec Update - [PR](https://github.com/ethereum/portal-network-specs/pull/403) was submitted based on last week’s discussion - Open for team review before merging #### 2.2 Ephemeral Header Gossiping Mechanism Discussion - The problem: Current doc suggests neighborhood gossip for ephemeral headers, but this may - Restrict propagation to nodes near the content ID - Risk incomplete network coverage - Team consensus: Random gossip preferred over neighborhood gossip - Neighborhood gossip filters peers based on radius related to content ID, which is irrelevant for ephemeral headers - Random gossip can be adjusted to prioritize peers closer to the sender, but should spread widely to ensure all nodes receive the headers - Action item - Submit PR to update the spec to clarify random gossip as the mechanism for ephemeral headers ## May 12th, 2025 - Call #55 ### 1. Team update - Update on [Trin](https://github.com/ethereum/trin) & [Glados](https://github.com/ethereum/glados) by [Milos](https://github.com/morph-dev) - Implemented support for Pectra types on the Beacon network - Started adding initial support for Sepolia - Ongoing work on implementing the bridge that will follow the chain head and generate e2hs files - Glados: Updated areaGraph for cleaner visualization - Update on [Fluffy](https://github.com/status-im/nimbus-eth1/tree/master/fluffy) by [Kim](https://github.com/kdeme) - Fixed a bug in HistoricalSummaries storage in beacon_db - Explored experimental support for providing HistoricalSummaries as an Oracle to the History network (currently in testing, not merged yet) - Implemented several performance improvements on State network for synchronous data processing - Removed deprecated neighborhoodGossip logic - Proposed PR to clarify that HistoricalSummaries must be from finalized beacon state - Nodes should only respond to HistoricalSummaries requests if they have data for the requested epoch or newer - Otherwise, nodes should rely with an empty response (pending further clarification in the specs) - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [ScottyPoi](https://github.com/ScottyPoi) - Progress on HistoricalSummaries and updating SSZ types - Continued implementation of ephemeral headers - Started implementing typing and distance functions of Head-MPT State Network - Explored support for Sepolia recognized challenges in supporting multiple networks (mainnet & testnet) simultaneously - Update on [Shisui](https://github.com/zen-eth/shisui) by [Qi Zhou](https://github.com/qizhou) - Fixed Hive test failures by addressing missing validation for historical network in findContent, especially for post-Shanghai data - Addressed a lightclient error due to an JSON-RPC typo - Split the ephemeral header PR into smaller parts for easier review - Implemented & merged the storage part - Remaining features will be implemented in the coming weeks - Created a new benchmark to compare performance between Trin vs Shisui using the same payload - Showed that discv5 from Geth implementation has significant overhead and slower compared raw uTP implementation - The benchmark tests are currently ad hoc, but can potentially be modularized and reused across clients for performance debugging/ validation - Update on [Samba](https://github.com/meldsun0/samba) by [Meld](https://github.com/meldsun0) - Completed the accumulator functionality - Ongoing work on protocol versioning and accept code - Besu integration encountered some blockers (actively being addressed) ### 2. Discussion topics #### 2.1 LightClientUpdatesByRange Behavior Change Proposal - Current Spec: If a node is missing any update in a requested epoch range, it must not reply at all - Proposed Change: Nodes should reply with available updates starting from the earliest in the range, even if incomplete - Rationale: Syncing nodes benefit from partial progress (getting closer to chain head) - Impact: Able to avoids scenarios where nodes withhold all updates due to missing only the latest update - Team Consensus - No objections; PR to be submitted to formalize the changes #### 2.2 LightClientOptimisticUpdate Request/ Response Logic Optimization - Current Spec: Nodes request optimistic updates based on local clock’s current slot - Proposed Changes - Request Logic: Nodes should request the first missing slot (not just latest) if behind - Response Logic: Peers should always reply with their latest optimistic update, even if it's newer than the requested one - Example: If a peer stores only slot 100 but receives a request for slot 99, it should reply with slot 100 - Key discussion points - Some clients (eg. Fluffy, Ultralight) store only the latest optimistic update, so replying with older ones isn't feasible - There's consensus to avoid introducing new content keys to existing data structure unless absolutely necessary to avoid complexity and redundancy - Team Consensus - General agreement reached; PR to be submitted to formalize the changes #### 2.3 Config Standardization for Testnets - Unified Config Approach - Proposed a unified YAML config format combining CL, EL, and Portal-specific fields to simplify setup - Can support presets or custom paths - Automation & Maintenance - Generate configs from upstream (CL/EL repo) to minimize manual updates - Tradeoff between simplified UX vs maintenance overhead - Possible to automate yaml generation to reduce maintenance burden ## May 5th, 2025 - Call #54 ### 1. Team update - Update on [Trin](https://github.com/ethereum/trin) & [Glados](https://github.com/ethereum/glados) by [Kolby](https://github.com/KolbyML) - Bug fixes & improvements - Fixed a bug related to gossiping content to nodes outside their radius - Replaced hardcoded fork activations with dynamic network config - Integrated Trin as a library, removing slow HTTP JSON-RPC and redundant tests to improve performance and dev experience - Enabled HistoricalSummaries for the Beacon Bridge - Added Deneb header proof type, generation code, and Deneb header validation - Added Electra types and created header proof using BeaconBlockElectra - In progress - PR to add HistoricalSummariesWithProof for Electra - PR to generate e2hs files - Glados：Added internal transfer failure chart - Update on [Fluffy](https://github.com/status-im/nimbus-eth1/tree/master/fluffy) by [Kim](https://github.com/kdeme) - Bug fixes & improvements - Improved eth_createAccessList to return access lists even when the EVM execution reverts - Ensured only a single instance of AsyncEvm created and shared across API handlers - Added architecture docs to the Fluffy guide - Tested & activiated block proof tests - Implemented & tested the adjustment of historical_summaries proof for Electra - Update on [Samba](https://github.com/meldsun0/samba) by [Meld](https://github.com/meldsun0) - Continued implementing discv5 and awaiting upstream changes from Besu in the discv5 library - Continued implementing history endpoints, eg. historyTraceGetContent, historyRecursiveFindNodes - Nearly done with accumulator integration - Working toward release a Samba lib to be integrated into Besu client - Commit to complete the History network checklist soon - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [acolytec3](https://github.com/acolytec3) - Focused on implementing ephemeral headers, with progress on validation and purging old offers - Will work on historical summaries updates after current ephemeral headers implementation - Update on [Shisui](https://github.com/zen-eth/shisui) by [Grapebaba](https://github.com/GrapeBaBa) - Converted ExecutionBlockProof from list to vectors - Added recursiveFindNodes in putContent logic - Fixed bug in ping extension upgrades - Working on ephemeral headers implementation ### 2. Discussion topics #### 2.1 History Network Update - Most client teams progressing well on implementation; No major blockers - Link of History Network meta tracking issue: https://github.com/ethereum/portal-network-specs/issues/398 #### 2.2 Running Portal in Kurtosis - Goal - Enable Portal Network clients to run inside Kurtosis, for better devnet/ testnet support and integration testing - Initial target - Run local, small-scale devnets to experiment with setup, discover limitations, and determine necessary client-side changes - Topics discussed - Networking design consideration - Need for large super-nodes vs many small nodes in testing - Use of ephemeral devnets with isolated protocol IDs to avoid collisions - Flagged potential issues with ephemeral headers as client teams are still finalizing their implementation - Bridgework: Need mechanisms to insert historical data into the devnets for testing purposes - Protocol ID/ Chain ID: Discussed potentially using chain IDs for Kurtosis devnets, and how to map them to protocol IDs in Portal clients - Config consideration - Reuse existing EL/CL configs - Add Portal-specific settings via standardized spec file (shared across clients) and optional client-specific extensions - Collaboration - Kolby (Trin) - Will act as the point of contact from Portal devs - Kim (Fluffy) - Expressed interest to support Fluffy within Kurtosis - Barnabas (ethPandaOps) - Will provide Kurtosis integration support - Other teams/ clients are welcome and encouraged to contribute - Action items - Coordinate PRs and communication in ethpandaops [repo](https://github.com/ethpandaops/ethereum-package) - Create a tracking issue for Kurtosis integration blockers and client-side requirements to support Kurtosis devnet - Explore minimal devnet setups for initial testing ## Apr 28th, 2025 - Call #53 ### 1. Team update - Update on [Trin](https://github.com/ethereum/trin) & [Glados](https://github.com/ethereum/glados) by [Mike](https://github.com/mrferris) - Added AcceptCodes to census and bridge metrics for better tracking - Beacon network is now required when History network is enabled - Simplified header proof generation code to avoid duplication - Changed ExecutionBlockProof from list to vector - PR to enable HistoricalSummaries bridge and prepare for future testnet support - Glados - Fixed a broken spec link - Added protocol version support by client graph - Update on [Fluffy](https://github.com/status-im/nimbus-eth1/tree/master/fluffy) by [Kim](https://github.com/kdeme) - Added protocol versioning support - Implemented ephemeral header encoding/ decoding (disabled for now until database update work is complete) - Implemented offer cache to hold content IDs of recent offers - Added metrics to count offer accept codes (Grafana dashboard will be updated accordingly) - Adjusted tests to support HeaderWithProof using SSZ vectors for Capella & Deneb - Adjust tests for supporting HeaderWithProof updated version with SSZ vector instead of list - Adjusted generation of test vectors and enabled validation accordingly - Updated portal-spec-tests test vectors - Revisited the PR of adding the HistoricalSummariesWithProof endpoint in Nimbus consensus client (expected to be merged soon) - PR to add HistoricalSummariesWithProof for Electra - Update on [Shisui](https://github.com/zen-eth/shisui) by [Qi Zhou](https://github.com/qizhou) - Added ephemeral types support and test vectors - Improved performance with multi-threading validation, reducing rejection rates from ~66% to <40% - Continued pushing for integration with Geth but delayed due to recent Pectra upgrade priority - Update on [Samba](https://github.com/meldsun0/samba) by [Meld](https://github.com/meldsun0) - Continued internal adjustements to integrate Samba with Besu - Working on completing all discv5 endpoints in the implementation - Working towards deploying a Besu-Samba integrated instance - Plan to differentiate client version string for tracking Besu-Samba integrated version and Samba standalone version - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [acolytec3](https://github.com/acolytec3) - Merged discv5 fixes into ChainSafe implementation (not yet deployed to boot nodes) - DevOps continued improving boot node stability through VM placement updates - Plan to update History network container for Deneb headers - Start to review ephemeral header gossip logic, and prepare to active the already-implemented basic types ### 2. Discussion topics #### 2.1 History Network Deployment & Monitoring - Goal - Finalize History network deployment to unblock State network development - Primary focus is to get the ephemeral head fully live and deployed - Action items - Create a meta-issue (led by Kolby) to track remaining tasks across client teams - Glados monitoring - Separate infra needed for tracking ephemeral head data - Minimal Beacon network monitoring to verify gossip availability - Next step - Client teams to contribute status updates via the meta-issue #### 2.2 Pectra Upgrade & Testnet Integration - Pectra Fork Readiness - Most Pectra-related updates are client-specific - No major cross-team sync needed now - Hive tests can be updated after the Pectra fork ships - Future Fork Testnet Integration - Integrate Portal into testnets before future forks to avoid post-fork delays - Prioritize **Sepolia** as the primary testnet for Portal-related work, as it contains pre-merge data ## Apr 21st, 2025 - Call #52 ### 1. Team update - Update on [Trin](https://github.com/ethereum/trin) & [Glados](https://github.com/ethereum/glados) by [Milos](https://github.com/morph-dev) - Improved bridge functionality - Supported e2hs files for gossiping history - Updated all bridges to use census / offer, enabling direct offer instead of gossip logic - Added ephemeral header content type definition (not active yet) - Ongoing PRs on Deneb proof generation & validation, and Electra consensus types - Glados: Implemented protocol version tracking for peers - Trin, Fluffy, and Shisui now all officially supports v1 - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [ScottyPoi](https://github.com/ScottyPoi) - v1 is implemented but not publicly updated yet - Continued working with DevOps to revamp boot node setup - Awaiting ChainSafe to release an update that includes the bug fix discussed last week - Update on [Shisui](https://github.com/zen-eth/shisui) by [Qi Zhou](https://github.com/qizhou) - Profiling ongoing to resolve performance issues (especially in comparison to Trin) - Implemented new RPC endpoints to support missing JSON-RPC APIs - Added caching for the highest compatible version to speed up lookups - Update on [Samba](https://github.com/meldsun0/samba) by [Meld](https://github.com/meldsun0) - Focused on integration with Besu - Currently managed to run Samba inside Besu locally, but still need further polishing & adjustments - Aim to support Samba both within Besu and as a standalone solution - Working toward deployment / testing of Samba in Besu across versions to evaluate performance and behavior ### 2. Discussion: Head-MPT State Network Spec - Key problem - How to update slices of state for nodes whose section had no updates in a given block - Proposed Solution: Introduce block-level MPT diffs as a formalized content type - Each block contains a diff from the previous block's state - Allow nodes to update and re-anchor their data to the latest MPT root, even if their slice didn’t change - Enable a lightweight initial rollout - nodes can sync & operate without full account state storage initally - Sync Mechanism - New nodes can sync from the current trie and fast-forward using accumulated diffs - Trie diffs will be the main data gossiped on the network - While the protocol supports querying individual trie nodes, gossiping individual nodes isn't the focus - Implementation Roadmap for client teams - 1st priority: Finalize ephemeral header support in the History Network - 2nd priority: Push the State Network as close to the chain head as possible - Afterwards: Define block-level diff data type, and implement diff generation & propagation - Relevant PR/ issue: https://github.com/ethereum/portal-network-specs/pull/389 ## Apr 14th, 2025 - Call #51 ### 1. Team update - Update on [Trin](https://github.com/ethereum/trin) & [Glados](https://github.com/ethereum/glados) by [Kolby](https://github.com/KolbyML) and [Mike](https://github.com/mrferris) - Implemented Protocol versioning & v1 - Fixed bug in JSON-RPC result for the offer method - Performance improvements of e2hs writer - 44× speedup for converting era1 to e2hs files - 6× speedup for converting era to e2hs files - Now pre-merge e2hs files can be generated in ~10s and post-merge e2hs files in ~5 minutes - Plan to run Bridges fully on e2hs files with public endpoints soon - Discussion on adopting e2hs format as a standard post-merge history format - Nearly ready for broader feedback & adoption once the full repo is generated and publicly available - Glados update - Finished schema change last week, allowing UTP health visibility - Deploy sync auditing this week - Plan to work on protocol versioning visibility next week - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [ScottyPoi](https://github.com/ScottyPoi) - v1 protocol version merged and unit tested - Fixed bugs in discv5 test suite - 3 PRs submitted to ChainSafe’s discv5 implementation to fix ID encoding, challenge re-sending, and byte/int conversion bugs - Continued revamping public node setup with devops team for better reliability - Update on [Samba](https://github.com/meldsun0/samba) by [Meld](https://github.com/meldsun0) - First call with Besu team to align on plugin use and integration - Need to change certain discv5 endpoints from Besu side to integrate with Samba - Working in parallel on - Endpoint completion - Design shift from standalone to plugin-compatible mode - Update on [Shisui](https://github.com/zen-eth/shisui) by [Qi Zhou](https://github.com/qizhou) - Implemented protocol versioning v1 and updated offer interface - Fixed a panic issue related to closing connections on errors - Addressing performance issue - Working on utp rate limiting to optimize gossip - Benchmarking ongoing to resolve performance gaps vs Trin ### 2. Discussion topics #### 2.1 Head-MPT State Network Spec - Design overview - The Head-MPT State Network will initially be a separate sub-network (not merged with the existing state network) for faster iteration, with potential future integration - Scope - Focused on the last 256 blocks, with nodes anchoring content via state roots or block hashes - Content Types - Gossiped data includes account trie nodes, bytecodes, and storage trie nodes - Each node anchors content to the state root - Gossip mechanism - Bridges split the trie diff into 256 subtries and gossip each as a key-value pair - Nodes store and update assigned subtrie based on their node ID - Nodes retain their subtrie for the last 256 blocks, discarding older ones - Querying & sync - Direct leaf queries are supported with proofs - Nodes can fetch missing individual trie nodes when bootstrapping or syncing - Implementation Phasing - Start with account trie only - Contract trie support will follow after validation - Next step - Client teams are encourated to review the [PR](https://github.com/ethereum/portal-network-specs/pull/389) and provide feedback - Deeper discussion scheduled for next week’s call (or the week after if there is light attendance next week due to Easter) - Relevant PR/ issue: https://github.com/ethereum/portal-network-specs/pull/389 ## Apr 7th, 2025 - Call #50 ### 1. Team update - Update by [Piper](https://github.com/pipermerriam) - Working on history expiry coordination & organization - In early onboarding phase for history state spec - Update on [Trin](https://github.com/ethereum/trin) & [Glados](https://github.com/ethereum/glados) by [Kolby](https://github.com/KolbyML) - e2hs update - Fixed an invalid BlockIndex type in e2hs - Pre-merge e2hs files generated & uploaded, except some minor bugs - Once fixed, e2hs file will replace era1, with plan to sync up to the chain head minus ~27h - Fixed a bug where the discv5 ENR cache was to small to handle 500+ or more active peers by scaling the cache size to match the max number of sessions - A cleaner upstream fix coming soon via the discv5 lib - Glados - Updated Glados to support ethportal-api 0.6.0 which incl. ping extension - Fixing the census issue, which now works even without online nodes - Update on [Fluffy](https://github.com/status-im/nimbus-eth1/tree/master/fluffy) by [Kim](https://github.com/kdeme) - Updated ping endpoint to support ping extension - Worked on async EVM improvement and investigating segfault/ bugs in CI tests - Protocol versioning - Progressing on protocol versioning, but blocked by an unexpected compiler bug - Need to update discv5 for protocol versioning to properly handle ENR access during request/ response - Plan to store ENR in session cache for reliable access - Implemented the changes of accept codes and add varient size prefix - Investigating a clean solution to manage versioning info along with the encoding/ decoding process - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [ScottyPoi](https://github.com/ScottyPoi) - Working on protocol versioning, added initial support but v1 features not yet active - Current priority: Finish the v1 changes implementation - Progressed on history expiry support within EthereumJS team - Plan to revamp boot node setup, shifting from 2 overloaded VMs to multiple smaller VMs, for better performance and reliability - Merged a Tauri-based web interface project built on Ultralight by Justin, currently allowing basic RPC lookups - Update on [Samba](https://github.com/meldsun0/samba) by [Meld](https://github.com/meldsun0) - Implemented SSZ union removal - Fixed a bug in the putContent JSON-RPC endpoint, which is related to the utp lib usage - Passed local Hive tests against Trin, still debugging remaining test failures - Plan to begin regular biweekly sync with Besu team for planning and coordination - Update on [Shisui](https://github.com/zen-eth/shisui) by [Grapebaba](https://github.com/GrapeBaBa) - Merged PRs, incl. implementation on accept codes and protocol versioning - Implemented add size prefix in FindContent - Fixed the FindNodes response bug, due to packet too long - Added rate limiting policy, where gossips are dropped when resource usage exceeds the limit - Fixed discv5 whoareyou response issue in Geth, through in Hive tests still see issues when communicating with Fluffy nodes - The issue with Fluffy nodes - When two portal requests are sent simultaneously before a session is established, only the 1st succeeds and the 2nd fails - eg. Fluffy’s background Ping (for node revalidation) races with test-triggered FindContent, would cause one to fail - Potential solution - Queuing on Fluffy side: Queue the concurrent requests until the session is setup - Retry logic: Clients could retry dropped requests automatically - Test adjustment: Modify Hive test to better reflect real-world behavior - Ultralight related notes - Identified 3+ encoding-related bugs when previously debugging Hive test failures related to discv5 - Not sure if these fixes would resolve the concurrent-request problem ### 2. Discussion topics #### 2.1 Next step for Bridges - Curent status - Bridges are mainly run by the Trin team currently - Technical updates - Transition to generate e2hs file (much simplified without the accumulator) to replace era1 file for the entire range - Plan to maintain two bridge types: ephemeral bridge (handle reorgs & complex edge cases) and e2hs bridge - Dropping support for the older era1 bridge - Next step proposal - Community-runned bridges - Consensus: Shift from team-operated to community-operated bridges - Next steps: Document clear setup steps for external parties to get involved easily #### 2.2 Hive test - Achieved improved test coverage this week, with additional [Portal Hive test](https://portal-hive.ethdevops.io/#summary-sort=name) now passing successfully - Strong cross-team collaboration as client teams worked together on troubleshooting and debugging #### 2.3 Team presence at Interop event - Some members from Portal team will attend the event in Berlin in June ## Mar 31st, 2025 - Call #49 ### 1. Team update - Update on [Trin](https://github.com/ethereum/trin) & [Glados](https://github.com/ethereum/glados) by [Kolby](https://github.com/KolbyML) - Formalized the new e2hs file format to handle all EL blocks, which simplifies bridging process - Implemented e2hs, incl. adding reader & writer in e2store crate - Published e2store crates available for external projects to use - Fixed a utp packet handling bug, where 1st part of transfer is ignored by recipient if the STATE response to SYN is dropped - Cleaned up redundant test vectors - Glados: Now display header by numbers on the latest block status graph - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [ScottyPoi](https://github.com/ScottyPoi) - Added protocol versioning to ENRs and preparing for the PR on accept/decline code change - Updating EthereumJS to support EIP-4444 and drop day by integrating Ultralight as a dependency - Progressed on ephemeral headers implementation - Update on [Samba](https://github.com/meldsun0/samba) by [Meld](https://github.com/meldsun0) - Implemented new endpoints but facing delays with UTP library (putContent issue) and accumulator integration - Restructuring code for better modularity to integrate with Besu client - Need further exploration on the primary use case of Samba by the Besu client (eg. JSON-RPC serviing vs full block sync) - Update on [Shisui](https://github.com/zen-eth/shisui) by [Qi Zhou](https://github.com/qizhou) - Added support for protocol versioning and accept code implementation - Fixed a race condition bug and merged ping extension PR - Investigating the Discv5 whoareyou-challenge response issue and performance bottlenecks in Go implementation ### 2. Discussion topics #### 2.1 Protocol Versioning Implementation - Most of the client teams have started implementing the protocol versioning - Teams are encouraged to implement miniimal versioning to enable incremental updates and compatibility testing #### 2.2 Ephemeral Headers Spec Update - The problem - The initial assumption that all EL headers could be sourced from ```LightClientUpdates``` was invalidated as the CL derives tx/ withdrawal data via SSZ instead of RLP/ MPT - The spec is being updated to - Clarify limitation: EL headers can't be fully sourced from the Beacon Network - Define handling - Headers should be organized chronologically in payloads - Clients should retain headers during reorgs until chain validity is re-verified - Avoid unnecessary network requests that could case DDOS issue - Allow headers to propagate naturally across the network - Note edge cases: Notes added for scenarios like reorgs and data propagation - Next step - Team working on the implementation is encouraged to read the PR link for details - Relevant github issue/ PR - https://github.com/ethereum/portal-network-specs/pull/387 #### 2.3 Shisui & Ultralight Connectivity Issue - The problem - Milos obversed intermittent failures when sending ```Find_Nodes``` request to Shisui and Ultralight nodes - Pattern of failure - ```Find_Nodes``` requests fail at non-zero distance - Requests with zero distance or simple ping work fine - The issue isn't consistently reproducible but happens frequently - Investigation - Initial debugging attempts suggest the session or message keys might be incorrect - Equivalent Hive tests are passing, ruling out protocol-level bugs - Shisui identified a potential race condition in node assignment logic and a fix has been proposed & under review - Next step - Team agree to collaborate via a live debug session, with Shisui deploying a test node for targeted troubleshooting - Relevant link - https://discord.com/channels/890617081744220180/890617082243350560/1355946931540721876 #### 2.4 EIP-4444 Coordination Updates - Sepolia testnet: The initial May 1st drop date is now Sepolia testnet drop date, with efforts underway to align an EL client version ready for Sepolia so CL clients can test against - Mainnet drop date: The mainnet drop date is postponed to sync with Pectra upgrade, due to CL's dependency on pre-merge logs #### 2.5 State Network Progress - State network rollout has stalled due to - Gossip performance issue: Bottlenecks in propagating new state data across the network - Dependencies: Protocol versioning, accept codes must be finalized first to unblock state sync features - Action plan - Short-term goal: Focus on shipping a client version that can sync state even if it's slighly behind the head of the chain - Priority: - History network will take precedence over the State network due to higher urgency - Ephemeral state work and performance optimizations may follow later, depending on team availability - Team bandwidth constraints - There will be reduced manpower on Trin side during summer due to vacation/ parental leave - Potentially exploring collaboration with Fluffy team on the State network development #### 2.6 Hive test - Two key failures remain unsolved - JSON-RPC extension: Some clients still need to expose required extensions via JSON-RPC - Post-merge proof validation: Minor bug persist in validating post-merge proof formats - Next step - Teams are urged to prioritize these fixes within the next two weeks to get Hive tests fully green, as prolonged failures risk being overlooked ## Mar 24th, 2025 - Call #48 ### 1. Team update - Update on [Trin](https://github.com/ethereum/trin) by [Ognyan](https://github.com/ogenev) - Cleaned up history storage by removing old content value types - Refactored e2store header types for compatibility - Implemented ping extensions for JSON RPC endpoints - Added ephemeral store as a step toward ephemeral content for the History network - Update on [Fluffy](https://github.com/status-im/nimbus-eth1/tree/master/fluffy) by [Kim](https://github.com/kdeme) - Tuned the PoC syncing Nimbus EL from Portal Netowrk: incl. adjust recursive finContent concurrency (reducing requests from 3 to 2 slightly improved performance) - Observed that ~10% of recursive findContent requests timeout when sending 2-3 concurrent queries, potentially due to udp packet loss or async lib issue - Improved ContentDB pruning mechanism for speed - Integrated EVM and implemented eth_call, yielding a ~20% performance improvement - Started investigating the issue of discv5 protocol handling multiple concurrent requests - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [ScottyPoi](https://github.com/ScottyPoi) - Finalized ping extension for JSON RPC endpoints - Improved pruning mechanism (now more deterministic behavior with reduced overlaps) - Released major update with EthereumJS lib - Progressed on ephemeral header implementation - Preparing on Portal integration with EthereumJS client - Update on [Shisui](https://github.com/zen-eth/shisui) by [Qi Zhou](https://github.com/qizhou) - Geth integration process update - PR re. repeat challenge in "who you are" handshake: Merged and currently being tested on Hive, have some issues with Fluffy nodes (need further investigation) - PR re. ping extension for JSON RPC endpoints: In final review stasge, expected to be merged shortly - Collaborated with Geth to make Shisui run as an independent sub-process for Portal Network access - Initial implementation planned by end of this month - Other work incl. - Participated in protocol versioning discussion - Start working on detailed offer decline code - Fixed race condition issues - Update on [Samba](https://github.com/meldsun0/samba) by [Meld](https://github.com/meldsun0) - Close to passing all Hive tests, except a bug in putContent gossip handling - Slightly behind on SSZ union removal and ephemeral headers implementation - Working in parallel to integrate Samba as a plugin for Besu - Nearest goal is to fix the putContent bug and make initial progress on Besu plugin integration ### 2. Roll out network changes using Protocol Versioning - Deployment approach - Team agree to roll out offer/accept code changes using protocol versioning - Clients will start supporting both old & new versions and signal protocol version through ENR - Glados Monitoring - Need to add a tracking table on Glados to show the protocol versioning states - Issue discussed - Versioning semantic: Whether to rename portal wire protocol version to a more general portal version for future clarity - Lack of size prefix: FindContent/ Content request lacks a size prefix, making it difficult to distinguish between incomplete and invalid data, potentially leading to incorrect peer ban - Version 1 action items - Deploy offer/ accept codes - Add length prefix to uTP streams - Related PR/ issue - https://github.com/ethereum/portal-network-specs/issues/375 - https://github.com/ethereum/portal-network-specs/issues/380 - https://github.com/ethereum/glados/issues/382 ### 3. Banning/ Peer handling - Risk of peer banning - Could backfire in edge cases, eg. banning all peers during local network outage - May accidentally isolate nodes from the network - Peer scoring is preferred over hard bans - Prioritize good peers rather than punishing bad ones ### 4. Handling Ephemeral State Content - Main issue - Current trie-based model struggles with ephemeral state management during chain reorgs, as state content can exist on both canonical and non-canoncial chain - The issue arises when attempting to migrate state content to permanent storage or delete non-canonical chain related content after finality - Potential solutions - **Solution 1**: Modify content keys to include block hashes, enabling proof tracking across forks (but risks redundant storage) - **Solution 2**: Restrict trie-based model to finalized state only, using the flat model for ephemeral data near the chain head - Team consensus - Favored **Solution 2** for simplicity - Open questions - Define the threshold of "finalized" (EL/ CL finality vs practical limits) - Document verification rules for headers near the head of the chain - Action items - Formalize flat model spec, led by Milos - Clarify finality checks in the state network specs - Related PR/ issue - https://github.com/ethereum/portal-network-specs/issues/382 ### 5. Storage of Ephemeral Block Bodies/ Receipts - Main question - How clients choose to store the ephemeral block bodies/ receipts? Whether they should be store permanently? - Potential approach - Pre-finalized data: Keep in-memory cache (auto-purged on finalization) - Finalized data: Persist permanently (proofs are immutable post-finalization) - Client plans - Trin: Testing in-memory storage for pre-finalized bodies/ receipts - Fluffly: Exploring separate tables for headers/ bodies to enable efficient access & pruning, but still need further exploration ## Mar 17th, 2025 - Call ### 1. Team update - Update on [Trin](https://github.com/ethereum/trin) by [Mike](https://github.com/mrferris) - No major development update last week - Reth is able to sync the chain through Trin nodes - Two main areas for syncing performance improvement (from 34 hours to less than 10% of that) - Fix the issue of serving incorrect or corrupted data, potentially due to uTP - Address performance bottlenecks - Glados - Will focus on optimizing the Reth sync process, and maintain a scoreboard to track performance - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [acolytec3](https://github.com/acolytec3) - Ephemeral headers - Implemented the history network related features, incl. serving & storing content headers - Prepare to integrate the beacon network to track the head of chain and handle ephemeral headers - Question regarding the spec - The current spec requires storing the last 8,192 consecutive headers, which might be unnecessary and could potentially cause issues - Need further exploration/ clarification on how to handle headers once a sync period is finalized and historicalSummeries are pushed to the network - Update on Nethermind Portal implementation by [FLCL](https://github.com/flcl42) - Plan to prepare the config and integrate Nethermind Portal Client into Portal Hive test ### 2. Discussion on the spec change - Relevant PR: https://github.com/ethereum/portal-network-specs/pull/370 - Discussion over adding detailed offer decline codes to the spec - Concern: Adding the additional decline info might not be actionable and could overload the protocol - Beneift: Could be useful for optimizing content offering strategies and improving bridge operations - Team consensus - Would be better to clarify the purpose of adding the response codes - Will move forward with the change ## Mar 10-11th, 2025 - Portal Summit Berlin - Portal Summit notes: https://hackmd.io/DWsDCFooT-u7skmgAb-uSA?view ## Mar 3rd, 2025 - Call #47 - No call update this week as most of the team are travelling - There will be no call update next week (Mar 10th) due to in-person Berlin Summit ## Feb 24th, 2025 - Call #46 ### 1. General update - Update on Berlin summit by [Piper](https://github.com/pipermerriam) - Working on multiple talks for the summit - Portal implementer call schedule for the next 2 weeks - March 3rd: Client team attendance uncertain; Confirmation of the call will be posted on the ethereum/pm github issue - March 10th: No call expected due to the Berlin in-person event ### 2. Team update - Update on [Shisui](https://github.com/zen-eth/shisui) by [Grapepapa](https://github.com/GrapeBaBa) on EthR&D Discord - Ephemeral headers: Implemented the spec changes, incl. SSZ union removal, None removal in BlockHeaderWithProof, and add validation for post-Capella proof - trace_offer API: Work in progress - Geth integration: No updates this week - Berlin event: All team members have their visa ready - Update on [Trin](https://github.com/ethereum/trin) and [Glados](https://github.com/ethereum/glados) by [Jason Carver](https://github.com/carver), [Milos](https://github.com/morph-dev) and [Nick](https://github.com/njgheorghita) - Trusted block root: Now embedded instead of requiring command-line input - Database migration - Added progress tracking to display stats during the migration - Decided not to migrate post-merge headers with proofs, but still keep the post-merge block & receipts - Beacon network: Added a pruning task to remove outdated light client bootstrap data - Glados - Now able to track database entries for every transfer failure, even if the audit succeeds - Working on the state audits for block 21 million, expected to have a PR soon - Add e2hs file format proposal - Proposed a new file format to be used in the History Network for storing pre & post-merge history data - Feedback from the Nimbus team is needed to understand why receipts were excl. and whether should be incl. directly or stored separately - Further discussions will take place in Berlin to gather feedback and finalize decisions on this format - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [acolytec3](https://github.com/acolytec3) - SSZ container discrepancy: Found and fixed a mismatch between the implementation and the spec for post-merge pre-Capella headers; Now passed all related portal spec tests - Hive tests - Aim to rerun the hive tests to verfiy the fix works - Noticed that some gossip tests for pre-merge blocks are failing, pending further investigation - Ephemeral headers - PR is ~25% complete, with fundamental work completed, incl. storing and advertising headers via the ping extension - Work remains on handling gossip and responses for content retrieval - Update on [Samba](https://github.com/meldsun0/samba) by [Meld](https://github.com/meldsun0) - Hive tests: Passed 50+ tests related to the offer input, while some still failing - Recursive lookup: Work on implementing and finalizing the necessary endpoints - Additional endpoints: Implemented endpoints needed for testing the offer endpoint, including local content and history store - Besu integration: Planning to start research and integration in the coming weeks ### 3. Review ephemeral headers deployment plan - Deployment target for this week - Aim to remove SSZ union and enforce proofs for pre-merge headers - Client readiness - On the call, Trin and Ultralight are ready to deploy the change - Pending coordination with Fluffy and Shisui - Next steps - Confirm readiness from Fluffy and Shisui on discord - Team will move forward if the majority of the network is ready, even if some clients are temporarily offline ## Feb 17th, 2025 - Call #45 ### 1. General update - Update on Berlin summit by [Piper](https://github.com/pipermerriam) - Organization & Deck preparation: Working on a new deck to present Portal from first principle, focusing on a deep dive into technical aspects ### 2. Team update - Update on [Trin](https://github.com/ethereum/trin) by [Milos](https://github.com/morph-dev) - History content migration: Progress on migrating history content to new types, incl. introducing ephemeral headers and updating the encodings for headers with proofs - Performance optimizations: Continued discussions on improving performance for syncing history network data - Glados: No major updates this week - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [ScottyPoi](https://github.com/ScottyPoi) - Union removal: Working on a PR to remove the bucket/ proof type and update data types - Ephemeral headers: Developing support for ephemeral headers with a PR in progress - Execution client rebuild: Discussed rebuilding the EthereumJS client based on Portal, potentially starting from the Ultralight client - Update on [Fluffy](https://github.com/status-im/nimbus-eth1/tree/master/fluffy) by [Kim](https://github.com/kdeme) - Ban list: PR merged, but choose to not activate it by default due to potential contact availability impacts - EVM integration: Basic eth_call support is functional, but no support for recursive call yet; testing shows similar performance to Trin - UTP benchmarks: Improved block processing, achieving 40 blocks/second at the 14M block height mark; noticed a UTP decoding error from Ultralight nodes - Historical Summaries: Updated a previous PR to add historical summaries to the REST API for Nimbus, pending to be merged - Update on [Shisui](https://github.com/zen-eth/shisui) by [Qi Zhou](https://github.com/qizhou) - Geth integration: Waiting on Geth devs to review PRs for integration - History network: Worked on the new specs for the history network changes, incl. the nouns and SSZ union removal; code is ready, pending deployment plans - Beacon network: Progressed on beacon network integration, pending debug on some Hive tests - Update on [Samba](https://github.com/meldsun0/samba) by [Derek](https://github.com/Dsorken) - Ping Extension: Completed the implementation of Ping extension - Next step: Setup infra to support ephemeral headers and recursive search - Explore collaboration with Besu: Considering align with Besu to maintain synchronization until beacon network integration is complete ### 3. Discussion on ephemeral bodies and receipts handling - General consensus - Ephemeral bodies and receipts should be gossiped when created and be proven using ephemeral headers - If the ephemeral bodies and receipts become invalid later, they can be deleted - No need for a separate content type for ephemeral bodies and receipts - Security concerns - DDoS attacks where the network could be flooded with invalid ephemeral bodies and receipts - The risk is generally low as reorgs are rare and shallow in the beacon chain (typically 1-2 blocks) - Invalid data wouldn't be propagated; it might linger in some clients but wouldn't infect the network - Next step - Update specs to clarify client behavior on handling ephemeral bodies and receipts ### 4. Coordination on ephemeral headers deployment plan - Merged the [PR](https://github.com/ethereum/portal-network-specs/pull/362) to remove the SSZ union from header with proofs - Post-merge bodies & receipts handling - General consensus - It's a client-specific decision on how to handle post-merge bodies and receipts that are not verified using proofs - Since invalid data would only affect the individual client, it shouldn't impact the overall network - Different handling approaches - Pruning approach: Use files with block header hashes to identify and prune invalid data, but that could require significant effort - Leave-as-it-is approach: Assume the data is valid and take no action, potentially cleaning up later if issues arise - Missing test vectors - Comprehensive test vectors for post-merge headers with proofs (eg. historical rules and summaries) are missing - Current tests only cover partial proofs, not the entire encoded content - Next step: Generate and verify complete test vectors before full implementation - Implementation status by client - Trin: Can implement new encoding and union removal by next week but won’t support ephemeral content until after Berlin summit - Fluffy: Supports pre-merge headers but hasn’t integrated new post-merge types; Limited availability next week for debugging issues - Ultralight: Ready for union removal and new encoding by next Monday - **Deployment plan** - Proposed order - Phase 1: Remove union types and enforce proofs for pre-merge headers - Phase 2: Add support for post-merge headers with new proof types - Phase 3: Implement ephemeral headers once each team is ready - Timeline - Teams agree to **proceed with Phase 1 by next Monday** - Each client will implement Phase 2 & 3 at their own pace ## Feb 10th, 2025 - Call #44 ### 1. General update - Update on Berlin summit by [Piper](https://github.com/pipermerriam) - Preparing the opening talk for the Berlin event, aiming to provide a top-down overview of the Portal Network and get everyone onto the same page ### 2. Team update - Update on [Trin](https://github.com/ethereum/trin) by [Nick](https://github.com/njgheorghita) & [Ognyan](https://github.com/ogenev) - Performance improvements: Enabled WAL mode in SQLite databases, improving performance by 1.5x to 2x - UTP bug fix: Fixed a bug in the UTP library, which has been deployed - Hive testing: Merged a basic sync test for syncing light clients and added new JSON-RPC endpoints to the specs; portal clients will need to implement the new endpoints to ensure compatibility with the test - Minor refactor and new version of the Portal API - Header proof change: Preparing for the deployment of the new header with proof type, with the plan to convert/ update old data in the production database rather than trashing the database - Discussion on [experiment with history network data download](https://github.com/ethereum/trin/pull/1627) - Goal: To measure how fast data can be retrieved sequentially from the history network compared to geth sync - Results - Average download rate: c.5 blocks per second (c.20-23x slower than geth via devp2p). - UTP bottlenecks: Struggles with >60 connections and caps at 2-3 MB/s. - find_content latency: ~5 seconds (95th percentile). - Comparison with Fluffy - Can handle 50+ simultaneous transfer with larger batches - Performance is limited by chunked downloads and connection ramp-up time - UTP performance bottleneck: Currently struggled with inbound connection scaling, and encryption overhead - State network implication: The state network is much larger than the history network, requiring 10x more performance; current performance improvements (eg. range queries) may not suffice for state network requirements - Next step - Focus on UTP improvements, incl. improve UTP implementation and testing frameworks, and explore native discv5 connection methods - Further discussion on performance improvement strategy at Berlin summit - Update on [Fluffy](https://github.com/status-im/nimbus-eth1/tree/master/fluffy) by [Kim](https://github.com/kdeme) - UTP benchmarks: Improved UTP benchmarks to handle more nodes sending data simultaneously - Union removal & database migration: Optimized SQL statements for faster database migration, pending for further testing - EVM integration: Exploring EVM integration, potentially using EVMC API, allowing other EVMs to be plugged in - Others: Finished the bandwidth implementation, which is read to merge; working on a discv5 issue as part of ongoing improvement - Update on [Shisui](https://github.com/zen-eth/shisui) by [Qi Zhou](https://github.com/qizhou) - Geth integration: Most PRs are submmitted, pending for review - Discv5 issue: Addressed discrepancies between Portal and Geth’s discv5 implementations, particularly around request-response handling - Glados issue: Fixed an issue of Glados, which now allows timestamp as sequence number - UTP benchmark & improvement: Significantly reduced memory footprint (from ~60 GB to ~100 MB) and improved performance (able to handle up to 2000 concurrent UTP streams, up from 500) with peak memory usage of c. 80 GB during high traffic - Update on [Samba](https://github.com/meldsun0/samba) by [Meld](https://github.com/meldsun0) - UTP Implementation: Completed the first version of UTP, enabling read/write file exchange & the getContent endpoint, though some validation is still needed - Hive testing progress: Passing more Hive test, but currently still using an old version of the instance due to delay in ping extension implementation; expected to finish the update by the end of the week - Next step: Push a new instance of Samba, work on the implementation of offer endpoint, and continue discussion with Besu for integration - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [acolytec3](https://github.com/acolytec3) - Hive sync test: Implemented and merged new Hive sync tests for the beacon network - Header proof change: Ready to remove the union from block header proofs, passing all test factors; plan to start work on ephemeral headers this week - Browser client: Plan to reboot the browser client with help from the new contributor [Justin](https://github.com/cjustinobi) ### 3. Coordination on ephemeral headers deployment plan - Timeline: Teams agreed to target 14 days from the call for deploying the ephemeral headers change - Database migration - Plan to delete headers without proof and convert pre-merge headers to the new format - Discussed whether to delete or keep post-merge block bodies and receipts; consensus leaned toward deleting the post-merge data due to lower importance for now - A temporary availability blip during migration is expected but considered acceptable - Beacon network dependency - Historical summaries from the beacon chain are required to verify headers with proofs, and team is ensuring the beacon network is ready before the migration - Clients will stop accepting headers without proofs and only accept those with valid proofs once the Beacon Network is ready ## Feb 3rd, 2025 - Call #43 ### 1. General update - Update on Berlin summit by [Piper](https://github.com/pipermerriam) - A planning document is now available: https://notes.ethereum.org/_qEGASgDTLK4_EOAMSHctg?view - Team are encouraged to add topics for presentations & discussion - Expected attendance - Apart from Portal teams, members from Geth, Reth, Nimbus, Nethermind, and Besu are also expected to participate. Erigon won’t aattend due to a scheduling conflict. ### 2. Team update - Update on [Trin](https://github.com/ethereum/trin) & [Glados](https://github.com/ethereum/glados) by [Jason Carver](https://github.com/carver) and [Nick](https://github.com/njgheorghita) - Ping extension deployment - The deployment went smoothly, though ~30 nodes were lost, likely due to lack of upgrades - Debug improvements - Focused on fixing issues incl. setting gas_price when handling eth_call and resolving bugs in content storage checks - Node radius adjustment - Nodes starting at 100% radius were causing excessive work for state nodes - A fix to reduce the max-radiums param to 5% is in progress - Hive testing - Sync tests for the beacon network are being developed - Feedback and feature suggestions for Hive are welcome to drop in the #hive channel - Glados Metrics: - Improvements in global metrics were noted, with fewer failures observed recently - The exact cause is unclear but may relate to latest changes in header acceptance logic. Further investigation are welcome - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [ScottyPoi](https://github.com/ScottyPoi) and [acolytec3](https://github.com/acolytec3) - Glados metrics - Also observed a significant improvement in boot node census response rates on Glados - Ping extension - Discussed how to respond to unexpected type 2 pings without requiring a prior type 0 ping - Concluded that nodes will respond with the matching type without penalizing peers for unexpected type 2 pings - SSZ union removal - Worked on a PR to remove the union in BlockHeaderWithProof - Ephemeral headers - Plan to work on it later in the week, targeting a working implementation before the Berlin event - New contributor - A new contributor may join Ultralight soon to explore brower-based implementation - Update on [Fluffy](https://github.com/status-im/nimbus-eth1/tree/master/fluffy) by [Kim](https://github.com/kdeme) - Ban lists - A more generic, lower-level approach has been implemented, which can be applied on both discv5 and portal wire protocol - The part for discv5 has been merged, the part for portal wire is expected to be merged soon - Database pruning - A simple implementation is being tested, but it's inefficient due to the database layout - Challenges are the current approach is time-consuming, and pruning could potentially affect Glados metrics as nodes with pruned databases might behave differently - Input is needed from other clients (eg. Trin, Shisui) regarding whether to implement pruning or to delete & re-inject data to determine a consistant approach - Discv5 native stream: Start to explore and benchmark it - Update on [Samba](https://github.com/meldsun0/samba) by [Meld](https://github.com/meldsun0) - UTP Integration - Partically completed the getContent & store endpoints, and successfully started receiving content via UTP - Plan to enable writing content through UTP and finalize changes for the ping extension - Aim to start passing more Hive tests by the end of the week - Update on [Shisui](https://github.com/zen-eth/shisui) by [Qi Zhou](https://github.com/qizhou) - Integration with Geth: Several PRs have been submitted, incl. - Make APIs public for reuse in Portal Network - Introduce timeout parameters in discv5 - Introduce node caching - Remove redundant ping challenges - Time consideration: Some changes may require updates to the discv5 spec, which could take additional time - Prepare for Berlin event - Team members are applying for visas and aim to resolve outstanding issues during in-person meetings in Berlin ## Jan 27th, 2025 - Call #42 ### 1. General update - Update on Berlin summit by [Piper](https://github.com/pipermerriam) - Venue confirmed - More details, incl. event agenda, will be shared soon ### 2. Team update - Update on [Trin](https://github.com/ethereum/trin) by [Ognyan](https://github.com/ogenev) and [Milos](https://github.com/morph-dev) - Added UTP duration metrics to measure performance - Fixed issues with the gas_price when handling eth_call - Experimented with block fetching from the network, generated metrics which could be further optimized - Made progress on developing testing for light client nodes on the beacon network - Initialized the big storage nodes, with basic tests (eg. eth_call, getbalance) showed some flakiness, indicating intermittent issues with content retrieval - Need mechanisms to propagate retrieved data back into the network for broader availability - Update on [Samba](https://github.com/meldsun0/samba) by [Meld](https://github.com/meldsun0) - Worked on addressing issues with package exchanging using UTP - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [acolytec3](https://github.com/acolytec3) - Completed a fully working prototype for verifying post-Capella headers on Ultralight - Developed a bridge script to pull historical data, construct proofs, and push them to the nodes - Local multi-node gossip testing is planned to ensure proper header propagation - Update on [Fluffy](https://github.com/status-im/nimbus-eth1/tree/master/fluffy) by [Kim](https://github.com/kdeme) - Added support for ephemeral headers, incl. encoding/decoding - Created PR with test-vectors regarding the content key - Proposed removal of SSZ unions in BlockHeaderWithProof type by making the proof an SSZ encoded ByteList - Fixed bugs related to UTP socket handling and improved data validation process - Experimented running Glados within the Nimbus infra, but encountered a dependency on PandaOps - Identified an undefined failures field in the trace object spec, causing issues when running Fluffy - Update on [Shisui](https://github.com/zen-eth/shisui) by [Qi Zhou](https://github.com/qizhou) - Most of the team members are on national holidays starting from last weekend, which may cause delays in responses - Merged the PR on ping extension and ready for deployment - 1st PR (adding stun protocol) submmit to Geth merged successfully - Prepared for the next PR to update the discv5 APIs, which will allow Shisui to run on top of Geth without the need to merge or modify Geth code directly ### 3. Coordination on Ping extension - Relevant link: https://github.com/ethereum/portal-network-specs/issues/360 - Update by [Kolby](https://github.com/KolbyML) - All clients have passed the Hive tests and no further comments from the team - Confired to deploy ping extension post today's weekly call ### 4. Coordiantion on ephemeral headers implementation and SSZ union removal - Relevant link: https://github.com/ethereum/portal-network-specs/pull/362 - Background - Multiple breaking changes, incl. getting ephemeral headers alive, remove SSZ union, and database migration, needs team coordination to avoid repeated operation - High priority task, as portal network approaches history network adoption - Goal - Setup a tracking issue to define timelines, migration plans, and readiness checks for these updates - Coordinate different teams to ensure client readiness and minimize disruption during the transition - Ideally within the next few weeks, before March - [Kim](https://github.com/kdeme) will take the lead on coordination ## Jan 20th, 2025 ### 1. General update - Berlin Event update by [Piper](https://github.com/pipermerriam) - Working on finalizing the event venue - Prepared event materials, incl. agenda, documentation etc. ### 2. Team update - Update on [Trin](https://github.com/ethereum/trin) & [Glados](https://github.com/ethereum/glados) by [Ognyan](https://github.com/ogenev), [Kolby](https://github.com/KolbyML) - Trin - General - Refactored the code to improve the repo structure - Improved test coverage for light clients and improved the content lookup service to check the local database at OverlayService level - Identified and working on a bug where queries persist during UTP transfer - Ping Extensions - Streamlined the ping extension by removing a serialization layer through combining the custom ping payload with the actual ping message (PR pending to be merged) - State Network - Trin node performance currently is the primary bottleneck - Working on achieving 10x performance improvement - Glados - Updated node visualization with weekly census tracking - Update on [Fluffy](https://github.com/status-im/nimbus-eth1/tree/master/fluffy) by [Kim](https://github.com/kdeme) - Ping Extensions - Proposed the streamline suggestion in the portal discord and implemented the ping extension - Pending team coordination for deployment - Peer Banning List: Working on implementing the temporary peer ban list (conditions for banning: validation failures and invalid offers) - UTP Reset: Identified a bug where connections timed out without sending a FIN or RESET packet - Node Deployment: Planning to deploy c.100 small nodes (2-4 GB) to test default user config - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [ScottyPoi](https://github.com/ScottyPoi) - Ping Extensions - Implemented and ready for deployment - Pending team coordination for deployment - UTP Reset: Implemented the reset packet to match team expectations and align with other clients' behavior - Peer Banning: Implemented peer banning at the UDP level using a mechanism discovered in discv5 - Update on [Samba](https://github.com/meldsun0/samba) by [Meld](https://github.com/meldsun0) - UTP Library: Completed a new version. Aim to integrate & pass through the hive tests, and refine the UTP library based on feedback - Coordination on Ping Extension - Deployment date: Aim to deploy next Monday, pending [Shisui](https://github.com/zen-eth/shisui)'s readiness - Hive test: Coordinate to set up experimental Hive tests with merged client branches to validate changes before deployment ### 3. Other discussion - [Chloe](https://github.com/Chloezhu010) raised the point of making Portal's weekly meeting agenda and notes more publicly accessible through ethereum/pm github and Magician forum ### 4. Next week's call agenda - Jan 27, 16:30 UTC: https://github.com/ethereum/pm/issues/1257 ## Jan 13th, 2025 ### 1. General update - Upcoming Event update by [Piper](https://github.com/pipermerriam) - Near confirmed dates for March 10th & 11th in Berlin. It will focus on collaboration with execution client teams for Portal integrations - Details will be confirmed and finalized soon ### 2. Team update - Update on [Trin](https://github.com/ethereum/trin) & [Glados](https://github.com/ethereum/glados) by [Kolby](https://github.com/KolbyML), [Nick](https://github.com/njgheorghita) - Trin - State Diff Bridges: Deployed to gather data, but high traffic and bandwidth bottlenecks seen on Trin nodes may hinder research on implementing range queries - State Nodes Deployment: Full state data from block 21 million will be deployed this Tuesday - Trin desktop: Added account-related features, allowing users to save accounts across sessions and look up data by account address - Ping Extensions: Planned for deployment alongside Fluffy, and will coordinate the update across all teams, incl. Ultralight, Samba, and Shisui - Glados - Fixed visualization crashes and added a weekly census count to track node fluctuations - Update on [Fluffy](https://github.com/status-im/nimbus-eth1/tree/master/fluffy) by [Kim](https://github.com/kdeme) - Portal sync: Worked on the portal sync proof of concept for execution client - JSON-RPC Methods - Added content validation + local storing to Portal history JSON-RPC methods - Raised the issue of validation requirements for getContent and putContent, where the spces are ambiguous and full validation significantly impacts bridge performance - Proposed adding a validation flag to toggle between different validation levels - Ping Extensions: Initial exploration completed, with plans to develop a basic version supporting standard extension types - Noticed unusual traffic patterns in metrics - Significant increase in UTP connections but decrease in connection duration - High session cache miss ratio (spiked to 40-50%), indicating many new nodes are connecting but not staying long - Update on [Samba](https://github.com/meldsun0/samba) by [Meld](https://github.com/meldsun0) - UTP Library: Continued working to improve test coverage and synchronization. Aimed to expose getContent and offer s for Hive testing next week - Update on [Shisui](https://github.com/zen-eth/shisui) by [Qi Zhou](https://github.com/qizhou) - UTP Performance: Testing a new UTP library in Glados, showing significant improvements (e.g., memory usage reduced from 10GB to <1GB). - Public APIs: Created reusable public APIs of the discv5 in node discovery, with plans to merge the PR into upstream - Update on [Ultralight](https://github.com/ethereumjs/ultralight/tree/master) by [ScottyPoi](https://github.com/ScottyPoi) & [acolytec3](https://github.com/acolytec3) - LIbrary update: Work with ChainSafe (maintainer of discv5 lib in TS) to update the cryptography library used in EthereumJS - UTP transfer issue - Identified issue of receiving out-of-order packets from Trin nodes, causing duplicate responses and clogging the UTP stream - Current solution involved slowing down transfers and implementing packet queuing/ resorting - Ping extension: Will prioritize the implementation of ping extension - Hive test failure issue - All hive tests failed currently due to a low-level bug between Hive and the discv5 lib - Bug location has been identified and the fix is in progress ## Jan 6th, 2025 ### 1. General update - Upcoming team event: The event may either be a Portal Summit or an Execution client interop event, depending on attendance. The event is currently scheduled for March 10th & 11th in Berlin. ### 2. Team update - Update on [Trin](https://github.com/ethereum/trin) & [Glados](https://github.com/ethereum/glados) by [Michael Ferris](https://github.com/mrferris), [Kolby](https://github.com/KolbyML), and [Milos](https://github.com/morph-dev) - Trin - Update portal_*Gossip to portal_*PutContent based spec - Work on implementing ping extensions - State network - Hardware availability & node deployment: Hardware delays are resolved, and 20 nodes are being deployed to gossip state data from block 20 million to the latest. Initialization of the nodes with a snapshot is expected to take a day, with gossiping completion estimated in 15-20 days. - Bridges & Testing: A few bridge nodes are being setup to help gossip heads and address missing state data during primary testing, ensuring smoother gossiping and synchronization. - Timeline: Near-head state is expected to be live by the end of Jan - Glados - No major updates on Glados over the holidays - A few PRs were merged, including a graph visualization for node census over time, helping track network dips and activity - Update on [Samba](https://github.com/meldsun0/samba) by [Meld](https://github.com/meldsun0) - Work is ongoing on the UTP library to improve coverage and test - Work on content serialization and integration into the testing framework - Update on [Shisui](https://github.com/zen-eth/shisui) by [Qi Zhou](https://github.com/qizhou) - Merge code updates to align with the latest PutContent spec - Pass all tests for the UPP code migrated from Trin - Fix issues with concurrency and large data transfers in the Go implementation - Test nodes on Glados to ensure stability ### 3. Other discussion - Hive test failure issue raised by [Jason Carver](https://github.com/carver) - Hive has been pinging with failures since Christmas Eve, likely due to clients updating to the new PutContent spec - Teams are encouraged to investigate and resolve these failures to ensure test stability