--- tags: eth2devs description: Notes from the regular Eth2 implementers call image: https://benjaminion.xyz/f/favicon-96x96.png --- # Eth2 Implementers’ Call #46 - 2020-08-20 [Quick contemporaneous notes by Ben Edgington] Agenda: https://github.com/ethereum/eth2.0-pm/issues/173 Livestream: https://youtu.be/g3fKTfBXArU ## Medalla Incidents since last Friday. See [Prysm write-up](https://medium.com/prysmatic-labs/eth2-medalla-testnet-incident-f7fbc3cc934a) for origins. This triggered many more issues. Good outcome: it would be hard to simulate these stresses! All clients are in much better shape now. EF plans to do some client stress tests on cloud instances. Danny is inclined to keep Medalla moving forward, but it's up for debate. [Most inter-team communication has been taking place on an ongoing basis, so not much more to say here.] ## Testing and Release updates Minor release coming to integrate Solidity contract into the repo. Two additional gossip validations have been added. Network unit testing is ongoing. Adding more tests. Complex blocks-by-range tests would be good to have. Fork choice testing is in progress. Phase 1 tests also making progress. [Mehdi] Fuzzing update. Community fuzzing initiative has gone well. Four people found bugs: three Lodestar; one Nimbus; one Teku. Progress on the structural differential fuzzing. Found a consensus bug in batch verification on Prysm. Continuing to work on the structural differential fuzzing; next step is to get Teku integrated. Will deploy on dedicated infrastructure. Building custom mutators for the differential fuzzing is happening in parallel. See the [blog post update](https://blog.sigmaprime.io/beacon-fuzz-07.html). ## Client updates **Lighthouse** Dealing with Medalla chaos: improvements in syncing during non-finalisation; processing attestations could slow the core executor and cause deadlock, so implemented a queuing system; improved fork handling. Added support for Prysm keystores; Interop improvements with other clients. Attestation inclusion rates. Finished gossipsub 1.1 and peer scoring. Recommend everybody adopts the same scoring parameters for network stability. **Teku** Main issues from Medalla (see [mega-issue](https://github.com/PegaSysEng/teku/issues/2596)). - State regeneration. Regeneration of hot states was a problem, so now have state-regeneration queue that deduplicates requests. - Cache improvements - make sure it's always used! - Start up processing was taking too long pre-processing things. Now periodically persist hot states to reduce work. - Race condition in fork choice. - Sync processing improvements: pick common ancestor. - Fixed deadlock on finalisation. [Mamy] Nimbus also had to fix some state-regeneration issues. Propose a document to share tips. **Prysm** Many improvements! Assumptions regarding peers and finding best branches were reworked, improving caching. Peer scoring improved. Issue on reaching finality that required restart. Community has been extremely supportive. Much more confident in the code than previously. Have discussed the Roughtime issue with Cloudflare. CF may make some changes (e.g. taking median time of servers). Prysm now no longer relies on Roughtime, but will warn if there is a discrepency. [Dankrad] Best practice is not to make large adjustments to clock time based on external input (for some value of large). **Nimbus** Also Medalla issues: 1. Losing peers. Rejected by other clients if gossip is enabled during sync. PR is in progress to not subscribe to gossip too early. Should help here and in #2 2. Sync performance 3. High memory usage. Agressive optimisations for public key caches had contributed to this. Caching of fork choice. Cache could grow unboundedly. Now implementing an optimisation from Protolambda. Maintaining the multinet script to create small testnets with Lighthouse. Looking to add Prysm. Able to easily swap over to use other clients now in case of a critical bug. Nimbus is currently tracking head on Medalla and attesting. **Trinity** Node operation: restore fork choice context from DB. Node configuration improvements. Getting Noise to work. Improving sync performance. **Lodestar** Update is [out on Medium](https://medium.com/chainsafe-systems/lodestar-medalla-update-59865a5cce1a). Lodestar has not been able to sync Medalla during the incident. Gossipsub 1.1 is ready to go - looking to do pre-release very soon. Fixed a bug in discv5 that may have contributed to peering issues. Looking at integrating Blst BLS library: offer a switch between native code and a WASM C implementation. **Nethermind** Got another senior dev onboard. Currently working on Eth1 deposit processing. Expect swifter progress now. ### Other updates [Danny] Hsiao-Wei is working on a checklist for launch. Will host on GitHub for visibility. [Afri] See [questions on the agenda](https://github.com/ethereum/eth2.0-pm/issues/173#issuecomment-677486610). Clients should move to having a release track branch, with other feature branches alongside. Strategies to stabilise code-bases and not make breaking changes. Consider initial launch a "mainnet candidate" - could be mainnet, but if something bad happens we can reset and reuse deposits. What happens if there is a mass-slashing event due to spec or client bugs? Multiple launchpads would be good for decentralisation. [Dankrad] Don't want to encourage complacency by offering too big a safety net to stakers. [Carl] Launchpad update. Tracked down bug that caused double deposits. Web dev will be moving again soon. In the long run supportive of decentralised Launchpad, but maybe better to take the burden off client teams for now. [Raul] Migration of keys between clients is good. Migration of anti-slashing data is also important. A minimal interchange format has been proposed by Lighthouse team. Take a look. May make this an EIP/ERC standard. Also need to document the process of swapping clients well. EF will do some work on this. [Carl] To what extent are wallet stores used currently? [Raul] Prysm is using wallet stores under-the hood. [Carl] Over time the wallet and keystore formats have converged - we may be at the point where we should collapse them into one. ## Research Updates Skipped due to time. ## Networking Discussion around tuning gossipsub1.1 peer scoring params with Protocol Labs. Will test out the recommendations. One issue is clients subscribing to topics while syncing. This can degrade the network as we cannot verify the messages. Better if clients do not do this. Are any clients doing this? [Nishant] Prysm does, but basically ignores them, so looks like it's censoring. Subscribing later adds some complexity when dropping in and out of sync. Prysm doesn't subscribe to subnets until fully synced. [Jacek] How do you decide you are fully synced? It's hard to quantify. [Danny] The best you can do is knowing that you have all branches from all your peers. Try to use the networking channel on the Eth R&D discord for specific conversations. ## Spec discussion [Hsiao-Wei] Update on the deposit contract. The second release of the Solidity version has been made and ported back to the repo. The only change is the metadata, the bytecode is the same. Please use [the latest](https://github.com/ethereum/eth2.0-specs/tree/dev/solidity_deposit_contract) if you are making a testnet. Expect this to be the final version. ## Open discussion Out of time. * * * # Chat highlights From danny to Everyone: 03:02 PM : https://github.com/ethereum/eth2.0-pm/issues/173#issuecomment-677486610 From terence to Everyone: 03:03 PM : https://medium.com/prysmatic-labs/eth2-medalla-testnet-incident-f7fbc3cc934a From Mamy to Everyone: 03:11 PM : actually found it: https://blog.sigmaprime.io/beacon-fuzz-07.html From Meredith Baxter to Everyone: 03:19 PM : https://github.com/PegaSysEng/teku/issues/2596 From JosephC to Everyone: 03:45 PM : watch out for scammers launchpads? From Aditya to Everyone: 03:47 PM : Ones that appear legit, but send your ETH to some malicious contract From Zahary Karadjov to Everyone: 03:51 PM : Carl, Raul, would you mind having that discussion in the #key-management channel? From Carl Beekhuizen to Everyone: 03:54 PM : Sure From Hsiao-Wei Wang to Everyone: 03:57 PM : https://github.com/ethereum/eth2.0-specs/tree/dev/solidity_deposit_contract