--- tags: newineth2 description: The latest update on Ethereum 2.0 development image: https://benjaminion.xyz/f/favicon-96x96.png --- <style> a {text-decoration: underline;} a {color: #0000ee;} a:visited {color: #551a8b;} </style> # What's New in Eth2 - 22 August 2020 ![My avatar](https://benjaminion.xyz/f/ms-icon-144x144.png =32x32) Ben Edgington ([PegaSys](https://pegasys.tech/), [ConsenSys](https://consensys.net/) — but views expressed are all my own) Edition 50 at [eth2.news](https://eth2.news/) ## Medalla Meltdown redux In case you missed it, things have been turbulent on the Medalla beacon chain testnet. I wrote a mostly accurate [initial account](https://hackmd.io/@benjaminion/wnie2_200817) earlier in the week. Read that first if you've no idea what happened. Prysm has since published their own [detailed account](https://medium.com/prysmatic-labs/eth2-medalla-testnet-incident-f7fbc3cc934a) of things from their viewpoint. The long and short of it is that the Medalla testnet suffered some big shocks and a great deal of stress, beginning on Friday the 14th, and continuing for five days while we recovered things. ### The incident <p style="text-align:center"> <a href="https://benjaminion.xyz/images/200820_medalla_participation.png"><img width="800" height="307" alt="Medalla network paticipation graph" src="https://benjaminion.xyz/images/200820_medalla_participation.png"></a><br> <i>My Teku node's view of Medalla participation in the 6 hours after the incident (click to enlarge)<br>Time is UTC on 2020-08-14.</i> </p> What you see here is an initial massive drop off of validators over a thirty minute or so period. This is the Prysm nodes all disappearing from view because their clocks got set to four hours in the future. The horizontal red line is the 66.7% participation that allows the network to finalise: we dropped way, way below that. Things slowly begin to recover as people transfer keys from their Prysm validators to other clients. Then a bit more rapidly from about 2 hours after the incident as the clock error was resolved by Cloudflare, and Prysm validators that had _not_ been updated came back into view. As per [Prysmatic Labs' account](https://medium.com/prysmatic-labs/eth2-medalla-testnet-incident-f7fbc3cc934a), the update designed to fix the original issue contained a critical flaw that unfortunately rendered updated Prysm clients inactive, compounding the problems. Then, just over four hours after the initial issue, chaos breaks out and participation dumps again. Nodes of all types were struggling to process all the "attestations from the future" that the Prysm validators had produced, and which had now become newly valid. The network begins to fragment as nodes try to make sense of all this. Memory gradually bloats up in Teku and Lighthouse causing performance issues and crashes; updated Prysm nodes are still not participating... it's a mess. But it's also one of the greatest tests I can imagine! If we had called a meeting about testing the clients and brainstormed some extreme ways to break the network, I very much doubt that we would have come up with something as exciting as this. The "time transport" aspect led to some fascinating features, not least a large number of validators getting slashed. As a result, all sorts of previously unexplored code-paths were exercised. Huge improvements have been made across the board. Everything is better than it was. We learned a lot of valuable lessons. ### Latest status Over the early part of this week, the client teams got busy hardening their clients to handle this newly hostile environment. Agonisingly slowly, we crept up towards the magic 66.7% participation rate we need for the network to finalise and be considered healthy again. Eventually we got there around 6.30pm UTC on Wednesday the 19th, almost exactly 5 days after the original incident. Throughout that time, there were always clients able to build the beacon chain: it never stopped running. The beacon chain is robust; the beacon chain can recover :tada: Some articles and writings on the whole train of events: - My initial [write up](https://hackmd.io/@benjaminion/wnie2_200817) - Prysmatic Labs' contemporaneous [incident report](https://docs.google.com/document/d/11RmitNRui10LcLCyoXY6B1INCZZKq30gEU6BEg3EWfk/edit#) and [retrospective](https://medium.com/prysmatic-labs/eth2-medalla-testnet-incident-f7fbc3cc934a) - A view from the Ethereum Foundation: Carl on [why client diversity matters](https://blog.ethereum.org/2020/08/21/validated-why-client-diversity-matters/) (the latest in his "Validated, staking on eth2" series) - Evan talks sense in his [tweetstorm](https://twitter.com/evan_van_ness/status/1296542806652133382), and the accompanying [video](https://www.youtube.com/watch?v=dEYzsolkxd0) - See also all the client team updates, [below](#Client-team-stuff). ## Phase 0: The beacon chain All quiet on the spec front: nothing new to report. My brilliant R&D colleagues at ConsenSys published a nice progress report on their efforts to [formally verify the Phase&nbsp;0 specification](https://consensys.net/blog/blockchain-development/formally-verifying-the-ethereum-2-0-phase-0-specifications/). And the latest news on the multi-client [fuzz testing](https://blog.sigmaprime.io/beacon-fuzz-07.html) work by Sigma Prime. ## The Great Explainers Here's something fun: you can learn about Eth2 and earn yourself a [POAP](https://www.poap.xyz/) at the same time! Info is [here](https://www.reddit.com/r/ethstaker/comments/iej8uo/announcing_the_eth2_studymaster_program_earn_a/), Get yourself onto the [Ethstaker Discord](https://discord.gg/B2wkhH4) and check out the `#eth2-studymaster` channel. Full announcement [here](https://discord.com/channels/694822223575384095/694836320786317392/746831562422747254). The idea is, over ten weeks, to read ten Eth2 articles and then answer questions on them. Do well enough and you get the POAP. I'm really hoping that Eth2 devs are not excluded :joy: Somer Esat continued his excellent series of guides with an in-depth article on setting up [Teku on Medalla with Ubuntu](https://medium.com/@SomerEsat/guide-to-staking-on-ethereum-2-0-ubuntu-medalla-teku-170e2c52bd23?source=friends_link&sk=5f88cc288bb247f1711b729ef91de003). This is now my go-to reference :slightly_smiling_face: Somer and Super Phiz made a [video walkthrough](https://www.youtube.com/watch?v=isPUoKfaelE) of the whole thing. Super Phiz has also done a video with Cayman for [setting up the Lodestar client](https://www.youtube.com/watch?v=k5_fywXxdD8). Coinmonks has a guide to setting up [Lighthouse on Ubuntu](https://medium.com/coinmonks/how-to-setup-ethereum-2-0-validator-node-lighthouse-meddala-goerli-4f0b85d5c8f). On the subject of Teku, I made a quick [Teku trouble-shooting guide](https://docs.google.com/document/d/1YWi4HqkGqs9PQA67agHON4PciPKO39mlZwH_3AKg68I/edit?usp=sharing). Keep the feedback coming, people! Simon de la Rouviere produced a nice [tweet-form overview of Eth2](https://twitter.com/simondlr/status/1295048216942084099) for those who like things bite-sized. And the Narkasa exchange published a nice, gentle "What Is Ethereum 2.0?" [article](https://medium.com/@narkasacom/what-is-ethereum-2-0-f27a7d4be640). ## Research It barely qualifies as research, but I did [an analysis](https://github.com/ethereum/eth2.0-specs/issues/2018) on the effectiveness of the mechanism used by the Eth2 beacon chain to agree on the state of the Eth1 chain. Tl;dr: there's room for improvement and I make a suggestion. The ConsenSys TX/RX team has made a simulator that runs an Eth1 client in an Eth2 sharded environment (so-called Phase&nbsp;1.5). Now you can [run it too](https://github.com/txrx-research/eth1-shard-demo). It's built on Teku for the Eth2 part and Catalyst (a fork of Geth by Guillaume Ballet) for the Eth1 part. Danny did a live demo of this during his [EDCON talk](https://www.youtube.com/watch?v=6iW1MeT-Cug) last week. Also on [ethresear.ch](https://ethresear.ch/): - Vitalik [defends](https://ethresear.ch/t/explaining-the-liveness-guarantee/4228/7?u=benjaminion) the beacon chain's preference for liveness over consistency when a choice needs to be made. An interesting discussion ensues. ## Media It seems like a long time ago already :sweat_smile:, but the [talks from EDCON](https://www.youtube.com/c/AsiaPacificEthereumCommunity/videos) are available. Lots of good Eth2 stuff: - [Vitalik](https://www.youtube.com/watch?v=nnY22N_MnOc) from Eth1 to Eth2, opportunities and challenges. - Watch [Danny](https://www.youtube.com/watch?v=6iW1MeT-Cug) wrestle with his cat while demonstrating a simulation of the Eth1/Eth2 merger. - [Hsiao-Wei](https://www.youtube.com/watch?v=-j5suWmE-DY) talks Phase 1 shard data chains. - [Aditya](https://www.youtube.com/watch?v=9avhMNJWnmw) on weak subjectivity. - [Afri](https://www.youtube.com/watch?v=7yQLN49bb30) on the final road to Eth2. - And [Terence](https://www.youtube.com/watch?v=VZyapwu90lA) (Prysm) and [Paul](https://www.youtube.com/watch?v=wp2--tbGJZM) (Lighthouse) with more client-focused presentations. Paradigm did a nice [interview with Danny](https://medium.com/paradigm-fund/interview-with-danny-ryan-eth2-researcher-at-the-ethereum-foundation-9059abc330b9). Find out what he likes to do when not wrestling with cats (it involves vegetables). Vitalik in [CoinTelegraph](https://cointelegraph.com/news/vitalik-buterin-ethereum-20-presents-a-much-harder-challenge-than-we-thought): Ethereum 2.0 Presents a "Much Harder" Challenge Than We Thought. Nobody panic! We love a good challenge :grinning: ## Regular Calls ### Implementers Call #46 took place on the 20th of August. * [Agenda](https://github.com/ethereum/eth2.0-pm/issues/173) * [Video](https://youtu.be/g3fKTfBXArU?t=275) * Notes [from me](https://hackmd.io/@benjaminion/S1b2sJhzv), and [Mamy](https://gist.github.com/mratsim/370f96bd0e50f3bc8bce4a7e4010f887). We did only a very brief review of the Medalla situation since everyone had been in constant contact pretty much throughout, and time was a bit constrained. Thoughts are turning towards beacon chain launch-readiness. Afri walked through some [proposals](https://github.com/ethereum/eth2.0-pm/issues/173#issuecomment-677486610) he has around launch preparation. And Hsiao-Wei is tracking tasks: see the [project board](https://github.com/ethereum/eth2.0-pm/projects/1) for a glimpse behind the scenes. ## Client team stuff Lots of client team updates this week, recounting our adventures on Medalla: update from [Nimbus](https://our.status.im/medalla-update/), and [a follow-up](https://our.status.im/medalla-update-august-21/); and from [Lighthouse](https://lighthouse.sigmaprime.io/update-28.html); and from [Lodestar](https://medium.com/chainsafe-systems/lodestar-medalla-update-59865a5cce1a). Prysm's latest update is the [write-up](https://medium.com/prysmatic-labs/eth2-medalla-testnet-incident-f7fbc3cc934a) and analysis of the Medalla testnet incident. ## And finally... What better than to combine a cosy chat about Eth2 with a little wine tasting? Quantstamp and Cred have teamed up to do a remote wine tasting and panel discussion: "Wine Not Talk About Ethereum 2.0?" :wine_glass: Info and registration [here](https://www.eventbrite.com/e/wine-not-talk-about-ethereum-20-tickets-116121397225). They ship the wine to you. I am on the panel, alongside some of my favourite people in the Eth2 world. It turns out to be starting at 2am my local time, so I'll likely be fairly drunk before we even begin. Should be fun! * * * [![[Twitter]](https://benjaminion.xyz/newineth2/img/twitter.svg =40x40)](https://twitter.com/benjaminion_xyz) Follow me on [Twitter](https://twitter.com/benjaminion_xyz) to hear when the next edition is out 🙌. [![[RSS]](https://benjaminion.xyz/newineth2/img/rss.svg =32x32)](https://benjaminion.xyz/newineth2/rss_feed.xml) We also have an [RSS feed](https://benjaminion.xyz/newineth2/rss_feed.xml).