2024-11-24

EPF5: Final Development Update

After five months, EPF Cohort 5 and my project, "Network Simulations with Shadow", have finished.

In this final development update I will first provide an abstract about my project. Additionally, using my initial project proposal, I will evaluate the results of my project and check if I met the goals I set therein. To conclude, I will report on the challenges I faced, the planned future of my project, and finally my personal experience as fellow within this cohort.

Abstract

Testing networking at scale is hard. Using actual networks, you need to either globally distribute your nodes or emulate network parameters on multiple local machines, as conventional single-machine testing tools such as Kurtosis can't handle large test networks. In my project, I developed Ethshadow, a tool for testing Ethereum networks using the simulation tool Shadow. Aside from developing Ethshadow, I used it to run several experiments on client networking, and thereby validated its usefulnes as another tool for the Ethereum community.

Evaluation of goals

In my project proposal, I defined four goals for my project. In this section, I will investigate whether I met these goals and gauge the progress on the goals unmet.

Evaluation of Shadow

Figure out if Shadow is a good tool to test Ethereum networking

As prerequisite for the other goals, I had to figure out whether Shadow was actually useful for the use case of Ethereum core development. I consider this goal met, with the results of Shadow being useful with a few caveats.

During my PeerDAS experiments, after some hiccups, I got to explore the problem behind the failing networks within the simulations. I inspected the simulation results, which empowered me to form hypotheses and confirm or reject them using further simulations. After identifying the root cause of the problem, I was able to verify solutions with more simulations. This cycle of verifying my theories with simulations was very gratifying: Shadow was a really valuable tool in this context. As PeerDAS can only really be tested with networks at scale, I believe that this would not have been possible with existing developer tools such as Kurtosis. Therefore I conclude that Shadow is useful.

However, Shadow has also shown weaknesses during my simulations. The lack of the following features have strong impact on the realism of the simulations and therfore their usefulness:

• Lack of support for modeling CPU time consumed
• Missing implementation for certain systemcalls and socket options
  Furthermore the rather long simulation times for larger simulations remain to be a restriction.

Implementation of Ethshadow

Prepare a nice framework for sophisticated simulations
Empower all the client teams and researchers to use Shadow if it proves to be useful

Originally two seperate goals, I've decided to combine them here. The first goal aimed for a tool for my simulations during the project, while the second goal aimed at an extended tool for use by everyone.

The first goal was certainly met, as I throughoutly used it during my project simulations. The second goal, however, is clearly unmet, as today most clients are unsupported by Ethshadow. I designed Shadow from ground up to be modular and client support can later be added while not requiring modifications to its core logic. While attepting to implement support for most clients during the final phase of the project, I realized that some features missing from Shadow commonly caused problems in the clients. I remain hopeful that these issues will be fixed and that at some point Ethshadow will support all clients. We received positive reception from some members of the core development community, so hopefully this interest can be further increased and channelled into others contributing to Ethshadow.

Supporting R&D

Improve client networking and help with the specs for the upcoming hardfork

This goal is unmet. While I investigated PeerDAS, I found several approaches to mitigate the issues, but my conclusion was that there are bugs to be fixed in Lighthouse and any conclusions with respect to spec changes are too early. As the time for the project is limited and Lighthouse has a competent team to approach the existing bugs, I abandoned the goal.

My experiments around IDONTWANT were more focussed around creating an argument for implementing support for it in other clients and compared its effects on bandwidth with the effects of increasing the blob limit and target. My results did not (by themselves) support a blob increase, and did not suggest any novel techniques to client networking.

Challenges

The biggest challenge was time. Some of it was wasted, some of it was misallocated. Especially some of the simulations I ran were completely useless due to unneccessary oversights by me. I should have focussed more on onboarding clients. While the simulations were interesting (and to a degree necessary to be able to honestly claim that this is a good testing tool), I believe it would have been better for the future of the tool to focus more on implementing the necessary features in Shadow to support most clients.

The largest technical challenge was debugging the issues I had with clients in the beginning of the project. While some issues were found easily through Shadow's and the client's logs, others required serious debugging, which is pretty hard with unfamiliar projects.

Future of Ethshadow

I hope that Ethshadow finds use! I will certainly keep it in mind for my future endeavors.

Unfortunately, only few clients are supported right now. As I won't have much time to work on it in the future, I hope I have generated some interest in it during my Devcon talk. I will help to maintain it in my free time, but need rely on others helping with adding client support. For example, Anton Nashatyrev is the first developer who contributed to Ethshadow by getting Teku to work! I hope many more follow.

My experience as Fellow

EPF was an amazing time. I had a lot of fun, and learned a lot: technical stuff, about the Ethereum community, and soft skills. For example, when I started EPF I struggled a bit with the weekly updates, as I found it really hard to get started when writing. When writing this document, I realized halfway through that I managed to start just fine and probably would have sweated over an empty document for a lot longer if it were Daniel from six months ago writing this. :D

The sense of community between the fellows, mentors, and of course Josh and Mario was really awesome and motivated me a lot. At no point I felt like us EPFers were seen as "the bothersome new guys" or similar. The acceptance and support from everyone I talked to was inspiring!

If you are reading this, you are either someone from the people mentioned above, or maybe someone interested in the next EPF cohort. If the former: THANK YOU! Thank you for a great time, thank you for all your support, and thank you for reading this. If the latter: Do it. You will learn so much during EPF. Even if you are not selected for a stipend immediately, try to participate as much as you can anyways. You will not regret it!