EPF Final Update: Conclusion

The project

The idea started from me wanting to make a tool to see if my Ethereum node was accessible from the internet, so I would know that I've configured by router correctly. I posted my idea to the Discord channel and Mario suggested I work on the Node Crawler, and extend it to add the functionality I was looking for. It had been almost a year since the last commit, and the website was not working because the database would eventually get too big, and then the server would stop working.

I thought this sounded like a great project to work on. I would make something useful to others, learn how the Ethereum nodes communicate, and I would get the feature I was looking for.

Status

I have a running website with my current version running: node-crawler.angaz.io.

It's been running there since 2023-09-02, and I've been updating it every time I push any commits.

I think I've achieved what I wanted and then some. The original plan was to have a page where a user could enter their node's enode, and then the crawler would try to connect to it, but after having experience with the crawler, I realized we probably don't need this. Through the discovery process, it should find almost every node on the network within 6 hours or so. So the node will eventually be found. I think 6 hours is enough time, but we can still add this page if we find this approach is insufficient.

One big revelation I had while looking at the syslogs of the server where the node-crawler is running is that there were a LOT of messages being logged about dropped connections. So I thought that I should accept these connections, because for nodes which cannot accept connections, them dialing the node-crawler is the only way we would be able to get the information about these nodes. This turned out to be very important because there are quite a few nodes out there we would not have information about otherwise. For client diversity, these nodes still matter.

Being able to see if the crawler can connect to your node, what I did was save a history of the crawling, and showed the last 10 attempts on the node's details page. This way, the user can see when the crawler attempted to connect to their node, what the error message was, and when the next attempt will be.

One thing I noticed while looking through the database and metrics is that most dial attempts get an i/o timeout, this means that the node is not exposed to the internet, and will only find peers by dialing out, it will never have incoming peers. This is not great when you restart your node, because

1. your node will mostly be hitting i/o timeout's itself, and
2. the nodes your node had previously found can't reconnect.

Node Updates - Shows the rate of dialed and accepted connections, with success or error.

Update Errors - Shows the rate of errors occurring from dialed and accepted connections, with the reason. You can see how i/o timeout is by far the most common type of error.

The Crawl History page showing recent dial attempts and accepted connections.

The Node's details page showing the details of the node, a map of where the node is located based on data from Maxmind's GeoLite2 database, and a list of the last 10 accepted and dial attempts.

We can see, in this case that this node is not correctly exposed to the internet because all the dial attempts fail with i/o timeout. The only reason we have any details about this node is because we accepted an incoming connection.

Future of the project

There are so many possibilities going forward, but I think the most important is the data. I would like to find a way to automate making a snapshot of the database and uploading it to IPFS or something, so others can look at the data and using it in their own research. I think the openness of this data is important to it being a trust-worthy source of information.

I would like to improve the stats page with historical information, meaning that we would need to save the stats already being collected for the stats page to a time-series table, so we can make graphs from them. It's not complicated, I just haven't done it yet.

The stats page can also be improved with more stats. At the moment, we have countries, and client type, and OS/Architecture which come from the client's name. I think it would be nice if we could have links on each client type, which will give you a list of the versions of that client, same with country linking to a list of the cities of that country. This could also be done as filters.

I would like to add a filter for when the next fork is announced, so we can see how many nodes are ready for the fork. Cancun should start getting dates announced, so when that happens, we will update the Geth dependency, and add the filter. The node is ready, there's just nothing on the front-end yet.

I started working on a page to give information to the users about how to get their node to connect to the crawler if they are seeing only i/o timeouts, or "too many peers", so they can get their node's info on the site even if the crawler can't connect. This would be great for the stats to have as accurate set of data as possible.

I'm not going to claim that I have a beautiful front-end. So it would be nice if we could get someone to design something which looks nicer.

The current state of the code is in a branch on my own fork of the repo. The code is still a bit of a mess, using random functions from the previous implementation. There's old flags which are not being used anymore, and lots of dead code, so I would need to spend time time cleaning things up before it's ready for a PR.

A JSON API for other tools to use would also be pretty nice. For example, an integration with the Rocketpool CLI could make it very easy for users to find their node on the site, and/or give information to the user in the CLI.

In order of importance, I would say:

1. Improve the node name parser so we can parse the versions.
2. Save the stats, so we can see the history.
3. Fork readiness filter, and this would be saved to the database as well, so we can track the number of nodes over time.
4. More info on the help page.
5. JSON API.
6. Beautiful front-end. Sure, it's not the best to look at right now, but it's got everything you need.

Self-evaluation

I was contributing as a permissionless participant. I still had a full-time job to do, so my work on the project was pretty limited to my free time.

At the start, I wasn't super motivated, but when I got going, and really started to understand the code, and how Ethereum nodes communicate, I got a lot of inspiration from all the ideas I was having, and this really motivated me to work on it almost every day. It was a really fun project to work on and I'm really glad I got past that unmotivated stage. I think I've made something truly awesome!

I hope to get more involved with Ethereum's development. I have to keep up at least with new stuff which can be added to the crawler. Hopefully we will get more contributors to it, but I would like to keep it updated so it's always relevant and useful.

Unfortunately, the weekly meetings often collided with weekly meetings at my web2 job, but there were some times when I would join when these meetings were canceled, or I would watch the recordings.

Feedback on the EPF

The EPF is a really cool idea. Thanks Mario for getting me to apply for it, and for coming up with the idea to work on this project. It was really fun, and I would say, even a counter-burnout for my day job.

Getting to hear from the core devs on the office hours meetings was really great. I mostly watched the recordings, but I was there for one or two of them. It's really inspiring to see the people who work on Ethereum and see they are just normal people.

The only unfortunate thing I could say is that there were sometimes issues with the video chat software. But I agree that we should be using FOSS software where ever possible, and not force people to use proprietary software. For me, it worked 99% of the time, with one or two issues. I would say I've seen more problems with MS Teams :P

I give a big thank you to Josh, Mario, and the Ethereum foundation for making this possible. It really is amazing and I hope it continues into the future.

The dev updates I didn't really like at the start. But after the second one, I really started to like them. Putting your thoughts into words really makes you think about them more, and I even had some ideas while writing the updates. I think this might be something I continue to do with my other projects. Blog posts from others have really helped me to learn, so maybe it's a good thing for me to give back what I have learned too.