Try   HackMD

EPF Dev Update #7

This past week, I continued working on gathering transaction data through Indexer (Repo) from a Geth node using an RPC (provided by Mario) method and storing it in a CSV file. I have made progress in several areas, but have also encountered some limitations.

  1. There were a few logical mistakes & wrong assumptions about the APIs I was working with for which I had to dive into the Geth codebase. Since I was working with the RPC endpoint I took the opportunity to explore the internals of ethersJS & web3JS as to how they encapsulate RPC calls into functions. Pretty Intresting.
  2. I have extracted the initial dataset for the deposit contract on Ethereum using the event indexer. I am using the dataset from 14th October, 2020 i.e (from the point the contract was deployed) up untill 25th December, 2022. That is around 500k i.e (503580 to be exact) validators information. but I am assuming there are some discrepancies. I will have to double check the following.
    • Make sure all of the validators are captured. I plan to check by summing all of the validator indexes & its sum should be equal to the sum of first 503580 numbers! I will optimize this later.
    • Make sure no event is missed.
    • Make sure events from uncle blocks are not included. This took me 2 days to figure out. As captured validators were more than number of active validators!
  3. I also encountered some network delays that caused the data retreival to fail due to too many request or bandwith problems from my end which led to the missed event data i.e (events from blocks were partially downloaded). After checking this I realized the indexer missed about 90k validators! I will optimize this later too.
  4. The event data provides the most information needed but it doesnt have the to & from fields which are essential to the analysis. And it is only retrievable from get_getTransactionByHash maybe the APIs could get somewhat better? I dont know. Therefore I plan to extract those using the transaction hash I stored in the csv file. This also means that for each 500k validator entries I will have to make around 300k request with added delays i.e (As there are some duplicate tx hases in the dataset that are probably called as a btch in a single transaction) to the ethereum node (and hope I dont break Mario's node).

Overall, while developing the indexer, I have tried to overcome these limitations as much as possible by trial & error which are robust and efficient, but it's important to keep in mind that working with distributed systems like Ethereum can be challenging and these limitations can't always be avoided.

Next steps

  1. Double-check the dataset to ensure all validators are captured and that no events were missed or duplicated, and that events from uncle blocks are not included.
  2. Extract the to and from fields from the transaction data using the get_getTransactionByHash method.
  3. Optimize the performance of the indexer.
  4. I have some metrics in mind to match with eth-metrics addresses for comparision.
Last changed by 
 
Abdul Sami J.
0
147

Read more

Untitled

Final EPF Development Update (Cohort 3) This is the final report of my project On-chain analysis of staking pools attestations. It was around the study of the analysis pools & how they impact the ethereum mainnet. Project Summary My main goal was to analyze performance of the validators controlled by the staking pools to validators that are not the part of any pool. If there is a notable difference between them then it meant that clustering of many nodes is impacting ethereum mainnet. When I chose this project, I was planning to utilize the existing tools, information & data that is currently available on staking pools to get a headstart. After exploring this for a week on different platforms & talking with various teams building on Ethereum it seemed like this type of data was either not easily available or was private information yet to be released. Briefly analyzing the information gave me an unsettling feeling that there might be some discrepencies & unaccuracies. Due to this, performing analysis on this data won't make much sense. Therefore I divided my main project into 2 parts. Produce a dataset after Identify & Labelling the network with different staking pools with low false-positives & higher accuracy. [Finished]Here my goal was to associate pools with their public keys & atleast identify >70% of the network which I was able to do.

Feb 28, 2023
Untitled

EPF Update 10 This past week I have been working on finalizing & refining the dataset of the pools in the overall ethereum network. I foucused on Lido Finance, Binanace & Rocket Pool & few other smaller pools. Starting with Lido & Rocket pool, both staking pools have a complex system to identify their validators & address through which they deposited on the Eth2 deposit contract. My initial approach to identify these pools was using the etherscan-label feature which gives an starting point. But these cannot be marked as authentic or reliable as anyone can label ay address on eherscan. For both, Lido & Rocket pool, I manually extracted few address by quickly scanning through the displayed addresses & by reverse-sorting the addreses in my dataset in which the 'To' address was not the deposit contract. Then I dived deeper into the contracts for LIdo & rocket pool. Lido has a list of node operators i.e 30 at the time of writing from which which deposit to Eth2 contract. In their smart contract, they have a function that has two parameters, The index of node operator. The index of the deposit. This allows to get the node operator detail from which the addresses can be extracted by incrementing index. I quickly created a script to extract this information retrived the addresses which has a very low false positives.

Feb 28, 2023
Untitled

EPF Dev Update #9 This past week have been a very busy, debuggish & informative week. I have finalized the indexer for retrieving data and shaped it to a more useful form. The indexer is now able to retrieve data in a format that is more easily usable for analysis. I have been in touch with the authors of a repository (https://github.com/alrevuelta/eth-metrics/) where they statistically determined and associated staking pools and their addresses. However, after having the conversation it turns out that the data is becoming outdated and by manually analyzing the hard-coded address in that repo gav me doubts about association of addresses with a specific pool i.e Binance for example. We are currently discussing this and see where this goes. Another important thing (alrevuelta) mentioned that asking for address association to discord pools is not a very good approach as asking them to verify doesn't work apparently. While verifying my dataset authenticity, I found an anomaly in my data & the data that is displayed on the Beaconcha.in website regarding validator indexes. My dataset matches with that of etherscan but it differes when it is extracted After diving into the open-sourced codebase, I discovered a section where it directly queries the Lighthouse client for data retrieval. This anomaly doesn't seem to be an issue for the explorer, but rather the Lighthouse client. This still needs to be verified, I will probably get in touch with team later. I also thought of an idea for associating validators to staking pools, similar to how Uniswap maintains its token list. However, this approach relies on the optimistic assumption that the data associated is correct. We might also need to associate some incentives to the contributors to increase authenticity. This will be very beneficial for the comuunity as a whole. I might persue this after cohort. Currently, I am in the phase of classifying the staking pools and have a few approaches that I will follow, this may deviate a bit from the main project as this ground needs to be set before the analysis for accuracy. I want to note that if we don't have enough data, we may conclude that our analysis is incomplete or inaccurate. However, if I see the complications growing, I will shortlist the major staking pools with a few false positives and continue from there. This will not include the performance of all pools, but rather a few pools to get the performance analysis. This approach will still give us valuable insights while being more practical. Here is how I plan to label the data:

Jan 23, 2023
Untitled

EPF Dev Update #8 This past week, I have been working on indexing data and storing it in MongoDB. The data I am collecting is transaction hashes from the blockchain, specifically for the 'from' and 'to' fields. The process of collecting this data took longer than expected, as there were 286k transaction hashes to gather. However, I was able to successfully gather all of the data needed. In addition to the data indexing, I also worked on developing the front-end for retrieving data. This includes creating a search bar for users to easily search for specific data, as well as general statistics that can be displayed on the front-end. With the indexer and front-end development now complete, I will be working on integrating the two. I anticipate this step will take one day to complete. Once the indexer and front-end are integrated, I will be compiling the data. This process will also take one day to complete. Next steps After the integration of the indexer and front-end is completed, I will conduct thorough testing to ensure everything is working properly. This will include checking that data is being properly indexed and stored in MongoDB, as well as testing the search bar is working properly. Label & associate the addresses retrieved from the data to staking pools.

Jan 17, 2023
Read more from Abdul Sami J.
Published on HackMD