Initial steps (week 6 update)

Past week I have been looking into the ELK stack and was working to get hands-on experience with it. This started by going over and reading the ELK docs and following tutorials and courses to quickly get me up to speed with the stack.

I initially set up the ELK stack on my local machine to quickly get familiar with building the data pipelines as well as to understand how data is formatted and serialised into the ELK. I also installed several of the lightweight modules including metricbeat and packetbeat on my local server, which enabled me to track OS metrics locally (such as RAM, CPU and incoming traffic) to visualise the data through Kibana.

Subsequently, I created a quick repo on my GitHub as a reference to the project and set up the ELK infrastructure on an AWS t2 medium machine (4GB RAM and 20GB memory). As an initial exercise, I worked with public APIs from QuickNode, Bitquery and Etherscan to process, index and visualise the data streams using the Elasticsearch and Kibana.
To keep the architecture fairly extensible in these phases the data processing and querying was done using python (which can be viewed here) and Logstash was not utilised in this section.

Types of queries

The Local and Global queries will be broken down further into two sections, which are Active and Passive queries. Active queries are scheduled queries between the monitoring tool and the beacon node, which are initiated by the monitoring tool. However, passive queries are eventstream data from the beacon node which trigger a query on the monitoring tool.

At the current moment, the server makes active queries routinely to fetch data from the public APIs and indexes it to display it through Kibana. The test dashboard is momentarily running down below, but it may be down as I iterate upon it in the upcoming weeks/months. Some of the metrics being tracked include:

• Attester slashing
• Proposer slashing
• Block count
• Eth deposits
• Gas price, tx_count, smart contracts calls and more

Upcoming steps

During the week, I also reached out to Fredrick with my update and he kindly was able to connect me with the EF DevOps team, which are also working on extracting client data for analysis purposes. The initial plan is to get some of the client data and stream and analyse it through the ELK stack. However, since the projects are in early iteration there is still a lot to figure out on how to structure this efficiently.

The following steps include:

• Looking into constructing passive query pipelines using Logstash
• Research data archival architectures and platforms (Bigquery)
• Setting up a prysm node to start reading event data