Initial steps (week 6 update)

<h1> Initial steps (week 6 update) </h1> Past week I have been looking into the ELK stack and was working to get hands-on experience with it. This started by going over and reading the ELK docs and following tutorials and courses to quickly get me up to speed with the stack. I initially set up the ELK stack on my local machine to quickly get familiar with building the data pipelines as well as to understand how data is formatted and serialised into the ELK. I also installed several of the lightweight modules including metricbeat and packetbeat on my local server, which enabled me to track OS metrics locally (such as RAM, CPU and incoming traffic) to visualise the data through Kibana. Subsequently, I created a quick <a href='https://github.com/BemTG/Monitoring-and-visualization-of-Eth-client-data'>repo on my GitHub </a> as a reference to the project and set up the ELK infrastructure on an AWS t2 medium machine (4GB RAM and 20GB memory). As an initial exercise, I worked with public APIs from QuickNode, Bitquery and Etherscan to process, index and visualise the data streams using the Elasticsearch and Kibana. To keep the architecture fairly extensible in these phases the data processing and querying was done using python (<a href='https://github.com/BemTG/Monitoring-and-visualization-of-Eth-client-data/commit/619732c897081721ef2302d121cb11ab36625170'>which can be viewed here</a>) and Logstash was not utilised in this section. <image src='https://daoagents.s3.amazonaws.com/Quick_ct/arc_11.png' width='60%' height='50%' style= "display:block; margin: 0 auto;" > </image> <h2> Types of queries </h2> The <b>Local</b> and <b>Global</b> queries will be broken down further into two sections, which are <b>Active</b> and <b>Passive</b> queries. <b>Active queries </b> are scheduled queries between the monitoring tool and the beacon node, which are initiated by the monitoring tool. However, <b>passive queries</b> are eventstream data from the beacon node which trigger a query on the monitoring tool. At the current moment, the server makes <b>active queries</b> routinely to fetch data from the public APIs and indexes it to display it through Kibana. The test dashboard is momentarily running <a href='https://eth-monitor.xyz/app/r/s/h9LqY'>down below</a>, but it may be down as I iterate upon it in the upcoming weeks/months. Some of the metrics being tracked include: - Attester slashing - Proposer slashing - Block count - Eth deposits - Gas price, tx_count, smart contracts calls and more  <iframe src="https://eth-monitor.xyz/app/r/s/h9LqY" height="600" width="800"></iframe> <h2> Upcoming steps </h2> During the week, I also reached out to Fredrick with my update and he kindly was able to connect me with the EF DevOps team, which are also working on extracting client data for analysis purposes. The initial plan is to get some of the client data and stream and analyse it through the ELK stack. However, since the projects are in early iteration there is still a lot to figure out on how to structure this efficiently. The following steps include: - Looking into constructing <b>passive query </b> pipelines using Logstash - Research data archival architectures and platforms (Bigquery) - Setting up a prysm node to start reading event data