Monitoring network traffic and OS metrics

Going from the last update, I integrated and synced metricbeat, packetbeat and filebeat onto the beacon node I have running. This is to enable real-time monitoring of the actual server, as well as the network traffic (e.g udp traffic), being exchanged between the various nodes in the p2p network. The data that is captured is sent to Elasticsearch where it is indexed, making it easy for us to visualise it using the Kibana platform. At the current moment, the peer count on geth is over 160, and on lighthouse is around 50, but these values tend to fluctuate frequently as new peers are discovered.

Below is a breakdown of some of the data that is being monitored and recorded using the lightweight modules.

Metricbeat:

• CPU usage
• Memory usage
• Disk usage
• Network traffic
• Process statistics
• System load

Packetbeat:

• Network Latency
• Round-trip time (RTT)
• Network Throughput
• DNS Query and Response information (e.g. query type, response code, query time)
• TCP/UDP packet information (e.g. payload size, direction, flags)

Filebeat:

• System logs (e.g. syslog, auth.log)
• Application logs (e.g. Lighthouse logs, Geth logs)
• Error logs (e.g. logs containing error messages from the server itself)
• Performance logs (e.g. logs containing performance metrics)

The dashboards

The real-time data from the beacon node being streamed is displayed using the ELK stack below. The following dashboards present data that got aggregated over the past 15 minutes of monitoring. The chart is refreshed every two minutes for the network dashboard, whereas for the OS dashboard is two seconds. This is to ensure that new data points are included iteratively giving a real-time feedback to how the node is operating and behaving.

The OS metrics

The Network metrics

The next steps

• Continue to add more nodes that are geographically distributed with multiple client implementations
• Create pipelines using data from the p2p network using the beacon APIs to the ELK