# Observability Ideas

| Date | 2022-10-13 |
| --- | ---------- |
| Authors | dkomisar, ncapadona, anthomas, canderson |

## What in Sirius should we monitor?

* prefect agents
* builds of our new builder images
* any failure of any check on https://github.com/anaconda-distribution/rocket-platform
* merges to `main`
* [scheduled](https://github.com/anaconda-distribution/rocket-platform/actions?query=event%3Aschedule++) GHA states (success, failure, etc)
* builder health
* trends of memory usage / cpu utilization
* agent health / crashes
* system events like shutdows / reboots / etc

## What are some alert conditions we care about?
* agent is not running but builder system is
*

## Ideas to explore
* can we expose some of the Prefect metrics directly to DataDog to centralize the information there?
* they provide an API, so maybe. would be a good spike.
* the prefect logs are in the UI and available via the API - but it's paginated and slow to access
* data can be streamed, but when a process stops and restarts - the problem is "where should streaming resume in order to prevent dupes"
* an instrumented agent that 'tattles' on prefect and is baked into the AMI / builder where prefect runs
* where possible, create links that cover the duration of the flow run / build / event
* slackbot
* improvements needed?
* smaller responses
* links to DD (with time range set correctly)

## Dashboard ideas
* list of all the prefect builders in one place (hostmap of the builders embedded as a widget)
* ability to drill into the health of each one
* subgroup builders by OS, environment, etc - facets (windows, osx, etc)
* number of concurrent builds / builder - might not be doable now
* links back to the PR in github (see all the in-flight PRs on a builder)

### Package Dashboard View
* useful for perseverance
* shows the builders involved in a package build as a host-map?
* stats about the package build itself
* ability to view multiple package builds together in aggregate

## Priorities / Next Steps
- [ ] Dashboard implementation
- [ ] _Carl to draft Jira Epic & Issue for this undertaking_
- [ ] Begin to instrument prefect
- [ ] _Carl to draft Jira issue for this feature_
- [ ] Things to instrument / send to datadog:
- [ ] Events
- [ ] Flow events (started, stopped, failure, etc)
- [ ] Filenames generated by flow
- [ ] File checksums / sha256 / byte counts
- [ ] Metrics
- [ ] prefect version number
- [ ] aws instance id
- [ ] machine stats (uptime, cpu count, etc)?
- [ ] useful only if the system agent isn't there (linux-s390x builders, ppc)