---
# System prepended metadata

title: Observability Ideas

---

# Observability Ideas

| Date | 2022-10-13 |
| ---  | ---------- |
| Authors | dkomisar, ncapadona, anthomas, canderson |

## What in Sirius should we monitor?

* prefect agents
* builds of our new builder images
* any failure of any check on https://github.com/anaconda-distribution/rocket-platform
    * merges to `main`
* [scheduled](https://github.com/anaconda-distribution/rocket-platform/actions?query=event%3Aschedule++) GHA states (success, failure, etc)
* builder health
    * trends of memory usage / cpu utilization
    * agent health / crashes
    * system events like shutdows / reboots / etc

## What are some alert conditions we care about?
* agent is not running but builder system is
* 

## Ideas to explore
* can we expose some of the Prefect metrics directly to DataDog to centralize the information there?
    * they provide an API, so maybe.  would be a good spike.
    * the prefect logs are in the UI and available via the API - but it's paginated and slow to access
    * data can be streamed, but when a process stops and restarts - the problem is "where should streaming resume in order to prevent dupes"
* an instrumented agent that 'tattles' on prefect and is baked into the AMI / builder where prefect runs
* where possible, create links that cover the duration of the flow run / build / event
* slackbot
    * improvements needed?
    * smaller responses
    * links to DD (with time range set correctly)


## Dashboard ideas
* list of all the prefect builders in one place (hostmap of the builders embedded as a widget)
* ability to drill into the health of each one
* subgroup builders by OS, environment, etc - facets (windows, osx, etc)
* number of concurrent builds / builder - might not be doable now
* links back to the PR in github (see all the in-flight PRs on a builder)

### Package Dashboard View
* useful for perseverance
* shows the builders involved in a package build as a host-map?
* stats about the package build itself
* ability to view multiple package builds together in aggregate

## Priorities / Next Steps
- [ ] Dashboard implementation
    - [ ] _Carl to draft Jira Epic & Issue for this undertaking_
- [ ] Begin to instrument prefect
    - [ ] _Carl to draft Jira issue for this feature_
    - [ ] Things to instrument / send to datadog:
        - [ ] Events
            - [ ] Flow events (started, stopped, failure, etc)
            - [ ] Filenames generated by flow
            - [ ] File checksums / sha256 / byte counts
        - [ ] Metrics
            - [ ] prefect version number
            - [ ] aws instance id
            - [ ] machine stats (uptime, cpu count, etc)?
                - [ ] useful only if the system agent isn't there (linux-s390x builders, ppc)

