---
tags: CDEvents
---

# Metrics with CDEvents PoC

This proof of concepts aims to demonstrate the use case of ["observability & metrics"](https://cdevents.dev/docs/primer/#use-cases).
Specifically it focus on collecting metrics to evaluate DevOps performance.

## Architecure

The diagram below shows an high level overview of the architecture we are aiming at for the PoC. The "Pipeline Visualization" part is out of scope for the PoC, but we may include it if have something for it.


## Metrics

For the PoC we will use an existing framework of metrics, which will help us design CDEvents towards a real life use case. Specifically, we will use CDEvents to collect [DORA Metrics](https://services.google.com/fh/files/misc/state-of-devops-2019.pdf).
DORA (DevOps Research and Assesment) identified four metrics that are good indicators of the DevOps performance of an organisation:

1. *Deployment Frequency*: How often an organization successfully releases to production
2. *Lead Time for Changes*: The amount of time it takes a commit to get into production
3. *Change Failure Rate*: The percentage of deployments causing a failure in production
4. *Time to Restore Service*: How long it takes an organization to recover from a failure in production

### Deployment Frequency

The *deployment frequency* metrics requires the following subjects:

- `deployment` (events): this is the subject whose events we would need.
- We don't have `deployment` this subject today, the closest approximation is `service`.
- If we consider `service` as a generic abstraction of an application deployed, it could be fitting. I think this was the intention with `service`, as opposed to referring to the k8s `Service` abstraction specifically.
- `environment` (filter): we only consider deployments to production, and we will use this subject to select only relevant events

Event types:

- `dev.cdevents.service.deployed`
- `dev.cdevents.service.upgraded`

#### Mattias comments

I guess the protocol need to support the case when a deployment consists a number of services that is needed to function. We could use a another event to group these together. In Eiffel we call this grouping a composition so I will borrow that word here and call it a composition defined.


\[Andrea\] Composition sounds like a powerful feature to have. I don't think it's something strictly required for the PoC specifically, but definitely something we should explore further for CDEvents.

#### Erik's comments

There are two event types listed as relevant to the DF metric, `service.deployed` and `service.upgraded`. The second one I can sort of understand, but not the first one.

While I can see some use cases where `service.deployed` would be used when a new version of something is deployed, I was under the impression that it would mostly be used for "scaling out", e.g. deploying multiple instances of the same version to handle more load. I don't think that contributes to the DORA Deployment Frequency, right?

So, some `service.deployed` messages should be counted, but perhaps not all of them?

Or, in fact, should only one `service.deployed` or `service.upgraded` per new service version be considered?

### Lead Time for Changes

The *Lead Time for Changes* metric tracks a change from its commit through releases and into production.
Relevant subjects are:

- `change` (events): we need events about merged changes
- `release` (events): we need events about releases created
- We don't have a `release` subjetc today, the closest approximations available are the `build` and the `artifact`. The two subjects `build` and `release` share the fact that they result in some kind of artifact, but they are otherwise distinct. The `release` is a kind of artifact. A few options:
- Introduce the `release` subject (my favourite)
- Make `release` a special type of `build`, we could use a `type` attribute on the `build`
- How do we link a `release` to a `change`? A few options:
- Assume that a `change` is always included in the closest `release` after the `change` was merged. This is not necessarily true, as for instance minor releases for instance will only include the minimum set of changes required to fix reqgressions and security issues.
- The release includes the target `change` ID, typically the git sha. We assume that any ID before that sha is included in the release. This however require the event client to know about the VCS system in use and to have access to it to extract the history (e.g. `git log`)
- We try and model this relation in CDEvents. Release could have `changelog` and/or `changelog-uri` attributes to include the list of changes included, but since when? The answer may depend on the versioning model used by the application. With semantic versioning, a major release would include the delta from the latest minor of the "major - 1" release. A minor release would include the delta from the "major" release.
- `deployment` (events): same as for [Deployment Frequency](#Deployment-Frequency)
- The `deployment` (or `service`) subject must include a reference to the `release` (or `build`) subject. This is missing in the the `service` events today.
- `environment` (filter): same as for [Deployment Frequency](#Deployment-Frequency)

#### Mattias comment
If we use links we could use them to trace back from a deployment to the commits that where part of it.

As we have the open question on how to deal with multirepo applications we might the Composition Defined event (described in previous section) to group items together.

When a service is deployed we will need to identify what the service contains. I guess we would link to the artifact e.g. container image but then we probably also want to include the helm charts. Again we could use a grouping event.


\[Andrea\] It would be great to have all those levels of compositions available, but not everyone will have all that information, so I think we should think the protocol in a way that can work also with less granularity - which is what we need to do for the PoC as well :)

### Change Failure Rate

For this metric, we have no events in the current `stages` that could help with this.

We could add an `incident` subject, with `start` and `finished` events.
For reference, Keptn defines a ["problem"](https://github.com/keptn/spec/blob/master/cloudevents.md#problem) object.

Alternatively we could re-use the `service` subject, and consider new predicates like `degraded` and `restored`. The two options are not necessarily orthogonal, and which is used may depend on what is supported by the platform in use.

#### Mattias comments

Another idea on reporting failures on a deployment would be to denote some kind of confidence about the service. In Eiffel we call this Confidence Level Modified. I don't know if we want to keep that name but to give an example we could have the following:


\[Andrea\] +1 I like the idea of confidence levels. It's similar in a way to Keptn quality gates, and it's definitely an area that I would like to see covered by CDEvents. The link between a deployment and a change in confidence level may not be obvious though. I think for the PoC we could make the semplification that the last deployment before a loss of confidence is the cause, but in future we might want event that allow operators to express their root cause analysis and express the association.

### Time to Restore Service

For this metric, we have no events in the current `stages` that could directly help with this, even though `Rolledback` type of events for the service `subject` could be used for this. Restoring the service does not necessarily require a rollback though. If we had an `incident` subject, it would be very useful to reference an `incident` in `service` type of events.

#### Mattias comments

By using the linking idea above we could link between versions of a deployment. Using the linked graph we could then calculate the time it took for a deployment to become healthy. In a sequence like this:


We could measure the time between the Confidence Level Modified was sent and Deployment version 2

\[Andrea\] Sounds good.

## Open Questions / Discussions for the future

* Do we consider multi-repo applications? How do we deal with it?
* `release` subject has an optional `repo` attribute
* if `repo` is set, the `release` is for that `repo` only
* if `repo` is not set, the `release` is for a software that may span multiple repos

* Modeling changelogs in CDEvents?
* How do we deal with different versioning models?
* Including or linking a changelog or both options?
* Rely on the event client having access to the VCS?