---
tags: CDEvents
---

# Connecting Events

Status: discussion
Contributors:
- Andrea Frittoli
- Emil Bäckmark
- Ben Powell
- Evan Elms

## Summary

CDEvents are generated by tools that are used in CD pipelines and, more generically, as part of the software lifecycle, from the concept, through production and consumption.

The software lifecycle includes various _workflows_ that may overlap, intersect and exist at different levels of abstraction.

Some high level examples could be:
- business idea, implementation and execution
- software change, build, test, deployment
- dependency change, build, test, deployment
- deployment, performance monitoring, degradation, resolution

A single event may belong to several _workflows_.
CDEvents should provides means to store _workflow_ relevant data, and could recommend effective ways to identify all events in a _workflow_.

## Motivation

CDEvents today include _subjects_ and _predicates_; _subjects_ can be globally identified via a combination of _id_ and _source_. This allows CDEvents users to extract specific _workflows_ such as the entire lifecycle of a specific _subjects_, through the various _predicates_ executed over time.

For instance, an _environment_ may be _created_ and _deleted_. An artifact may be _packaged_ and _published_. Timestamps and semantics attached to the predicated allow building _subject_ specific _workflows_.

Splitting events associated to the same subject into different _workflows_, and including events associated to multiple _subjects_ in a single _worklow_ are both difficult to achieve in CDEvents v0.1, as both operations may require assumptions and guess-work based on the data available in CDEvents today.

## Goals

- Specify changes required to the context, subjects and bindings
- Describe _workflows_ in the primer
- Identify type of _workflows_ CDEvents aims to support
- Include examples of _workflow_ retrieval through CDEvents data

## Non-goals

- Develop optimal event selection logic for _workflows_
- Support centralised workflow orchestrators (e.g. Keptn). The work done here will most likely inderectly help support this use case, but it's not a direct goal.

## Use Cases

- Retrieve the lineage of a subject from a certain event backwards. The lineage can be based on different kind of semantics:
- trigger semantic: identify the series of events that lead to a certain event, for instance identify the release that lead to a deployment in a staging environment
- content semantic: identify all the componets included in a certain subject, for instance identify all the code changes, dependencies, security scans and artifacts associated to that release
- Since the goal of CDEvents is interoperability, there will be services that can be plugged into a whole CI/CD system that may not share a similar UI or information about the whole CI/CD flow. Instead users would need to go to each CI/CD site to see if tests succeeded, deployments ran, and so on. However, if we had a way of linking all events, from start to finish, a single UI could be used with any CI/CD system.

## Requirements

1. A _workflow_ may span multiple tools
1. A _workflow_ is not limited in time
1. If a producer includes event receiving capabilities, the events received must provide enough information to produce new event that propagate existing workflows
1. An event way belong to multiple _workflows_
1. Workflows may fan out and fan in

## Concepts
We sometimes use the words "tracing" and "traceability" as synonyms, but I think that causes some confusion. "Tracing" is what is provided by e.g. OpenTelemetry, and "traceability" is the possibility to track a certain activity/-ies up- or down-stream to another activity/-ies. We could take some inspiration from [this Wikipedia article about Tracing](https://en.wikipedia.org/wiki/Tracing_(software)) and specifically the table on "Event logging versus tracing".

- Tracing is designed to deal with high granularity

## Related Work

- Eiffel links
- [Eiffel `links` object](https://github.com/eiffel-community/eiffel/blob/master/eiffel-syntax-and-usage/the-links-object.md)
- [Activity Linking in Eiffel](https://github.com/eiffel-community/eiffel/blob/master/eiffel-syntax-and-usage/activity-linking.md)
- [Distributed tracing over CloudEvents][opentelemtry-cloudevents]

## Alternatives

### Trace Context

The [W3C trace-context][trace-context] reccomendation defines a way to propagate over HTTP a global trace ID, along with vendor specific context across severall tools. The trace context format can be transported over HTTP as well as [CloudEvents][opentelemetry-cloudevents].

CDEvents could use the trace context format to indicate the _workflow_ they belong to. If no incoming trace context exists, a new trace context is generated.

The trace context approach works well for the first three requirements, but it does not align well with the requirement #4 of multiple workflows. About requirement #5, trace context support fan-out not fan-in.

What does trace context provide?

* an unique identifier for individual traces and requests, allowing trace data of multiple providers to be linked together.
* an agreed-upon mechanism to forward vendor-specific trace data and avoid broken traces when multiple tracing tools participate in a single transaction.
* an industry standard that intermediaries, platforms, and hardware providers can support.

#### Pros

- Well established existing standard
- Documentation already provided

#### Cons

- No semantic relations (?)
- Less flexibility if applied on Cloudevents level, but it could instead be applied on CDEvents level with our own syntax
- (Ben) if we go with tracing approach we could have a simplified syntax - we would not need all features
- (Emil) we'll need parent/child relationship
- https://www.w3.org/TR/trace-context/#relationship-between-the-headers
- Different specifications available: w3c, opentracing, zipkin

#### Spec Changes

#### SDK Changes

#### Tool Adoption

### Workflow Data

The workflow data approach consists in extending the CDEvents context by adding one or more workflow IDs that reprent the workflow(s) that the event belongs to.

An event producer MUST honour incoming workflow IDs when sending events as a reaction to incoming ones. We would need to better define the semantic of these "reactions" and whether we shall encode them in the context as well.

The workflow ID MUST be independent from the event ID and from the subject ID. A workflow ID can be associated with a subset of the events for a specific subject ID, and can be used in events that span multiple subject IDs.

- Emil: we need semantics associated to the input/output associations
- Ben: what will CDEvents look like for specific use cases like fan in, fan out etc

- Similar to trace ID, but allows multiple workflow and moved into the CDEvents context

- Split pipeline execution (fan out), then fan-in waits for all the parallel activies to complete before proceeding

- Q: How can workflow ids be indexed efficiently in a database?

#### Pros

#### Cons

- Non-existing standard. We need to invent all surrounding support.

#### Spec Changes

#### SDK Changes

#### Tool Adoption

### Event Links

A event _link_ is a reference from one event to a previous (target) event. It is created by adding the target event's context.id (uuid) into a dedicated section in the current event.

#### Link Example
The below example show how an `artifact.published` event references an `artifact.packaged` event by adding a reference to the `context.id` of the `artifact.packaged` event to its `links` object.

##### artifact.packaged

{
   "context": {
      "id" : "A234-1234-1234",
      "type" : "dev.cdevents.artifact.packaged",
      ...
   },
   "subject" : {
      "id": "pkg:golang/mygit.com/myorg/myapp@234fd47e07d1004f0aed9c",
      "content": { ... }
   }
}

##### artifact.published

{
   "context": {
      "id" : "AD34-7234-1238",
      "type" : "dev.cdevents.artifact.published",
      ...
   },
   "subject" : {
      "content": { ... }
   },
   "links": [
      {
         "type": CAUSE,  // What caused the activity behind this event to happen?
         "target": {
             "context": {
                 "id": "A234-2345-4567"  // E.g. a testsuiterun.finished event id
             }
         }
      },
      {
         "type": ARTIFACT,  // What artifact was published?
         "target": {
             "context": {
                 "id": "A234-1234-1234"
             }
         }
      }
   ]
}

##### artifact.published, with some optional fields added

{
   "context": {
      "id" : "AD34-7234-1238",
      "type" : "dev.cdevents.artifact.published",
      ...
   },
   "subject" : {
      "id": "pkg:golang/myarm.io/myproj/myapp@mytag", // Optionally used to
          // declare a new uri onto which the artifact has been uploaded
      "content": { ... }
   },
   "links": [
      {
         "type": CAUSE,  // What caused the activity behind this event to happen?
         "target": {
             "context": {
                 "id": "A234-2345-4567"  // E.g. a testsuiterun.finished event id
             }
         }
      },
      {
         "type": ARTIFACT,  // What artifact was published?
         "target": {
             "context": {
                 "id": "A234-1234-1234",
                 "domain": <some unique domain identifier to make the target unique> (optional, not needed if events are not expected to be federated out of the domain),
             },
             "subject": {
                 // Optionally added fields from the target's subject object,
                 // for convenience to not need to look up all data from an event
                 // store
                 "id": "pkg:golang/mygit.com/myorg/myapp@234fd47e07d1004f0aed9c"
             }
         }
      }
   ]
}

##### Is links\[\].type necessary?
I believe so, in order to distinguish between multiple links in a links array.

#### Event Links in Eiffel
The links object is an array of trace links to other Eiffel events, which always reference backwards in time. Each trace link is an object consisting of a type, a UUID corresponding to the meta.id of the target event, and optionally the id of the domain where the target event was published.

Software traceability is the practice of tracking changes, documents, and other engineering artifacts throughout the development process, and Eiffel events are used to represent these artifacts and their relationships to each other. By analyzing the resulting graphs, any number of questions related to the development process can be answered.

- [Eiffel `links` object](https://github.com/eiffel-community/eiffel/blob/master/eiffel-syntax-and-usage/the-links-object.md)
- [Activity Linking in Eiffel](https://github.com/eiffel-community/eiffel/blob/master/eiffel-syntax-and-usage/activity-linking.md)

#### Pros

- Based on graph theory
- Can be used in combination with a graph database to store and efficiently process events
- Linking to event ids gives the possibility to efficiently index the database for efficient graph querying
- Used with good results in the Eiffel event protocol
- Uniform way to relate all types of events

- Linking events would let us "tell us a story" through the data
- When linking events together, from the first event of a git commit to the final event of the production release, would provide better visiblity to organizations regardless of the tools being used. May they be using BitBucket or Github to AWS or Azure, the ability to link CD Events regardless of the tool will help application teams visualize their CI/CD lifecyle and patterns.

#### Cons

- Not a widely adopted standard (mostly used in Eiffel)
- Querying a database for connected events requires graph traversal

#### Spec Changes

#### SDK Changes

#### Tool Adoption

### Hybrid Approach (Eiffel & Global ID)

A hybrid approach would utilize the benefits of Eiffel while still providing fast lookups for downstream consumers by providing a global ID associated with all necessary linked events. The idea would be the global ID would pull everything associated with that ID, but Eiffel would make sense of how things are represented in that graph which could be organized in the frontend.

Below represents the payload of a single event, `pipelineRun`, associated with some global ID:

{
    "id": "namespace/pipelinerun-1234",
    "source": "",
    "pipeline_name": "my-pipeline",
    "outcome": "success",
    "url": "http://mypipeline.com/my-pipeline",
    "context": {
        // having a global ID, would let us make a single query to return
        // all events associated with this global ID.
        "global_id": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
        // timestamp can be used to do some basic sorting of when events
        // occured, which can alleviate some stress from the frontend when
        // traversing Eiffel links to construct the whole graph
        "timestamp": 1677477232395,
        "links": [ // Eiffel links
        
        ]
    }
}

#### Pros
* Single query look up with no graph traversal
* Rather than having servers make sense of what a graph looks like, we can offload that to the frontend
#### Cons
* Relies on two solutions rather than one
## References

- [Link produced events to consumed events](https://github.com/cdevents/spec/issues/10)
- [Add global ID for events](https://github.com/cdevents/spec/issues/100)
- [Honeycomb Buildevents example](https://github.com/honeycombio/buildevents#buildevents)

[opentelemetry-cloudevents]: https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/cloudevents/
[trace-context]: https://www.w3.org/TR/trace-context/