# Subscriptions and Granularity

## Issue Summary

This document is a detailed view into the issue around aligning granularity across topics, subscriptions, and notifications.
* subscriptions are requested at *topic* granularity,
* *filters* and *shapes* are defined/instantiated at the *resource type* granularity
* notifications are issued per *subscription* - which by definition is *topic* granularity

This can make it hard for clients to understand what has occurred when a Topic could represent many different events.

## Definitions

* **filter**: a way of narrowing a broad *trigger* for a client.
* For example, a topic can represent all new perscriptions with a filter allowing for `Patient.id`.
* Allowed filters are *defined* in the `SubscriptionTopic` resource.
* Allowed filters are *instantiated* (given values) in the `Subscription` resource.
* **notification**: data related to the occurence of a *trigger*, sent to a client.
* This could be a 'ping' that a client should follow-up with, a list of ids to retrieve, full resources, or queries that a client should perform.
* Sent in a `Bundle` with a `SubscriptionStatus` resource.
* *notifications* are related to a single `Subscription`, and thus a single `SubscriptionTopic`.
* **shape**: resources relating to a *trigger* that are relevant to a client.
* The *shape* is used to define which resources can/should be communicated in a *notification*.
* A *shape* includes a **focus** (main) resource and can include **context** (additional) resources.
* For example, a 'patient discharge' trigger could be focused on `Encounter`, but *shaped* to include the relevant `Patient`, `Location`, `Practitioner`, etc..
* Potential/recommended shapes are *defined* in the `SubscriptionTopic` resource.
* Shape inclusions are *implemented* by servers at their discretion.
* **subscriber**: person/client requesting and receiving *notifications*.
* Defined in the `Subscription` resource.
* **subscription**: request for *notifications* based on one more *triggers*.
* Defined in the `Subscription` resource.
* The `Subscription` resource maps to a single `SubscriptionTopic`.
* **trigger**: something that happens.
* Can be described via a change in FHIR data, a coded event, etc..
* The `SubscriptionTopic` resource contains one or more *triggers*.

## Context

Topic-based subscriptions are concerned with *triggers* and *notifications*, using a *subscription* as the linking mechanism.

As the work was done defining these mechanisms, a lot of flexibility was left in by design - especially regarding pieces we did not have a lot of implementation experience with. In this document, we are specifically concerned with the granularity of *triggers* and *subscriptions*.

## What is a Topic, Really?

Noticable by absence in [Definitions](#Definitions) is a **topic**. That is because the definition in this context is what this document is exploring.

As currently specified, a **topic** contains at least one *trigger*, the set of allowed *filters*, and zero or more recommended *shapes* for related *notifications*.

This simplest case then is when a *topic* expresses exactly one *trigger*, one or more *filters* related to that *trigger*, and a desired *shape* for notifications for that *trigger*. This is easily expressable and traceable through all stages (definition in topic, instantiation in subscription, and notifications).

The trick (as usual) comes with complexity. For many expressed use cases, a single *trigger* is insufficient. There are three groups possible: a *single trigger*, *related triggers*, and *arbitrary triggers*. I will note that the lines between the groups are fuzzy (at best), but the distinction is useful nonetheless. For now, we will consider the triggers in isolation, outside of their definitional artifacts.

A *single trigger* is represented as a single set of *trigger* values in a topic, today as either a `resourceTrigger` or an `eventTrigger`. In the case that a trigger is defined by a single FHIR resource change, this is wonderfully simple - e.g., when an `Encounter` resource is created or updated to have a status of `in-progress`. With event definitions, this can a bit more nebulous - while something like a v2 `ADT^A04` event is very 'singular', nothing prohibits the use of a more generic v2 `ADT` as a trigger. Despite this, the parts we are concerned with here mean we can treat all *single trigger* definitions the same - regardless of the actual changes being monitored, these represent a single *trigger*. The defined *filters* and *shape* will be applied to any notifications generated for it, so all modeling choices remain the same.

*Related triggers* are grouped together logically by definition. As mentioned earlier, this grouping is done subjectively by the topic author or implementer. As examples, logical groupings could be: 'Encounter notifications', 'Care Team notifications', or 'Medication notifications'. While not required, *related triggers* will often have filters that can be applied sensibly across all *triggers* in the group. For example, all 'Encounter notifications' can be filtered by `patient`, `practitioner`, `location`, etc..

*Arbitrary triggers* are somewhat orthogonal to groupings in *related triggers*, and are generally more concerned with mapping topics to *subscribers*. For example, a patient may want all Encounter and Medication notifications; a public health agency may want all notifications related to certain Observations, Conditions, or Medications; etc..

## Model and Relationship Summary

For this discussion, the current models can be summarized as:

erDiagram

    Topic {
        canonical URL "1..1"
        backbone resourceTrigger "0..*"
        backbone eventTrigger "0..*"
        backbone canFilterBy "0..*"
        backbone notificationShape "0..*"
    }
    
    Subscription {
        id id PK "1..1"
        canonical topic FK "1..1"
        backbone filter "0..*"
    }
    
    Notification {
        reference subscription FK "1..1"
        canonical topic FK "1..1"
        backbone event "0..*"
    }

    Topic ||--o{ Subscription: "referenced by"
    Subscription ||--o{ Notification : "referenced by"
    Topic ||--o{ Notification : "referenced by"

* Topics define:
* The canonical URL used to reference them
* How notifications are triggered - by resource changes, events, or both
* What filters are allowed *for the topic*, *by resource type*
* What resource can/should/may be included (*shape*), *by resource type*
* Subscriptions define:
* The id used to reference them (logical id)
* The topic used to trigger notifications (*exactly one*)
* The filters applied to the topic, *by resource type*
* Notification details (protocol, endpoint, format, etc.)
* Notifications contiain:
* A single *canonical reference* to a topic
* A single *reference* to a subscription
* Zero or more events, all linked to a single subscription and (by definition) a single topic

## Issue Details

For this section, we want to consider the following group of potential triggers/events/topics at different 'levels' of granularity:

| Potential Topic | Trigger Type | Short | Definition |
| --------------------------------- | ------------ | ---------------------------- | ----------------------------------------------------------------------------------------- |
| HL7v2 `ADT^A01` | single | Admit/Visit Notification | Used to notify an addressee that a patient has been admitted. |
| HL7v2 `ADT^A02` | single | Transfer a Patient | Used when a patient is transferred from one location to another within the same facility. |
| HL7v2 `ADT^A03` | single | Discharge/End Visit | Communicates that a patient has been discharged from a hospital or their visit has ended. |
| HL7v2 `ADT` | related | Admit/Discharge/Transfer | Any Admit/Discharge/Transfer notification. |
| US Core `Encounter Notifications` | related | Change to Encounter resource | Changes to the `Encounter` resource that affects one or more elements used in US Core. |
| US Core `Notifications` | arbitrary | Change to any resource | Changes to a resource that affects one or more elements in any resource used in US Core. |

Specific considerations:

1. Implementers have enough granularity to support triggers in ways that make sense to them ([details](#c-1)).
1. Subscribers can ask for notifications at any granularity listed above ([details](#c-2)).
1. Filters need to be applied consistently ([details](#c-3)), e.g.:
* `patient` - e.g., notifications for me as a patient
* patient membership - e.g., notifications for a practitioner on a care team
* Arbitrary orgs/practitioners/etc. in the relevant resource - e.g., referral destination.
1. Subscribers can determine the *exact* trigger for a notification ([details](#c-4)).

### <a id="c-1"></a>Consideration 1: Implementer Granularity

In the current state, discovery is at the *topic* level. There is no consideration that a topic could be partially implemented.

Moving through the enumerated potential events above, this is problematic at any 'grouped' level (related or arbitrary). When a topic encapsulates multiple discrete events, the granularity is essentially lost.

While it is possible to use filters to select from discrete events within a topic, it gets painful quickly. For example, a topic could be defined with a trigger of "HL7v2 `ADT`" and create a *filter* of `adt-event` to narrow the triggering events to values such as `ADT^A01`, `ADT^A02`, etc.:
* subscriber will need to include the `adt-event` parameter in addition to each filter
* servers either need custom logic to determine which filters apply or will waste compute applying many filters that will never apply to the event (e.g., filters for other events)

The core issue here is that topics were defined as the 'smallest' granularity in the subscriptions framework.

### <a id="c-2"></a>Consideration 2: Subscriber Granularity

In the current state, subscription requests are made at the *topic* level.

Moving through the enumerated potential events above, this is problematic at any 'grouped' level (related or arbitrary). When a topic encapsulates multiple discrete events, the granularity is inaccessible.

While it is possible to use filters to select from discrete events within a topic, it gets painful quickly. For example, if the topic only exposed v2 `ADT` but allowed for a *filter* of `adt-event` for specifics:
* filters would need to include the `adt-event` parameter in addition to each filter
* using event filtering can be brittle - if a new version of the topic adds another event to the group, a subscriber may not get notifications for those events by default
* notifications are only granular to the topic level, so discrete notification events cannot be tied back to the actual triggering event

The core issue here is that topics were defined as the 'smallest' granularity in the subscriptions framework.

### <a id="c-3"></a>Consideration 3: Filter Application

In the current state, filters (and shapes) are defined and requested by *resource type*.

Moving through the enumerated potential events above, this is problematic at any 'grouped' level (related or arbitrary). Filters will be applied to any matching resource, which introduces a lot of complexity.

For example, if a 'core' topic covers all v2 `ADT` notifications, you may additionally want to expose a complex filter for discharge events (e.g., patients in a demographic that were prescibed a given medication for a specific condition). Once that filter is defined, it is *very easy* to mistakenly apply it transfer events.

The core issue here is that there is ambiguity between *filters* and *triggers* since they are *not* directly mapped 1:1.

### <a id="c-4"></a>Consideration 4: Notification Granularity

In the current state, notifications are only traceable to a subscription, which is 1:1 with a topic.

Moving through the enumerated potential events above, this is problematic at any 'grouped' level (related or arbitrary). Even though server implementers knew exactly which trigger caused a notification, subscribers will need to try and reconstruct that based on contents of notifications.

The current design was based on having multiple subscriptions if you need multiple topics. However this ends up introducing scaling issues if you have granular topics. It is not desireable to manage 10s of subscriptions per user.

The core issue here is that topics were defined as the 'smallest' granularity in the subscriptions framework.

## Proposal

To try and distill the issues above:
* `SubscriptionTopic` resources can define any number of *triggers*, whether or not they are related.
* `Subscription` resource map a single `SubscriptionTopic` to a single *subscriber*.
* *Notifications* are mapped to a single subscription.

TLDR: I believe the simplest solution is to merge Event and Resource trigger definitions in topic definitions, which allow for event tagging in notifications.

### <a id="proposal-details"></a>Details

Currently, the `SubscriptionTopic` resource separates out resource-based triggers and event-based triggers. By collapsing the two types into a unified `trigger`, the event coding could be used to tag any notification.

In order to understand how to apply filters to triggers, the `canFilterBy` definitions in topics also need to be collapsed into the same backbone element (or could use tagging like some other solutions).

Conceptually, this moves all of the definitions into an array of `trigger` definitions that include everything each time.

Note that for R4B and R5, I chose to add an extension to `SubscriptionTopic.resourceTrigger` instead of adding several extensions to `eventTrigger`.

In relation to each component:
* **topic**
* **trigger**
* In R6, `resourceTrigger` and `eventTrigger` are collapsed into `trigger`.
* In R4-R5, add an extension with an event concept to `resourceTrigger`, `canFilterBy`, and `notificationShape`.
* **filter**
* Filters are *only* defined per trigger.
* In R6, `canFilterBy` is moved under `trigger`.
* In R4-R5, add an extension with an event concept `canFilterBy`.
* **shape**
* Shapes are *only* defined per trigger.
* In R6, `notificationShape` is moved under `trigger`.
* In R4-R5, add an extension with an event concept `notificationShape`.
* **subscription**
* Subscriptions should/can tag filters by event. If no event is specified, the filter should be applied to any trigger that contains a matching definition.
* **filter**
* In R6, add an event concept element to `filterBy`.
* In R4-R5, add an extension with an event concept to `filterBy`.
* **notification**
* Notification instances can be explicitly linked to a specific `trigger` via the `triggerEvent`. If there is no event defined by a trigger, the granularity stops at the level of the topic.
* In R6, add an event-concept element to `SubscriptionStatus.notificationEvent`.
* In R4B-R5, add an event-concept extension to `SubscriptionStatus.notificationEvent`.
* In R4, use an additional `Parameters.parameter.part` to reference the event concept.

#### Pros

* Very explicit.
* Clean modeling in R6.
* Low conceptual changes.

#### Cons

* Most structural change.
* Backporting this change involves a lot of extensions.

#### Examples

<details>
<summary><code>SubscriptionTopic</code> excerpt (R6)</summary>

"url": "http://example.org/FHIR/SubscriptionTopic/encounter-notifications",
// collapse resourceTrigger, eventTrigger, canFilterBy, and notificationShape into trigger
"trigger": [{
    "description": "An Encounter has been started",
    "resource": "http://hl7.org/fhir/StructureDefinition/Encounter",
    "supportedInteraction": [ "create", "update" ],
    "queryCriteria": {
        "previous": "status:not=in-progress",
        "current": "status=in-progress",
    },
    "event": {
        "coding" : [{
            "system" : "http://terminology.hl7.org/CodeSystem/v2-0003",
            "code" : "A01",
            "display" : "ADT/ACK - Admit/visit notification"
        }],
    },
    "canFilterBy": [{
        "filterParameter": "patient"
    }]
},{
    "description": "An Encounter has been completed",
    "resource": "http://hl7.org/fhir/StructureDefinition/Encounter",
    "supportedInteraction": [ "create", "update" ],
    "queryCriteria": {
        "previous": "status:not=completed",
        "current": "status=completed",
    },
    "event": {
        "coding" : [{
            "system" : "http://terminology.hl7.org/CodeSystem/v2-0003",
            "code" : "A03",
            "display" : "ADT/ACK - Discharge/end visit"
        }],
    },
    "canFilterBy": [{
        "filterParameter": "patient"
    }]
}]

</details>

<details>
<summary><code>SubscriptionTopic</code> excerpt (R4B, R5)</summary>

"url": "http://example.org/FHIR/SubscriptionTopic/encounter-notifications",
"resourceTrigger": [{
    "description": "An Encounter has been started",
    "resource": "http://hl7.org/fhir/StructureDefinition/Encounter",
    "supportedInteraction": [ "create", "update" ],
    "queryCriteria": {
        "previous": "status:not=in-progress",
        "current": "status=in-progress",
    },
    // add the event-trigger extension
    "extension": [{
        "url": "http://hl7.org/fhir/uv/subscriptions-backport/StructureDefinition/event-trigger",
        "valueCoding" : [{
            "system" : "http://terminology.hl7.org/CodeSystem/v2-0003",
            "code" : "A01",
            "display" : "ADT/ACK - Admit/visit notification"
        }]
    }]
}, {
    "description": "An Encounter has been completed",
    "resource": "http://hl7.org/fhir/StructureDefinition/Encounter",
    "supportedInteraction": [ "create", "update" ],
    "queryCriteria": {
        "previous": "status:not=completed",
        "current": "status=completed",
    },
    // add the event-trigger extension
    "extension": [{
        "url": "http://hl7.org/fhir/uv/subscriptions-backport/StructureDefinition/event-trigger",
        "valueCoding" : [{
            "system" : "http://terminology.hl7.org/CodeSystem/v2-0003",
            "code" : "A03",
            "display" : "ADT/ACK - Discharge/end visit"
        }]
    }]
}],
"canFilterBy": [{
    "resource": "Encounter",
    "filterParameter": "patient",
    // add the event-trigger extension
    "extension": [{
        "url": "http://hl7.org/fhir/uv/subscriptions-backport/StructureDefinition/event-trigger",
        "valueCoding" : [{
            "system" : "http://terminology.hl7.org/CodeSystem/v2-0003",
            "code" : "A01",
            "display" : "ADT/ACK - Admit/visit notification"
        }]
    }]
}, {
    "resource": "Encounter",
    "filterParameter": "patient",
    // add the event-trigger extension
    "extension": [{
        "url": "http://hl7.org/fhir/uv/subscriptions-backport/StructureDefinition/event-trigger",
        "valueCoding" : [{
            "system" : "http://terminology.hl7.org/CodeSystem/v2-0003",
            "code" : "A03",
            "display" : "ADT/ACK - Discharge/end visit"
        }]
    }]
}]

</details>

<details>
<summary><code>Subscription</code> excerpt (R6)</summary>

"resourceType": "Subscription",
"topic": "http://example.org/FHIR/SubscriptionTopic/encounter-notifications",
"filterBy": [{
    "resourceType": "Encounter",
    "filterParameter": "patient",
    "value": "Patient/example",
    // add trigger event code
    "forEvent": [{
        "system" : "http://terminology.hl7.org/CodeSystem/v2-0003",
        "code" : "A01",
        "display" : "ADT/ACK - Admit/visit notification"
    }]
}, {
    "resourceType": "Encounter",
    "filterParameter": "patient",
    "value": "Patient/example",
    // add trigger event code
    "forEvent": [{
        "system" : "http://terminology.hl7.org/CodeSystem/v2-0003",
        "code" : "A03",
        "display" : "ADT/ACK - Discharge/end visit"
    }]
}]    

</details>

<details>
<summary><code>Subscription</code> excerpt (R5)</summary>

"resourceType": "Subscription",
"topic": "http://example.org/FHIR/SubscriptionTopic/encounter-notifications",
"filterBy": [{
    "resourceType": "Encounter",
    "filterParameter": "patient",
    "value": "Patient/example",
    // add event-code extension
    "extension": [{
        "url": "http://hl7.org/fhir/uv/subscriptions-backport/StructureDefinition/event-code",
        "valueCoding" : [{
            "system" : "http://terminology.hl7.org/CodeSystem/v2-0003",
            "code" : "A01",
            "display" : "ADT/ACK - Admit/visit notification"
        }]
    }]
}, {
    "resourceType": "Encounter",
    "filterParameter": "patient",
    "value": "Patient/example",
    // add event-code extension
    "extension": [{
        "url": "http://hl7.org/fhir/uv/subscriptions-backport/StructureDefinition/event-code",
        "valueCoding" : [{
            "system" : "http://terminology.hl7.org/CodeSystem/v2-0003",
            "code" : "A03",
            "display" : "ADT/ACK - Discharge/end visit"
        }]
    }]
}]    

</details>

<details>
<summary><code>Subscription</code> excerpt (R4, R4B)</summary>

"resourceType": "Subscription",
"criteria": "http://example.org/FHIR/SubscriptionTopic/encounter-notifications",
"_criteria": {
    "extension": [{
        "url": "http://hl7.org/fhir/uv/subscriptions-backport/StructureDefinition/backport-filter-criteria",
        "valueString": "Encounter?patient=Patient/123",
        // add event code extension
        "_valueString": [{
            "extension": [{
                "url": "http://hl7.org/fhir/uv/subscriptions-backport/StructureDefinition/event-code",
                "valueCoding" : [{
                    "system" : "http://terminology.hl7.org/CodeSystem/v2-0003",
                    "code" : "A01",
                    "display" : "ADT/ACK - Admit/visit notification"
                }]
            }]
        }]
    }, {
        "url": "http://hl7.org/fhir/uv/subscriptions-backport/StructureDefinition/backport-filter-criteria",
        "valueString": "Encounter?patient=Patient/123",
        // add event code extension
        "_valueString": [{
            "extension": [{
                "url": "http://hl7.org/fhir/uv/subscriptions-backport/StructureDefinition/event-code",
                "valueCoding" : [{
                    "system" : "http://terminology.hl7.org/CodeSystem/v2-0003",
                    "code" : "A03",
                    "display" : "ADT/ACK - Discharge/end visit"
                }]
            }]
        }]
    }]
}

</details>

<details>
<summary><code>SubscriptionStatus</code> excerpt (R4B, R5)</summary>

"resourceType": "SubscriptionStatus",
"type": "event-notification",
"eventsSinceSubscriptionStart": "3",
"notificationEvent": [{
    "eventNumber": "2",
    "timestamp": "2020-05-29T11:44:13.1882432-05:00",
    "focus": {
        "reference": "https://example.org/fhir/Encounter/..."
    },
    // add event-code extension
    "extension": [{
        "url": "http://hl7.org/fhir/uv/subscriptions-backport/StructureDefinition/event-code",
        "valueCoding" : [{
            "system" : "http://terminology.hl7.org/CodeSystem/v2-0003",
            "code" : "A01",
            "display" : "ADT/ACK - Admit/visit notification"
        }]
    }]
}, {
    "eventNumber": "3",
    "timestamp": "2020-05-29T11:44:13.1882432-05:00",
    "focus": {
        "reference": "https://example.org/fhir/Encounter/..."
    },
    // add event-code extension
    "extension": [{
        "url": "http://hl7.org/fhir/uv/subscriptions-backport/StructureDefinition/event-code",
        "valueCoding" : [{
            "system" : "http://terminology.hl7.org/CodeSystem/v2-0003",
            "code" : "A03",
            "display" : "ADT/ACK - Discharge/end visit"
        }]
    }]
}],
"subscription": {
    "reference": "https://example.org/fhir/r5/Subscription/123"
},
"topic": "http://example.org/FHIR/SubscriptionTopic/encounter-notifications"

</details>

<details>
<summary><code>Parameters</code> (status) excerpt (R4)</summary>

"resourceType": "Parameters",
"parameter": [{
    "name": "subscription",
    "valueReference": {
        "reference": "https://example.org/fhir/Subscription/admission"
    }
}, {
    "name": "topic",
    "valueCanonical": "http://hl7.org/SubscriptionTopic/admission"
}, {
    "name": "type",
    "valueCode": "event-notification"
}, {
    "name": "events-since-subscription-start",
    "valueString": "3"
}, {
    "name": "notification-event",
    "part": [{
        "name": "event-number",
        "valueString": "2"
    }, {
        "name": "timestamp",
        "valueInstant": "2020-05-29T11:44:13.1882432-05:00"
    }, {
        "name": "focus",
        "valueReference": {
            "reference": "https://example.org/fhir/Encounter/..."
    }}, {
        // add for-event part
        "name": "for-event",
        "valueCoding" : [{
            "system" : "http://terminology.hl7.org/CodeSystem/v2-0003",
            "code" : "A01",
            "display" : "ADT/ACK - Admit/visit notification"
        }]
    }]
}, {
    "name": "notification-event",
    "part": [{
        "name": "event-number",
        "valueString": "3"
    }, {
        "name": "timestamp",
        "valueInstant": "2020-05-29T11:44:13.1882432-05:00"
    }, {
        "name": "focus",
        "valueReference": {
            "reference": "https://example.org/fhir/Encounter/..."
    }}, {
        // add for-event part
        "name": "for-event",
        "valueCoding" : [{
            "system" : "http://terminology.hl7.org/CodeSystem/v2-0003",
            "code" : "A03",
            "display" : "ADT/ACK - Discharge/end visit"
        }]
    }]
}]

</details>