# 6-2. Event Design

In the previous courses, we have provided the event usage of "Lakeside" and "MA". By now, you should have understood what an event is and be more familiar with events through coding and implementations.

In this topic, we start to share "how" FST designs events to keep reference records so as to create an anchor.

Usually, events are used to
1. keep transaction/status/time series data (with `timestamp` nature)
2. centralise reference data record(data exchange between heterogeneous systems)
3. create data anchor for analyzing
4. create data lineage for auditing

We will be using Lakeside as an example and divide it into several aspects to explain step-by-step how we design events. Through this course, you will be familiar with:

- [Event Concept](#Event-Concept)
- [Event Schema](#Event-Schema)
- [Event Sourcing](#Event-Sourcing)
- [Event Analysis](#Event-Analysis)

---
## Event Concept
In computing, a graph database (GDB) is a database that uses graph structures for semantic queries with nodes, edges, and properties to represent and store data. (source: [Wikipedia](https://en.wikipedia.org/wiki/Graph_database))

Based on this GDB, in LOC, we design events used to record data and communicate with the business units with the graphical format of data. (Slide P2)

#### Event Format
The only thing LOC can keep is an "event" in the event store instead of the original data. The event format is:

- **Source DID**
- a string
- used to represent ++from++ where/what/who executes the event
- **Target DID**
- a string
- used to represent ++to++ where/what/who executes the event
- **Label Name**
- a string
- used to represent the event name; that is, the label itself
- (optional) **Meta**
- a string
- used to record an annotation or unstrunctured data
- **Type**
- by default
- used to categorise events into different groups

When emitting an event, we use the event format as follows.

{
 sourceDID: `Item: ${item.Name}`,  // event source
 targetDID: `Order: ${item.OrderId}`,  // event target
 labelName: `Set_Order: ${item.OrderId}`,  // event label name
 meta: JSON.stringify(item),  // convert item object to JSON string
 type: 'default',  // event group
}

Even though there is no time-stamp in coding, it will automatically generated. An example of the emitted event in the event store is:

Here are some other event examples: (click to see more)

:::spoiler Example 1
- In the Lakeside example, we firstly build a data process in which there is a set of events to record the order information. That is, we set order items onto each order with the item quantity stored as meta.
- Source DID
- `item:hamburger`
- Target DID
- `order:200`
- Label Name
- `Set_Order`
- Meta
- `10`
:::

:::spoiler Example 2
- In the previous courses, we also have the moving average use case in which we query a DB for stock prices and calculate moving average values to return as a result. Because MA is the key value we would like to keep, we design a set of events to record this vital information.
- Source DID
- `20MA`
- Target DID
- `stock_code:1301`
- Label Name
- `Calculate_20MA`
- Meta
- `${MA values}`
:::

:::spoiler Example 3
- The customer rating department would like to label `VIP_2022` on an end-customer whose email is `[e-mail]`, for the reason that this customer meets the 2022 VIP requirement.
- Source DID
- customer rating department
- Target DID
- `user:[e-mail]`
- Label Name
- `VIP_2022`
- Meta
- `to meet the 2022 VIP requirement`

:::

:::spoiler Example 4
- Record a piece of customer behaviour information to be transmitted from System A to System B
- Source DID
- `System A`
- Target DID
- `System B`
- Label Name
- `Send_Customer_Action_Data:open_page`
- Meta
- `{"customer_id":"abc123","action_name":"open_page","detail":"page/xyz456"}`
:::

:::spoiler Example 5
- Record that fact that LOC user accesses a certain data catalogue
- Source DID
- `loc_user:[e-mail]`
- Target DID
- `data_catalogue:Unit_A_FOOBAR`
- Label Name
- `Access_Data_Catalogue`
- Meta
- `Purpose: xxxxxx`
:::

:::spoiler Example 6
- A department head puts an authorisation and attribute label on a LOC user (advanced)
- Source DID
- `##abac:unit:loc_unit_a`
- Target DID
- `loc_user:[e-mail]`
- Label Name
- `##abac:label:analyst`
- Meta
- `##abac:digital_signature:loc_unit_a:0x53a087f64fa9892209c02eb86d4c08c14be6436fed42bcc76a713b583254a632`
:::

The reason to create events is because they could be visualised when it comes to event sourcing, data discovery, and data lineage. It can therefore reduce the reading and analysing complexity.

#### Event Naming Method
You can freely set the lineage relationship of events, depending on what kind of network diagram you want to draw in your scenario. Generally speaking, we recommend using the "Target DID" as an anchor to collect events. That means in the design stage, we will design the events direction based on which Target DID to be aggregated at the end, just like the "many-to-one relationship" approach.

Still, FST recommends 2 models to design events in a more standardised way. (Slide P3)

1. **Verb** (to record the action that Source DID does to Target DID): Label Name usually starts with a verb that indicates to set/update value on Target DID in the computing world.
- to make Label Name start with a verb. eg, `Set_XXX`, `Send_XXX`, `Get_XXX`, `Trigger_ABC`,etc.


2. **Attribute** (to record the status and/or attribute of Target DID)
- to make Label Name as an attribute or a pure label name. eg,`VIP_2022`, `High Risk`, `Department_A`, `Ignore the calculation conditions:ABC123`, etc.


---
## Demo Case - Lakeside Oasis Café

Next, we take the Lakeside example to explain how we design events and divide the entire business flow into a collection of events. (Slide P4)


---
## Event Schema

The event schema is used to help you clarify the business workflow of the data structure, allowing business units to communicate in their domain language/knowledge so that data can be effectively extracted and recorded in a highly readable way. This way reduces the costs of formatting and transforming data for IT units in the sense that it is easier for IT units to align with other units, making data integration and re-application more efficient. (Slide P5)

In the process of system development, we use several references as anchors to designing event:
- Data integration cross systems / workflow / domain(joint key value/vlookup reference)
- Results of Sync/Async mode workflow/process
- Data aggregation/analysis values
- Data lineage for audit trail

So how do we specify these [anchors](#Event-Naming-Method) to help us develop events and data processes? This echoes the cycle we have repeatedly mentioned before.

In the BA/SA analysis process, we firstly identify stakeholders who involve with this business process. Using events to combine records, once the earlier mentioned anchors are determined(eg, time series data, async mode, cross systems lineage), is the merit of events. (Slide P6)


After the above analysis process, we can split the entire process into an integration of multiple business/data processes. In these processes, through designing proper events, we can connect each event by mapping with the source / target DIDs, thus connecting all the relevant records by each department/system/process and drawing out complete lineage network. Through this way, the correlation between multiple systems can be recorded by a unified event, which is beneficial to subsequent applications and auditing. (Slide P7)


#### Lakeside Event Schema

With the event structure introduced previously, we can use the emit event function to create events. Therefore, whenever we have important additional information or items to be disclosed, we recommend separating them with "colon" or "semi-colon" on TargetDID/SourceDID/LabelName, to distinguish events with clearer names. Take Lakeside as an example. Each order item is specified by using "item:\[itemName]" to make each one more clear and unique. (Slide P8)


---
## Event Sourcing

Event Sourcing ensures that all changes to application state are stored as a sequence of events. (source: [Martin Flower](https://martinfowler.com/eaaDev/EventSourcing.html))

In this paragraph, we are illustrating two features of event sourcing:

- Events are append only and undeletable by default.
- Every event has auto-generated time-stamp to record date and time.

In order to maintain complete event records for traceability, all events are recorded in the "append-only" way, which cannot be deleted by default.

Therefore, to use the events of time series data, we perform event sourcing to search and aggregate events to obtain the most recent status of these data. If we need to cancel an event, we emit an event to offset the previous one, similar to the journal entry in accounting. In other words, we emit another event that carries a negative one value to cross out the previous order, shown in the graph below. (Slide P9)


---
## Event Analysis

After we store several events in LOC, in addition to aggregating them through event sourcing at a specific point of time for further usage, we can dive into these events with graphical visualisation tools to make more detailed analyses.

For analytics, we can retrieve event information with three dimensions - "Source"", "Target" and "Label Name", and extract them for different decision-making.

Take the Lakeside as an example. In the event of setting order, if we focus on "Target", we can conduct order/customer analyses, monitor daily orders and order conversion ratio, etc. If we focus on "Source", we can track the sales of each meal and hot-selling products, and ensure the inventory of each product and so on. Finally, if "LabelName" is of our interest, we focus on the category of "Set_Order to observe the customer behaviour - whether they put an order physically at the store or online, a useful way to analyse the impact from pandemic.

#### Dashboard (BI tools)
As long as we can get event data, such as through HTTP API, the results of the event search can be extracted by triggering the API route deployed by LOC. Afterwards, we can then display statistical data on a dashboard that you favour. (Slide P11)


#### Lineage for Audit
By using the graphic DB or other frameworks, events can be drawn as a collection of points and lines, and connecting them as a network graph allows us to analyse or audit further. (Slide P12)


---
## Advanced Application

Whenever we need to develop new processes, applications, or even systems, LOC ensures smooth deployment. With event, we can integrate data from the new processes/applications/systems effortlessly and reduce the analysis complexity.

Take Lakeside as an example, if we need to include the "Member System" into original one, we can add the new "Set_Member" data process and design a new event schema "Order -> Member". With more events accumulated, through event sourcing, we can easily extend the event lineage and ensure auditability. (Slide P13)


---
###### tags: `Workshop`