---
tags: Microservice Pattern, Microservice
---

# Managing transactions with sagas

why transaction?

維持資料一致性

> Transactions are an essential ingredient of every enterprise application. Without transactions it would be impossible to maintain **data consistency**.

transaction特性:
ACID -> Atomicity(不可分割性), Consistency(一致性), Isolation(隔離性), Durability(持久性)

[ACID介紹](https://zh.wikipedia.org/wiki/ACID)

single service可以透過ACID transaction處理，但若要跨service更新data，需使用saga處理

[Saga介紹](https://docs.microsoft.com/zh-tw/azure/architecture/reference-architectures/saga/saga)

但Saga只滿足ACD三種特性，缺Isolation
-> countermeasures

monolithic: 傳統ACID transaction
microservice: Saga + countermeasures

## 4.1 Transaction management in a microservice architecture
現在要處理transaction已經是很容易的事了(single database)，許多frameworks跟libraries都有好用的API方便開發(ex. Spring framework -> @Transactional)。

但在multiple databases與有message brokers情況下要處理transaction就複雜多了。如前面提到的，傳統transaction在現代的系統架構下已逐漸不可行，因此基於microservice的application都使用saga。

### 4.1.1 The need for distributed transactions in a microservice architecture
以createOrder()為例，此operation要執行以下操作:
* verify that the consumer can place an order
* verify the order details
* authorize the consumer’s credit card
* create an Order in the database.

在microservice架構下需至以下service操作:


在這樣的情況下要維持資料一致性(data consistency)是有挑戰的

### 4.1.2 The trouble with distributed transactions
distributed transaction依照[X/Open XA](https://zh.wikipedia.org/wiki/X/Open_XA)來達到Distributed Transaction Processing(DTP)。XA使用[二階段提交](https://zh.wikipedia.org/wiki/%E4%BA%8C%E9%98%B6%E6%AE%B5%E6%8F%90%E4%BA%A4)(two-phase commit,2PC)確保transaction所有participants都一起commit或rollback。

> An XA-compliant technology stack consists of
> * XA-compliant databases and message brokers
> * database drivers
> * messaging APIs
> * an interprocess communication mechanism that propagates the XA global transaction ID.

大部分的SQL資料庫都有XA兼容，部分message brokers也是，其他如Java EE application，也可使用JTA(Java Transaction API)處理distributed transactions。

但還是有不少技術無法支援distributed transactions，如NoSQL資料庫(MongoDB、Cassandra)，較新的message brokers(RabbitMQ、Apache Kafka)等等。

distributed transactions還有一個很大的問題是synchronous IPC(Inter-Process Communication），synchronous IPC必須在全部participating services都可用(available)的情況下才能commit，這樣大大減低availability。

> As described in chapter 3, the availability is the product of the availability of all of the participants in the transaction. If a distributed transaction involves two services that are 99.5% available, then the overall availability is 99%, which is significantly less.

> 根據[CAP定理](https://zh.wikipedia.org/wiki/CAP%E5%AE%9A%E7%90%86)，分散式系統只能滿足三項中的兩項而不可能滿足全部三項，而作者認為三項中可用性（Availability）是現代軟體架構中最需要的。

saga改善了distributed transactions的問題，打造loosely coupled且asynchronous的services。

### 4.1.3 Using the Saga pattern to maintain data consistency
Saga其實就是sequence of local transactions，每個local transactions都做普通的ACID transaction而已。以下將用例子解說Saga流程:

#### AN EXAMPLE SAGA: THE CREATE ORDER SAGA


> This saga consists of the following local transactions:
> 1. Order Service: Create an Order in an APPROVAL_PENDING state.
> 2. Consumer Service: Verify that the consumer can place an order.
> 3. Kitchen Service: Validate order details and create a Ticket in the CREATE_PENDING.
> 4. Accounting Service: Authorize consumer’s credit card.
> 5. Kitchen Service: Change the state of the Ticket to AWAITING_ACCEPTANCE.
> 6. Order Service: Change the state of the Order to APPROVED.

當一個local transaction完成後，Saga才會去觸發下一個流程，達到異步(buffer)、跨服務的功能。

Saga還是有以下挑戰:
* lack of isolation
* rolling back

#### SAGAS USE COMPENSATING TRANSACTIONS TO ROLL BACK CHANGES
傳統ACID transactions很容易可以達成roll back(直接undo DB即可)，但Saga的roll back會牽扯回很多步，如圖:


在Tn+1步驟發生錯誤時，必須要回頭執行**compensation transactions**，從Cn(Tn的compensation transaction)到C1

以Create Order為例，會有以下對應原來transaction的compensation transactions:


有些transaction可能是read-only或一定會成功的(失敗只要單transaction retry就好)，便不需要compensation transactions

> To see how compensating transactions are used, imagine a scenario where the authorization of the consumer’s credit card fails. In this scenario, the saga executes the following local transactions:
> 1. Order Service—Create an Order in an APPROVAL_PENDING state.
> 2. Consumer Service—Verify that the consumer can place an order.
> 3. Kitchen Service—Validate order details and create a Ticket in the CREATE_PENDING state.
> 4. Accounting Service—Authorize consumer’s credit card, which fails.
> 5. Kitchen Service—Change the state of the Ticket to CREATE_REJECTED.
> 6. Order Service—Change the state of the Order to REJECTED.

## 4.2 Coordinating sagas
Saga需要coordination logic來管理整個Saga transaction流程，coordination logic分為兩種:
* Choreography: ***Distribute*** the decision making and sequencing among the saga participants. They primarily communicate by exchanging events.
* Orchestration: ***Centralize*** a saga’s coordination logic in a saga orchestrator class. A saga orchestrator sends command messages to saga participants telling them which operations to perform.

### 4.2.1 Choreography-based sagas
Choreography不是靠central coordinator來告訴saga participant該怎麼做，而是靠participant自行Subscribe/Publish:


> The happy path through this saga is as follows:
>1. Order Service creates an Order in the APPROVAL_PENDING state and publishes an OrderCreated event.
>2. Consumer Service consumes the OrderCreated event, verifies that the consumer can place the order, and publishes a ConsumerVerified event.
>3. Kitchen Service consumes the OrderCreated event, validates the Order, creates a Ticket in a CREATE_PENDING state, and publishes the TicketCreated event.
>4. Accounting Service consumes the OrderCreated event and creates a CreditCardAuthorization in a PENDING state.
>5. Accounting Service consumes the TicketCreated and ConsumerVerified events, charges the consumer’s credit card, and publishes the CreditCardAuthorized event.
>6. Kitchen Service consumes the CreditCardAuthorized event and changes the state of the Ticket to AWAITING_ACCEPTANCE.
>7. Order Service receives the CreditCardAuthorized events, changes the state of the Order to APPROVED, and publishes an OrderApproved event.

#### RELIABLE EVENT-BASED COMMUNICATION
* The first issue is ensuring that a saga participant updates its database and publishes an event as part of a database transaction.
-> transactional messaging

* The second issue you need to consider is ensuring that a saga participant must be able to map each event that it receives to its own data.
-> correlation id

#### BENEFITS AND DRAWBACKS OF CHOREOGRAPHY-BASED SAGAS
* benefits:
* Simplicity:
在create、update或delete business objects時publish
* Loose coupling
* drawbacks:
* More difficult to understand:
沒有central coordinator，不好追蹤
* Cyclic dependencies between the services:
(Order Service -> Accounting Service -> Order Service)
design smell(設計異味)
* Risk of tight coupling:
service可能需要subscribe多個event，像上面例子Accounting Service訂閱了Order Service全部的event，可能造成Order Service跟Accounting Service的tight coupling

### 4.2.2 Orchestration-based sagas
Orchestration架構是由orchestrator透過async reply-style與participants溝通:


> the flow for the happy path is as follows:
> 1. The saga orchestrator sends a Verify Consumer command to Consumer Service.
> 2. Consumer Service replies with a Consumer Verified message.
> 3. The saga orchestrator sends a Create Ticket command to Kitchen Service.
> 4. Kitchen Service replies with a Ticket Created message.
> 5. The saga orchestrator sends an Authorize Card message to Accounting Service.
> 6. Accounting Service replies with a Card Authorized message.
> 7. The saga orchestrator sends an Approve Ticket command to Kitchen Service.
> 8. The saga orchestrator sends an Approve Order command to Order Service.

其中，第8步雖然Order Service request channel就是Order Service的一部分(可以直接更新)，但就理論而言，還是要將它視為其他的participant。

#### MODELING SAGA ORCHESTRATORS AS STATE MACHINES
要表示Orchestration架構的全部情況很難(Choreography也是)，**state machine**是一個不錯的總覽方式:


#### SAGA ORCHESTRATION AND TRANSACTIONAL MESSAGING
> As described in chapter 3, a service must use **transactional messaging** in order to atomically update the database and publish messages.

如上述例子中，Kitchen Service收到command message後將更新database並回傳reply message

#### BENEFITS AND DRAWBACKS OF ORCHESTRATION-BASED SAGAS
* benefits:
* Simpler dependencies:
no cyclic dependencies
* Less coupling
* Improves separation of concerns and simplifies the business logic:
簡化business logic
* drawbacks:
* smart orchestrator & dumb services:
centralizing too much business logic in the orchestrator，可在設計時避免掉

作者建議都使用Orchestration架構設計，但設計上要特別注意去避免它的缺點

## 4.3 Handling the lack of isolation
Isolation讓ACID transactions能同時執行多個transactions，好像每個transaction都有單獨時間存取database的錯覺。

> The challenge with using sagas is that they lack the isolation property of ACID transactions. That’s because the updates made by each of a saga’s local transactions are **immediately visible to other sagas** once that transaction commits.

Saga滿足ACD特性的原因:
* Atomicity:
roll back機制
* Consistency:
service自身的referential integrity(參照完整性)由local database負責，service之間的referential integrity則透過service負責
* Durability:
local database負責

lack of isolation這件事在database文件中稱為anomalies(異常現象)，可透過countermeasures解決。

> 雖然isolation看似是必備的，但實務上常為了performance降低對isolation的要求

### 4.3.1 Overview of anomalies
anomalies可能是以下三種情況造成:
* Lost updates:
saga在沒有讀到另一個saga更新的情況下又覆寫(overwrite)
* Dirty reads:
transaction或saga讀到尚未完成的更新
* Fuzzy/nonrepeatable reads:
兩saga同時讀同data，且一saga進行更新 -> get different results

#### LOST UPDATES example
1. The first step of the Create Order Saga creates an Order.
2. While that saga is executing, the Cancel Order Saga cancels the Order.
3. The final step of the Create Order Saga approves the Order.
> the Create Order Saga ignores the update made by the Cancel Order Saga and overwrites it.

#### DIRTY READS example
1. Cancel Order Saga—Increase the available credit.(增加信用卡金額)
2. Create Order Saga—Reduce the available credit.(減少信用卡金額)
3. Cancel Order Saga—A compensating transaction that reduces the available credit.
> the Create Order Saga does a dirty read of the available credit that enables the consumer to place an order that exceeds their credit limit

### 4.3.2 Countermeasures for handling the lack of isolation
其實之前舉的例子中已經有Countermeasures相似的例子，service會透過更新狀態(states)來避免lack of isolation，如Order service透過*_PENDING states表達其他saga的使用狀態。

以下將介紹幾種Countermeasures:
* Semantic lock— An application-level lock.
* Commutative updates— Design update operations to be executable in any order.
* Pessimistic view— Reorder the steps of a saga to minimize business risk.
* Reread value— Prevent dirty writes by rereading data to verify that it’s unchanged before overwriting it.
* Version file— Record the updates to a record so that they can be reordered.
* By value— Use each request’s business risk to dynamically select the concurrency mechanism.

首先必須先補充Saga內的transactions type

#### THE STRUCTURE OF A SAGA
Saga內可以分成三種transactions:
* Compensatable transactions: Must support roll back
* Pivot transaction: The saga’s go/no-go transaction.If it succeeds, then the saga runs to completion
* Retriable transactions: Guaranteed to complete


#### SEMANTIC LOCK
單純的flag表示record還沒committed(可能還會更動)，讓系統很方便可以管理，但管理lock只完成了一半，除了管理lock外還要設計各transaction遇到lock狀態時的行為

#### COMMUTATIVE UPDATES
透過相反的服務達到roll back的效果，便不會因為roll back影響其他saga的更新
ex. 取消扣款直接改為匯款

#### PESSIMISTIC VIEW
**reorder** steps降低dirty read機率，以Cancel Order來說:
1. Order Service—Change the state of the Order to cancelled.
2. Delivery Service—Cancel the delivery.
3. Customer Service—Increase the available credit.

#### REREAD VALUE
reread value countermeasure用以防止lost updates，在更新之前執行reread確認，如果record改變，則整個Saga流程將流程將取消(roll back)

#### VERSION FILE
類似SEMANTIC LOCK概念但是直接把lock寫入record中

#### BY VALUE
依照情況決定使用Saga或distributed transactions:
低風險: Saga
高風險: distributed transactions

## 4.4 The design of the Order Service and the Create Order Saga

下圖為用saga orchestrator詮釋的Order Service:


以下將依依介紹各部分

### 4.4.1 The OrderService class
> The OrderService class is a domain service called by the service’s API layer.

OrderService class的職責為**creating and managing orders**，並透過調用其他service的API來完成


以程式碼理解OrderService的createOrder() method流程:


### 4.4.2 The implementation of the Create Order Saga

下圖為CreateOrderSaga與其相關的class:


* CreateOrderSaga:
定義saga的state machine，調用CreateOrderSagaState產生command message來與各saga participant的proxy classes溝通(using message channel)
* CreateOrderSagaState:
saga的persistent state用以產生command messages
* Saga participant proxy classes:
每個proxy class定義了saga participant的messaging API，其需包含：
* command　channel
* command message types
* reply types

以上class由Eventuate Tram Saga framework撰寫，Eventuate Tram Saga framework包含[domain-specific language (DSL)](https://www.huanlintalk.com/2008/05/domain-specific-languages.html)，用以定義saga’s state machine，方便執行運作。

#### THE CREATEORDERSAGA ORCHESTRATOR
saga definition(處理state machine)為CreateOrderSaga的核心，如下程式碼(DSL):


接著看CreateOrderSaga如何以DSL與其他participant溝通:


其中:
* invokeParticipant():
產生forward transaction
* onReply():
當收到成功回覆後要做的事
* withCompensation():
產生compensating transaction

#### THE CREATEORDERSAGASTATE CLASS
CreateOrderSagaState class裡的instance是由OrderService產生並保存於DB中，其主要職責為產生傳給saga participants的messages


CreateOrderSaga調用CreateOrderSagaState來產生command messages，並寄送給SagaParticipantProxy classes的endpoints

#### THE KITCHENSERVICEPROXY CLASS
KitchenServiceProxy class為Kitchen Service定義了3個endpoints:
* create
* confirmCreate
* cancel

CommandEndpoint指定了不同command type


Proxy並不是必要的class，但使用proxy能有兩個好處:
* 減少寄送wrong message的機會(like filter)
* well-defined API(抽象)

#### THE EVENTUATE TRAM SAGA FRAMEWORK
Eventuate Tram Saga framework包了saga orchestrators、saga participants的功能，並用Eventuate Tram達到transactional messaging能力。


以時序圖來說明OrderService creates a saga流程:


> 1. OrderService creates the CreateOrderSagaState.
> 2. It creates an instance of a saga by invoking the SagaManager.
> 3. The SagaManager executes the first step of the saga definition.
> 4. The CreateOrderSagaState is invoked to generate a command message.
> 5. The SagaManager sends the command message to the saga participant (the Consumer Service).
> 6. The SagaManager saves the saga instance in the database.

接著是SagaManager receives a reply from Consumer Service流程:


> 1. Eventuate Tram invokes SagaManager with the reply from Consumer Service.
> 2. SagaManager retrieves the saga instance from the database.
> 3. SagaManager executes the next step of the saga definition.
> 4. CreateOrderSagaState is invoked to generate a command message.
> 5. SagaManager sends the command message to the specified saga participant(Kitchen Service).
> 6. SagaManager saves the update saga instance in the database.

### 4.4.3 The OrderCommandHandlers class
OrderCommandHandlers class的職責為處理command messages(like router):



### 4.4.4 The OrderServiceConfiguration class
Order Service使用Spring framework，[@Configuration class與Spring @Beans](https://matthung0807.blogspot.com/2019/04/spring-configuration_28.html)是Spring framework管理的好工具，可以透過其建立OrderServiceConfiguration class:


## Summary
microservice在管理transaction與設計business logic上都有很大的不同，幸好有saga orchestrators(simple state machines)、 countermeasures與framework等工具協助開發。