###### tags: `HPCAI`
# InfiniBand Transport & Software Interface

[TOC]
## InfiniBand Transport

### [InfiniBand](https://youtu.be/fLwTpP6GKao)
- 無限頻寬，縮寫IB
- High-performance computing
- High throughput and low latency

### The Transport Layer
- **Open System Interconnection** (OSI) Model 開放式系統互聯模型
- **The transport layer**
- Connection-oriented communication 連接導向式通訊

- 在傳送資料之前先建立連線
- TCP屬於連接導向式通訊
- :+1:優點：一旦建立好連線，網路流量是固定的，不會被其他通訊影響
- :-1:缺點：需要預先建立連線，資源成本較高，無法分享網路頻寬
- Reliability 可靠性
- 可以檢查資料是否損壞，並通過向傳送者傳ACK或NACK訊息確認正確接收。自動重發請求方案可用於重新傳輸遺失或損壞的資料
- In-order delivery 相同次序交付
- 網路層通常不保證封包到達順序與傳送順序相同
- Data integrity 資料完整性
- 確保資訊或資料不被未授權的篡改或在篡改後能夠被迅速發現
- Flow control/ rate control 流量控制
- 控制兩個節點之間的資料傳輸速率以阻止快速的傳送者傳輸超出接收緩衝器所能承受的資料
- congestion control 壅塞控制
- 可以控制進入到電信網路的流量
- Multiplexing (of multiple data sources) 多路復用、多工
- 埠可以在單個節點上提供多個端點
- 

:::danger
Not all transport services provide all above


OSI定義五個級別的傳輸模式(TP0~TP4)

- [Reference](https://notfalse.net/24/tcp-flow-control)

### RDMA operation
- Send/ Receive
- Two-sided communication
- RDMA write
- One-sided communication, responder is not involved
- RDMA read
- One-sided communication, responder is not involved
- Atomic operations
- One-sided communication, read-modify-write transaction

### InfiniBand transport services
:::warning
- **Reliable:** exactly once, in-order delivery, more slowly and with less scalability
- **Unreliable**
- **Connection:** one QP send/receive with exactly one QP
- **Datagram:** one QP send/receive with any QP
:::

- **Reliable Connection** (RC)
- Queue Pair is associated with only one other QP.
- Messages transmitted by the send queue of one QP are reliably delivered to receive queue of theother QP.
- Packets are delivered in order.
- A RC connection is very similar to a TCP connection.

- **Unreliable Datagram** (UD)
- A Queue Pair may transmit and receive single-packet messages to/from any other UD QP.
- Ordering and delivery are not guaranteed, and delivered packets may be dropped by the receiver.
- Multicast messages are supported (one to many).
- A UD connection is very similar to a UDP connection.

- **Unreliable Connection** (UC)
- A Queue Pair is associated with only one other QP.
- The connection is not reliable so packets may be lost.
- Messages with errors are not retried by the transport, and error handling must be provided by a higher level protocol.

- **Reliable Datagram** (RD)
- Reliable transport, not-connected
- First try to separate the transport from the software interface
- ==Never implemented==
- complicated protocol
- not efficient protocol
> UD doesn't support RDMA but is veryscalable.
> RD doesn't scale well but supports RDMA


### Reliable Connection vs. Unreliable Datagram
- **Reliable Connection** (RC) - the transport of choice
- Has scalability limitations
- The solution: **Dynamically Connected (DC) Transport Service**
:::info
**Scalability可擴縮性**
是否能有效利用不斷增加的CPU
ex. **search engine** - support increasing numbers of users, and the number of topics it indexes.
:::

- [Reference](https://www.openfabrics.org/images/eventpresos/workshops2014/DevWorkshop/presos/Monday/pdf/05_DC_Verbs.pdf)

- ***Scalable Connection-oriented Reliable*** Transport
- The best of both worlds - supports RDMA, with UD like scalability (single QP for multiple destinations)
- The winner is **Dynamically Connected (DC)**

| |UD|UC|RC|DC|
|------|:------:|:------:|:------:|:------:|
|Send/Receive|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|RDMA Write|:x:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
|RDMA Read/Atomic|:x:|:x:|:heavy_check_mark:|:heavy_check_mark:|
|Max Send Size|MTU|2GB|2GB|2GB|
|Reliability|:x:|:x:|:heavy_check_mark:|:heavy_check_mark:|
|Scalability(per-process for N prosesses)|1|N|N|1|

### Dynamically Connected transport
1. Reduces #QPs per node
2. Supports RDMA and Atomic Operations
3. Reliable
4. Scalable: One QP services multiple destinations
5. **Connection setup** is expensive slow operation
6. Fast, inexpensive and fully offload

- [Reference](https://www.openfabrics.org/images/2018workshop/presentations/303_ARosenbaum_DynamicallyConnectedTransport.pdf)

#### DC Example
- one requestor + two responders
- User is sending:
- Two messagers to responder A (SND1, SND2)
- One messager to responder B (SND3)

>主機通道配接器（HCA）通常用於描述InfiniBand的介面卡。

:::warning
Cannot be connected to more than one responder
:::

### Reliability Implementation
#### Good Cases
- one requestor + one responder
- 不用考慮scalability -> **Reliable Connection**(RC)
- PSN - Packet sequence number

#### Timeout
- The second packet was discarded

#### Out of Sequence

#### Receiver Not Ready

## Software interfaces

### User level networking

- Protocol offload: 簡化Kernel
- OS Bypass
- Zero copy
- Vitrual Interface Architecture (VIA)
- user-level zero-copy網路模型
- IB的基礎


## A Glance Into The Future

### Network Version of Moor's Law
- Increase (double) network BW every generation
- Reduce (half) network latency every generation
- Latency 很難再減少
:::danger
But we cannot keep reducing latency
Getting close to physics limitation
:::

### The Need for Intelligent and Faster Interconnect
- Reduce the latency 10 times

### In-network Computing - Persistent Communication
- Offload complex communication patterns
- Send to multiple destinations
- Receive from multiple sources
- Dependencies between the operations
- Define communication graph templates
- E.g. Tree based reduce algorithm
- Call template with parameters
- Source/ destionation buffers
- Buffer size

### Make Your Network Smarter

- SmartNICs becoming more dominant in the large datacenters
- Programable entities within the NIC
- Offload all kind of tasks
- Network security
- Network services(e.g. NFV)
- Hypervisor functionality
- Middleware functionality
:::info
- SmartNICs
為一個允許附加軟體的NIC，而這些附加軟體可以在購買後的某個時刻載入NIC，用於添加新功能或支援其他功能
>就像買了手機，然後從應用程式商店下載應用程式一樣
:::