---
title: Infiniband foundation
tags: APAC HPC-AI competition, training
---

# Infiniband Foundation training
[TOC]

## Why InfiniBAND?
Keys to success:
- Low latency
- Simplified Management
- High throughput

## Feature
- High BandWidth
- High Scalability
- easy scaling for data centers with great flexiblity
- enables a single subnet to scale of up to **48,000** managed nodes (nodes in a single subnet)
- Quality of Service(QoS)
- is the ability to provide different priority to different:
- applications
- Users
- Data Flows
- QoS implementation can be achieved by:
- Defining I/O channels at the adapter level
- Defining Virtual Lanes at the link level
- Qos allows control of congestion on the netwirk
- CPU offloads
- taking CPU's work and done by other element in infiniband environment.
- supports packet transportation with minimal CPU intervention
- RDMA support
- kernel bypass
- Hardware-based transport protocol
- Low Latency
- 1 microsecond(good number for latency :+1:)
- Simplified Management
- SM(Subnet Manager) -> a program that runs and manages the fabric
- Any IFB has its own single(master) SM
- the SM makes the fabric management simple
- plug & play end nodes environment
- centralized route manager

## Fabric components
- Switch
- SM
- Host/Server
- Channel Adapter
- Gateway/Bridge
- to get information or data from IB to Ethernet
- Router

infiniband link: any connection that speaks infiniband protocol.

## Infiniband Architecture and Management
### Infiniband Network Stack
- InfiniBand uses a multi-layer processing stack to transfer data between nodes
- CPU offloads functions
- Offers greater adaptability through a variety of services and protocols

router's job: to connect between defferent infiniband subnet.


### IB architecture layers
- Upper Layer Protocols(I/O Controller)
- Transport(Message(QP))
- Network(Inter Subnet Routing(GRH))
- Link(Link Encoding(LRH) -> Media Access Control)
- Physical Layer(End Node)

Open fabric enhances destribution (OFED)
### Each layer(Functions/Protocols/Devices):
- Provides services to the layer above
- Issues service requests to the layer below

### Data packet structure

> GRH: Global Routing Header
> LRH: Local Routing Header
> CRC: a police man to check if the data is correct (addtional data)

### Subnet Manager
- The Subnet Manager (SM) is responsible for
- Managing all the elements in the fabric
- Discovering the topology
- Assigning LIDs to devices
- Local Identifier(LID)
- Calculating and programming switch forwarding tables
- how many exits, how many rows
- Monitoring changes in subnet(all the time)
- instead of going dead way
- and to deal with traffic
- The SM can be implemented on any node in the fabric(or server)
- Only **one** master SM is allowed in a subnet

### The Switch IC
- IC (Integrated Circuit) – Switch ASIC
- EDR switches:
- Switches that contain up to 36 ports (as for today) have 1 IC
- Switches that contain more than 36 ports have several ICs
- HDR switches:
- Switches that contain up to 40 ports (as for today) have 1 IC
- In this Example – 72 ports switch, using 6 identical ASICs
> never say never
>
>switches would be stucked in such scenario, of trying to connect nodes within the same
>

### Fabric Addressing - GUIDs
- GUID (Globally Unique Identifier) is a 64-bit unique address that is burned onto the hardware by the vendor
- The GUID is persistent through reboots
- Three types of GUIDs:
- Node GUID
- Port GUID
- System Image GUID
- Switches that belong to the same chassis have the same System Image GUID

### Layer 2 addressing - LIDs
- LID (Local Identifier) is a 16-bit Layer 2 address
- LIDs are assigned by the Subnet Manager when a node becomes active
- HCAs are assigned with a LID per port
- Switches
- 1 IC switches are assigned with a single LID
- Director switches are assigned with a LID per switch module in the chassis

**IB command**: provides dozen of command, some of them provide information about link, nodes or whole network. very user friendly. (eg. ibstat)

> Mellanox is the maker of this hardware, each card is equiped with VPi, which could be used in both Infiniband network and Ethernet.

### Layer 3 Addressing - GID
- GID (Global Identifier) is a 128-bit field in the Global Routing Header (GRH) used to identify a single end port or a multicast group
- GIDs are globally unique (across multiple subnets)
- GID’s structure:
- Based on the a subnet prefix (a 64-bit identifier) combined with the Port GUID
- IPv6 type address
- Each HCA port is automatically assigned with a default GID that can be used only in the local subnet
> layer3 addressing include 2 part: Subnet prefix and GUID
> prefix: to work with multiple subnet.
>

---
### Fabric Segmentation - Partitions
- A partition describes a set of end nodes within the fabric that may communicate
- When is port is assigned to a partition its membership type can be set to:
- Full membership –> can communicate with any other node in the partition
- Limited membership -> can communicate only with nodes in full membership in the partition
- Nodes may be members of multiple partitions at once
- PKEY – is a field in the BTH header used to determine membership in a partition


---
## InfiniBand Architecture The Physical Layer
### Virtual Lanes
- Virtual Lanes provide a mechanism for creating multiple virtual links within a single physical link
- VLs are used to implement QoS:
- Different packets are mapped to different virtual lanes
- Each VL is configured with a weight and a priority
- Each VL uses different buffers
- IB specification allows 16 possible VLs
- VL15–usedformanagementtrafficonly
- VLs 0-14 – used for data traffic (VLs 0-7 are currently supported by Mellanox switches)

> - SMP: subnet manager packet
> - VL-15, why so many virtual lanes? well.. it's quite theroretical, that they can solve traffic problem.
> They may have different bandwidth and priority(virtual manager would process higher priority first)
> Why do we need priority guys? Cause different applications requires different latency.

---
## InfiniBand Architecture The Link Layer
### Switching
- The forwarding database of IB switches is called LFT (Linear Forwarding Table)
- LFTs contain DLID to Exit port mapping, so that incoming packets are sent on the correct port
- LFTs may contain SL to VL mapping which is used to implement QoS


> define multiple VL to give diff priority

### Credit Based Flow Control
> point => flow control
- **We need flow control when buffer is full** and so we descipline the sender.
- The credit based flow control is a mechanism that assures packets are not dropped even in the presence of congestion.
- How does it work?
- Link receivers grant to senders the receive buffer space credits
- Flow control credits are issued in 64-byte units
- Separate flow control per VL is provided so that congestion and latency on one VL does not impact traffic with guaranteed QoS on another VL even though they share the same physical link.

> credit: means how many data can sender sent to receiver.
> (1 credit = 1 unit of 64 bytes)


---
### SL to VL Example
- Weight value (0-255), indicates the number of 64-byte units, which may be transmitted from that VL when its turn in the arbitration occurs.
- If the High-Priority VLs don’t have an available packet for transmission a packet may be sent from the Low-Priority VLs.


## InfiniBand Architecture The Transport Layer
### Transport Layer Service
- The Transport Layer provides end-to-end communication services for applications
- An end-to-end “virtual channel”, connecting two applications that exist in entirely separate address spaces, should be created.
- The endpoints of the channel are called Queue Pairs (QPs), each QP represents one end of a channel.
- QPs constitute the structure that enables applications to:
- Bypass kernel during data transfer
- Exchange reliable data where hardware is in charge of reliability

> QP number, a 24-bit identifier, would be an identification number on layer4

### Offload Data Transfer Methods
There are two predominant methods for offload data transfers:
- Send / Receive operations
- RDMA Operations
- RDMA WRITE
- RDMA READ

#### Send / Receive Operation Example
> Complete queue: to know when the mission has completed and also reuse the buffer.

#### RDMA Write Example
> No need for the receive queue on receiving side! as well as the Completion queue.
> cause the responsiblity is now on a remote server.
>

using remote server's space like it is yours by giving a remote user a key, and let him use my memory. so save time, memory latency, making the whole proccess much more efficient.

## Vocabulary
- throughput (n.) 吞吐量，總處理能力
- topology (n.) 網路架構

## Questions
- def. of subnet?
- GUID(Globally Unique Identifier) & UID(Unique Identifier) & GID(Global Identifier)?
- Virtual Lanes?
-

## Common Icon

### [Join the session](https://mellanox-tech.webex.com/mw3300/mywebex/default.do?service=7&main_url=%2Ftc3300%2Ftrainingcenter%2Fdefault.do%3Fsiteurl%3Dmellanox-tech%26main_url%3D%252Ftc3300%252Fe.do%253FAT%253DMI%2526%2526Host%253DQUhTSwAAAARu2zXgRncJMfCHxH6jb87T_R-w1G-gJHDTEFGquQMMHLgg2OTmaaA_amynJ2oHyZq4FCqrBpO5HAFwWkmBGNxP0%2526UID%253D0%2526MTID%253Dt46aa698f61d84d067a0e6af042a6c53a%2526siteurl%253Dmellanox-tech%2526confID%253D164290731365631758%2526ticket%253D4832534b000000046d5955870927bfd1f2a114683d77cc1ab91bdd68ca106e34caf960268c933f9e&siteurl=mellanox-tech)

> Access code: 1289478916
>

                         ___====-_  _-====___
                  _--~~~#####// '  ` \\#####~~~--_
                -~##########//    王   \\##########~-_
               -############//  |\^^/|  \\############-
             _~############//   (O||O)   \\############~_
            ~#############((     \\//     ))#############~
           -###############\\    (oo)    //###############-
          -#################\\  / `' \  //#################-
         -###################\\/  ()  \//###################-
        _#/|##########/\######(  (())  )######/\##########|\#_
        |/ |#/\#/\#/\/  \#/\##|  \()/  |##/\#/  \/\#/\#/\#| \|
        `  |/  V  V  `   V  )||  |()|  ||(  V   '  V /\  \|  '
           `   `  `      `  / |  |()|  | \  '      '<||>  '
                           (  |  |()|  |  )\        /|/
                          __\ |__|()|__| /__\______/|/
                         (vvv(vvvv)(vvvv)vvv)______|/