---
title: 6/24 HPC and AI foundation
tags: APAC HPC-AI competition, training
---
[TOC]

[Course Video](https://zoom.us/rec/share/3uxbKq3Wz2hLQ4XntgaYSKt8BYe7eaa81icX-aUExXudAiWg7RHSZu5n99Zzn8A)
>password:2020@HPCAI

---
# Accelerating AI and HPC with RDMA

## Moore's Law -> amazing prediction :+1:
- The number of transistors on integrated circuit chips will double every two years
- but we don't count the exact figures
:::warning
the growing can't continue forever...
:::

CPU: higher frequency -> faster
Break through: innovation in way CPU works
but there are limited ways to innovate
Gap beetween CPU really is with our expectation.
GPU small versus engines, instead of upgrade every (),

(very flexible)
1. CPU -> flexible engine, but have limitation
2. GPU -> limit to specific type of operation (for parallel calculation)
3. FPGA/DPU -> hard performance (specifically for Deep learning)
4. ASIC -> Application Specific Integrated Circuit (customize)(can be ignored)
(very efficient)

**network** connect everything above, super important!!!

---
## Data growth exponential
Data ~~Wave~~ Tsunami: 90% of the world's data has been created only in the last 2 years

Data Driven Revenues in top 10 companies:
Amazon, Google, Facebook, Tencent, Alibaba...
data is new oil and gas??

### Example
Autonomous vehicle generates 4000GByte per day.

recognize items
AlexNet: 3 - 4% -> a neural network with (_)
in 4 years, the power grow 350 times faster
CPU provider?

When we have a complicated problem -> solve it with Neural network

To have all the above 4 engines in a single system, we need 3 flexible and protable networks.

---

## Pizza Trend

==More Data + Better Models + Faster Interconnect(GPUs, CPUs, ASIC, FPGAs...etc)==

:::danger
higher the frequency != faster CPU
:::

Faster data speed & **In-Network Computing** enable higher performance and scale.

CPU-Centric(Onload)
- Must wait for the data
- Create performance bottleneck

Data-Centric(Offload)(In-Network Computing)(offload 分流)
- Analyze data as it moves!!
- Like it
- Higher Performance and Scale
solve problem faster so you can solve more problems

fast is fhe most important thing.

### Example (CPU-Centric)
PE 1: Generate pizza
PE 2:
- order pizza
- PE 1 - prepare pizza
- pe 1 - put in the oven
- pe 1 - Pack and send
- network (Pizza delivary)
- PE 2 - pizza comsumption

---
## The need of speed

- Matching inter and intra node bandwidth
- communication within a system

### Q&A

Someone asked: Is the data centric processor also something like cpu?
>data centric processor can be cpu or gpu depending on what we want to do to the data(application choise)
視需求而定

---
## RDMA(efficient)
Unbeatable performance

Main features
- remote memory Read/Write semantics in addition to Send/Recv
- Kernel bypass / direct user space access
- Kernel會做整流和分流
- 可以直接讀取GPU的Memory
- Full hardware offload for network stack
- Secure, channel based IO

Application Advantage
- Lowest latency
- Highest bandwidth
- Lowest CPU consumption
- Direct memory access, no unnecessary data copies

RoCE: RDMA over Converged Ethernet
- Available for all Ethernet speeds 10 - 100G
- a special protocol (擁有RDMAㄉ)

TCP/IP

    App                           App
     |                             |
    socket                      socket
     |                            |
    trans protocol             trans protocol
     |                           |
    NIC dirver                NIC dirver
         |                      |
         NIC   -------------  NIC

RDMA

    App                       APP
     |                         |
    RNIC    -----RoCE------   RNIC

RNIC better than NIC
Tensorflow 真棒 :+1:

==越多GPU，RDMA和TCP的效能差越大==

10X Higher Performance with **GPUDirect^TM^ RDMA**
- Accelerates HPC and Deep Learning perfomance
- Lowest communication latency for GPUs.

GPU(hold GPU memory) & CPU(hold system memory)
(HCA) Host Controller / Adapter
Network
(HCA) Host Controller / Adapter
GPU(hold GPU memory) & CPU(hold system memory)

---
## In-Network Aggregation & Reduction Protocol

SHARP: Scalable Hierarchical Aggregation & Reduction Protocol
- reliable scalable general prupose primitive
- applicable to multiple use-cases in ML/HPC
- scalabe high performance collective offload

                  Switch
     aggregated  /       \
   data(up)     /         \
 result(down)  /           \
         Switch             Switch
        /  |   \
       /   |    \
    Host  Host  Host                   Same right


SHARP
- enables 75% reduction in latency, providing scalable flat latency.(AllReduce)
one by one(?
- It will reduce the amount of data running through the network
- It will reduce the latency because datawill go through a shorter path
- The operaton will be fully offloaded.


#### (Pay attention):angry:

**Infiniband**
- increase system performance
- Better scalability
- reduces amount of data traversing the network
[About Infiniband](https://www.gigalight.com/zh-TW/show-1118.html)

### Q&A
Garenth asked: Is RDMA available to link any type of GPU model or only to a specific RDMA-enabled type of GPU model?
> software problem, not hardware

---
## Distributed AI
- Model Parallelism
- like how our brain works
- use back propagation(do a huge amount of that)
- get many servers and large GPUs to accomplish this model
- each cell only process a portion of the neural network
- if you wanna connect with the upper layer, you have to go through server
- model size is limited by the compute engine(GPU for example)

- Data Parallelism
- (mini batch) local model -> updates main model/parameter servers/Allreaduce(mini batch 算是Deep learning的一種方法，將龐大的資料庫分成若干小份去做學習，可有效降低時間)
- each server train the entire neural network then update with the root network, and iterates the training process again and again(note: each server get the same model in each iteration).
- efficient networking is a key to enable data parallelism
- back propagation (Deep learning 的演算法，利用不斷修改神經網絡中間的輸出層(隱含層)的值來達到預期的最終輸出結果)
(此PPT中左邊的圖為神經網絡的基本構成圖，藍色為輸入層，橘色為輸出層)

**What does it looks like to us users?**
- Most of the time (AI) application is doing computation
- Which is what actually interests the users
- Overlap communication and computation
- Start non-blocking send/recv
- Compute the data for next phase
- Wait for completion of all requests
- Typical apllication flow
- Init:
- Read input file
- decompose the problem
- Work Loop:
- Compute
- Data exchange(okay, please calculate these gradients)
- Finalize:
- Write final file

The secret sauce of RDMA :yum:
- Low latency (<1 usec)(u =10E-6)
- High Bandwidth( > 200Gb/s)
- Scalability to very large number of compute engines
- OS bypass(direct access to the hardware from the user level)
- Remote Direct Memory Access(avoid memory copies in communication on stack)
- Offloads
- Collective operations, support for non-contiguous data, GPU-Direct, Peer-Direct, tag-matching, etc
- Low software overheads
- Low memory footpint (AMAP)
- Performance portable APIs

P2P communication
- Direct transfer between PCI-e devices


RDMA 就是比人家快
PeerDirect

---
### Code Example

---
**GPUDirect^TM^ RDMA/PeerDirect**
- CPU synchronizes between GPU tasks and data transfer
- HCA directly accesses GPU memory
- Direct path for data exchange
- when the GPU knows the data, all it needs to do is simply send it out.
- Eliminate the need to make a redundant copy in host memory.

**GPUDirect^TM^ Async vs. PeerDirect^TM^ Async**
==GPUDirect==
- Control the HCA from the GPU
- Enable batchingof multiple GPU and communication tasks
- Reduce latency
- Reduce CPU utitlization
- Light weighted CPU
- Less power

==PeerDirect==
- CPU prepares and queues compute
- communication tasks on GPU
- GPU triggers communication on HCA
- HCA directly accesses GPU memory

---

Point to Point message protocols(MPI)
- Eager
- sender sends the tag + data to receiver
- Send() is completed immediately*
- unless sneder is using zero-copy
- Receiver must store the data until it's matched
- May exhaust memory on the receiver
- Typically used for small-medium messages
- Usually means no GPUDirect RDMA

- Rendevous
- sender sends only the tag to the receiver
- Send() is not completed
- Receiver requests the data from the sender when it matches the tag
- Data can be stored directly to the receiver buffer, since it's already matched
- use RDMA for zero-copy receive
- Typically used for large messages

---
### Common Pitfall (to be studied)
- Rendevous is not Zero-copy, they are different.
- The relation between them is that zero-copy on receiver side can be done only with rendevous(or hardware assisted tag matching).
- Scale
- Which transport to use?
- Maybe hold set of multiple transport resources per use case?
- Zero copy might be slower
- Zero copy improves CPU usage, not bandwidth
- Whether zero copy is useful or not depends on the application.
- GPU Direct RDMA edge cases
- Not in all cases zero copy is better
- Depends on the topology of the peers(GPU & HCA), can also depends on the message size.
- Memory
- All buffers muse be registered, how many buffers to use
- Bounce buffers may be needed

Always think, should we use TCP or UDP or others.
> OK

### UCX - Unified Communication X
> an open-source, production-grade communication framework for data-centric and high-performance applications.

- gives us simple and consistent API

- UCX stack -> an interesting stack!

> Not the best solution? then go to another transport.
>
---
### Collective Communication

- one process communicate with multiple processes -> This is called **collective communication**

- RDMA only support a msg passing from one process to another(==point-to-point==)
- In many usercases, we want to let one process communicate with multiple(a group of) processes.
- so called collective communication
- Collective communication can be implemented on to of point-to-point communication.
- Using collective communication semantics make the code simpler and potentially faster.
- There are numerous algorithms to perform every collective operations(active academic research)
- Optimized for different scales / message sizes / networks.
- Examples: Recursive doubling, Pairwise exchange, Fain-in, fan-out, Reuced-scatter/Allgather, Linear, Bruck, etc...
- UCC implements hierarchical collectives, based on point-to-point, multicast, core-direct and other offloads

==we could write our own algorithms==

---

### Unified Collective Communication (UCC)(brother of UCX)

use it wise, so everyone good benefit from it.

How to we ensure security? that enable our peers to access some region of our memory and only aloow others to access the restricted area of our memory?
Well, I forgot..

### Q&A
Q: I have a simple question. Let say I have two separate computing engines with gpus. To enable the RDMA between them, I shall have the Mellanox SHARP switches to do the scalability leverage. Am it right? Am I allowed to connect two computing engines together using the direct connection?
>if I have multiple engines, we could collect them with switches. [name=Gil Bloch]
>
Q:How does MapReduce compare with SHARP?

---

Q1: Kernal VS System memory

#### (to be studied)
To check: what is
1. MapReduce
2. SHARP