Try   HackMD

Infiniband

本文部分內容出自 Mellanox 官方教學：https://academy.mellanox.com/en/

Wiki:
InfiniBand（直譯為「無限頻寬」技術，縮寫為IB）是一個用於高效能計算的電腦網路通信標準，它具有極高的吞吐量和極低的延遲，用於電腦與電腦之間的資料互連。InfiniBand也用作伺服器與儲存系統之間的直接或交換互連，以及儲存系統之間的互連。

大綱簡報

概略

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

IB 網路主要分為五層，而其封包格式如上圖。
其中紅色主要對應了 OSI 七層中的上三層。

物理層

物理層定義了電氣特性和機械特性，包括光纖和銅媒介的電纜和插座、底板連接器、熱交換特性等。定義了背板、電纜、光纜三種物理端口。

  • 背板

    • 主要用於機箱內的 IB 網傳輸

  • 電纜

    • Cable 中間走的是銅線，距離較短但方便且便宜
    • 單條一對一
      • Image Not Showing Possible Reasons
        • The image file may be corrupted
        • The server hosting the image is unavailable
        • The image path is incorrect
        • The image format is not supported
        Learn More →
    • DAC 一分二，可以平分頻寬
      • Image Not Showing Possible Reasons
        • The image file may be corrupted
        • The server hosting the image is unavailable
        • The image path is incorrect
        • The image format is not supported
        Learn More →

  • 光纜

    • Cable 間為光纖，距離長但貴
    • AOC
      • Image Not Showing Possible Reasons
        • The image file may be corrupted
        • The server hosting the image is unavailable
        • The image path is incorrect
        • The image format is not supported
        Learn More →
    • AOC = 兩個光轉換器＋光纖線
      • Image Not Showing Possible Reasons
        • The image file may be corrupted
        • The server hosting the image is unavailable
        • The image path is incorrect
        • The image format is not supported
        Learn More →
Layer Infiniband Ethernet
Dats Link GUID Mac Address
Data Link LID Like private IP
Network GID Like Net Mask

Infiniband 的規劃跟 IPv6 比較相近

L1

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

L2

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

L3

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →

應用 API

InfiniBand 定義了 L4 以下，但卻沒有標準的驅動程式。原始的 IB 標準只列出一組的動作例如 ibv_open_device 或是 ibv_post_send，這些都是必須存在的子程式或方法的抽象表示方式。這些子程式的語法由供應商自行定義。事實標準的軟體堆疊標準是由 OpenFabrics Alliance 所開發的。它以雙許可證方式發布，GNU通用公共許可證或BSD許可證用於 GNU/Linux 以及 FreeBSD，且 WinOF 在 Windows 下可以選擇 BSD許可證。它已被大多數 InfiniBand 供應商採用，用於 GNU/Linux、FreeBSD 以及 Windows。

OpenFabrics Enterprise Distribution (OFED)是一組開源軟件驅動、核心內核代碼、中間件和支持InfiniBand Fabric的用戶級接口程序。
我們可以想像在上面的定義後，OFED 進一步的撰寫了更多的 IB 功能，舉凡 IPoIB（在 IB 網路上實現乙太網的 IP 機制）、RDMA 等等。

OFED 除了由 OpenFabrics 官方釋出的開源版本外 https://www.openfabrics.org/downloads/OFED/
也有由 Mellanox 再次撰寫並打包的 MLNX_OFED https://www.mellanox.com/support/mlnx-ofed-matrix?mtag=linux_sw_drivers

不論何者，都能夠妥善利用 opensmd 管理 IB 網路，並提供進階功能 IPoIB、RDMA

OFED

參考這篇
https://hackmd.io/@William-Mou/SywOmi7kD

InfiniBand 定義了傳出規範，但卻沒有標準的驅動程式。原始的 IB 標準 (verbs) 只列出一組的動作例如 ibv_open_device 或是 ibv_post_send。子程式的語法由供應商自行定義。常見標準是由 OpenFabrics Alliance 所開發。它以雙許可證方式發布，GNU 通用公共許可證或BSD許可證用於 GNU/Linux 以及 FreeBSD，且 WinOF 在 Windows 下可以選擇 BSD 許可證。它已被大多數 InfiniBand 供應商採用，用於 GNU/Linux、FreeBSD 以及 Windows。

OFED 用途

OpenFabrics Enterprise Distribution (OFED) 是一組開源軟件驅動、核心內核代碼、中間件和支持InfiniBand Fabric的用戶級接口程序。
我們可以想像在上面的定義後，OFED 進一步的撰寫了更多的 IB 功能，舉凡 IPoIB（在 IB 網路上實現乙太網的 IP 機制）、RDMA 等等。

OFED 除了由 OpenFabrics 官方釋出的開源版本外 https://www.openfabrics.org/downloads/OFED/
也有由 Mellanox 再次撰寫並打包的 MLNX_OFED https://www.mellanox.com/support/mlnx-ofed-matrix?mtag=linux_sw_drivers

不論何者，都能夠妥善利用 opensmd 管理 IB 網路，並提供進階功能 IPoIB、RDMA

Last changed by 
 
牟展祐
0
3394

Read more

CSST 程式設計討論會

Computer Science STudents :::info 本筆記位置：csst.tw :::

Mar 1, 2025
CSST 程式設計討論會

About CSST

Dec 6, 2024
Spack 指令教學

Spack 是套件管理器，但不同於 apt 懶人打包好，Spack 全部都 Compile from Source，且允許同軟體多版本並存，有 Code 在，問題可控、性能可調，且能享受大家貢獻的 Install Script，唯一缺點就是安裝可能慢了點。

Apr 3, 2024
MPI

MPI 介紹 MPI 標準是由 MPI 論壇（由來自行業和學術界的並行計算專家組成的一個開放小組）創建和維護的。MPI 定義了一種 API ，該 API 用於高性能 Process 間通信（IPC） MPI 定義了一個高級 API，這意味著它抽象了實際上用於在進程之間傳遞消息的任何底層傳輸。這個想法是，發送者 Process X 可以有效地說「將此陣列的 1,073 個雙精度值發送給 Process Y」。相應的接收者 Process Y 簡單的「從 Process X 接收一個 1,073 個雙精度值的陣列」。輕而易舉的，由 1,073 個雙精度值組成的數組到達 Y 的等待緩衝區。 MPI 標準的許多實現都支持多種平台，操作系統和網絡類型。有些實現是開源的，有些則是閉源的。顧名思義，Open MPI 是開源實現之一。典型的 MPI 傳輸網絡包括（但不限於）：以太網上的各種協議（例如 TCP、iWARP、UDP、ETH Frames），或者 share memory 和 InfiniBand MCA 架構 由於 MPI 涵蓋不同層次的技術（從最上層抽象的並行計算、POSIX、網路通訊等）所以需要定義良好的架構，才能夠妥善地實現以上功能，並且為了因應 HPC Application 差異極大的特性，需要提供可算法的更換性。

Mar 27, 2023
Read more from 牟展祐
Published on HackMD