PP hw4 Ans - HackMD

# PP HW 4 Report Answer 50% [TOC] > - Please include both brief and detailed answers. > - The report should be based on the UCX code. > - Describe the code using the 'permalink' from [GitHub repository](https://github.com/NTHU-LSALAB/UCX-lsalab). ## 1. Overview 10% > In conjunction with the UCP architecture mentioned in the lecture, please read [ucp_hello_world.c](https://github.com/NTHU-LSALAB/UCX-lsalab/blob/ce5c5ee4b70a88ce7c15d2fe8acff2131a44aa4a/examples/ucp_hello_world.c) 1. Identify how UCP Objects (`ucp_context`, `ucp_worker`, `ucp_ep`) interact through the API, including at least the following functions: ==Total 3%== :::warning 在程式碼中找到以下三項，附上連結，並且大致描述功能就能給分，一項一分。 - [ ] `ucp_init` ==1%== - [ ] `ucp_worker_create` ==1%== - [ ] `ucp_ep_create` ==1%== ::: 2. What is the specific significance of the division of UCP Objects in the program? What important information do they carry? ==Total 6%== :::warning 要明確描述「為什麼要切分成三個層級？」，以 `ucp_worker` 為例子： ❌ worker 負責管理 ep（為什麼 UCX 不直接寫成 context 管理 ep 就好？） ⭕ 可以為 rank 的每個 thread 分配一個專用的 worker，是溝通時的基本單位。 - [ ] `ucp_context`：提及用來隔離不同應用程式，例如 MPI 和 OpenSHMEM 底層都用 UCX 通訊，所以各創立一個 context。主要用於記錄不同的設定。 ==1%== - [ ] `ucp_worker`：worker 只會屬於一個 ucp_context，可以為應用程式(ucp_context) 的每個 thread 分配一個專用的 worker。是溝通的基本個體。 ==1%== - [ ] `ucp_ep`:endpoint 只會屬於一個 worker，每個 woker 可能會與不同 worker 溝通（thread 與 thread 溝通），所以會建立很多個 ep。其中也有可能有一組 ep 同時屬於同一個 woker，代表 woker 內的互聯（thread 內的 community）。用來定義溝通的「路」。 ==1%== - [ ] 繪圖評分: - [ ] 僅畫出三個元件的關聯箭頭 ==0.5%== - [ ] 有標記出 process/thread ==0.5%== - [ ] 1 worker 有 2 組 EP 或類似概念 ==1%== - [ ] 有標記 rails 和 TLS 在 EP 上 ==1%== > 繪製出下圖該項滿分 ![UCX_UCP.drawio](https://hackmd.io/_uploads/HyARoSP_6.png) ::: 3. Based on the description in HW4, where do you think the following information is loaded/created? ==Total 1%== :::warning 分別描述兩項的可能位置，描述合理就給分。舉例來說： - `UCX_TLS`:可能在 `ucp_context` 內紀錄，因為它包含了設定檔。 - TLS selected by UCX: 可能在 `ucp_ep` 被選中，因為他真正決定溝通的 `rail` ::: ## 2. Implementation 22% > Describe how you implemented the two special features of HW4. 1. Which files did you modify, and where did you choose to print Line 1 and Line 2? ==Total 5%== :::warning 看他 code 解釋是否合理， - [ ] Line 1: 有寫出來解釋合理。 ==2%== - [ ] Line 1: 不是直接用環境變數抓到 `UCX_TLS=all`。 ==1%== - [ ] Line 2: 有用 printf 或類似的 function 在 `ucp_worker.c` 就給分。 ==2%== ::: 2. How do the functions in these files call each other? Why is it designed this way? ==Total 1%== :::warning - [ ] 不要胡說八道，跟 code 吻合。 ==1%== ::: <style> @media print { .page-break { page-break-after: always; } } </style> <div class="page-break"></div> 3. Observe when Line 1 and 2 are printed during the call of which UCP API? ==Total 2%== :::warning - [ ] Line 1: `ucp_ep_create` ==1%== - [ ] Line 2: `ucp_ep_create`，除非他 code 寫得很特別。 ==1%== 這題要回答的是「UCP API」，所以只回答 `ucp_worker_get_ep_config()` 或 `ucp_worker_print_used_tls()` 都不給分。 ::: 4. Does it match your expectations for questions **1-3**? Why? ==Total 2%== :::warning 承上題，程式碼在 ucp_worker 內，但最後都是在創建 ep 時被 call。所以若能提及以下，才算給分。 - [ ] `UCX_TLS`:雖然在 `ucp_context` 設定，但因應作業要求，其實要到 ep 時才能印出來。 ==1%== - [ ] TLS selected by UCX: 在 `ucp_ep` 被選中，因為他真正決定溝通的 `rail`，但是為了避免印太多次，程式碼實作在 `ucp_woker.c` 中。 ==1%== 以下是一個滿分的答案 ref: 109062332 陳錫宏 >有點不太一樣，原本預期實際上用到的 tls 會在 ucp_ep 被創建時決定，雖然確實是如此，但實際上在印這兩行時用到的資訊都不在 ucp_ep 裡面，一方面 UCX_TLS 在最開始 context 被創建時已經被解析並創建好對應的資源而必須重新存取環境變數才能確定，至於最後使用的 tls 則因為避免太多重複的 ep_config 而實際上是在 ucp_worker 中才確定，所以兩者都沒辦法在 ucp_ep.c 當中直接找到。 ::: 5. In implementing the features, we see variables like lanes, tl_rsc, tl_name, tl_device, bitmap, iface, etc., used to store different Layer's protocol information. Please explain what information each of them stores. ==Total 12%== :::warning 根據 1.2 的圖思考學生答案的正確性，一項一分。 - [ ] lanes: endpoint 所走的 rails，預設兩種 tl_rsc。 ==2%== - [ ] tl_rsc: Array of communication resources (ex: rc_verbs/mlx4_0:1) ==2%== - [ ] tl_name: resources name ==2%== - [ ] tl_device: resources device ==2%== - [ ] bitmap: tl_rsc 有被 workers 挑來用的放這裡面。 ==2%== - [ ] iface: ep 選到的 lanse 會交給 uct 真正負責溝通，uct 與底層交互的 struct 是 iface ==2%== ::: <style> @media print { .page-break { page-break-after: always; } } </style> <div class="page-break"></div> ## 3. Optimize System ==18%== 1. Below are the current configurations for OpenMPI and UCX in the system. Based on your learning, what methods can you use to optimize single-node performance by setting UCX environment variables? ``` ------------------------------------------------------------------- /opt/modulefiles/openmpi/4.1.5: module-whatis {Sets up environment for OpenMPI located in /opt/openmpi} conflict mpi module load ucx setenv OPENMPI_HOME /opt/openmpi prepend-path PATH /opt/openmpi/bin prepend-path LD_LIBRARY_PATH /opt/openmpi/lib prepend-path CPATH /opt/openmpi/include setenv UCX_TLS ud_verbs setenv UCX_NET_DEVICES ibp3s0:1 ------------------------------------------------------------------- ``` > Please use the following commands to test different data sizes for latency and bandwidth, to verify your ideas: ```bash module load openmpi/4.1.5 mpiucx -n 2 $HOME/UCX-lsalab/test/mpi/osu/pt2pt/standard/osu_latency mpiucx -n 2 $HOME/UCX-lsalab/test/mpi/osu/pt2pt/standard/osu_bw ``` :::warning 這題目基本上加上 `UCX_TLS=all` or `UCX_TLS=sm` 就能獲得很快的性能提升。要確認同學不是抄來的，看一下他對於「為什麼要這樣加？」的解釋是否與前面的理解有相關性。 - 正確性 - [ ] 有做(回答不是 `all` or `sm` 的時候就只勾選這個) ==2%== - [ ] 至少測試 all 或只測試任意一個（有勾選這個一定會勾選到前一個） ==2%== - [ ] 測試多個 sm, mm, verb 分數 ==5%== - 呈現與說明 - [ ] 對於實驗方式說明合理 ==3%== - [ ] 對於最高分的說明合理 ==3%== - [ ] 以表格/圖表清楚呈現 ==3%== ::: <style> @media print { .page-break { page-break-after: always; } } </style> <div class="page-break"></div> ### Advanced Challenge: Multi-Node Testing ==+5%== This challenge involves testing the performance across multiple nodes. You can accomplish this by utilizing the sbatch script provided below. The task includes creating tables and providing explanations based on your findings. Notably, Writing a comprehensive report on this exercise can earn you up to 5 additional points. - For information on sbatch, refer to the documentation at [Slurm's sbatch page](https://slurm.schedmd.com/sbatch.html). - To conduct multi-node testing, use the following command: ``` cd ~/UCX-lsalab/test/ sbatch run.batch ``` :::warning 標準答案應該是 `UCX_TLS=ib` 或 `UCX_TLS=all` 最快，如果有跟 `UCX_TLS=ud_verbs` (baseline) 比較並說明合理，基本可以給滿分。 - 正確性 - [ ] 有在多 nodes 上做出來 ==1%== - [ ] 有跟 `UCX_TLS=ud_verbs` (baseline) 比較 ==2%== - 呈現與說明 - [ ] 對於「實驗方式」或「最高分」說明合理 ==2%== ::: ## 4. Experience & Conclusion +2% 1. What have you learned from this homework? 2. Feedback (optional) :::warning 就看誠意給分 >.< 有什麼特別的意見可以 TG 丟給 @WilliamMou 感謝 Orz :::

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.