Parabricks License
===
###### tags: `Parabricks-v3.8`
###### tags: `基因體`, `NVIDIA`, `Clara`, `Parabricks`, `License`
<br>
[TOC]
<br>
## 摘要
- ### 地端授權 (GPU Based License)
- **單機版 (Node Locked) (節點鎖定版)**
- 綁定 GPU UUID
- 只能在綁定的 GPU 上使用
- **多機版 (Flexera based) (浮動授權版, floating license)**
- 不綁定 GPU (相同 GPU 數量的單機版和多機版，多機版通常較貴)
- 一次可以使用一定數量的 GPU
- [使用說明](https://docs.nvidia.com/clara/parabricks/3.7.0/GettingStarted/FlexeraDebian.html#setting-up-a-flexera-license-server-on-ubuntu-quick-start-guide) ：
1. 登陸 Nvidia license portal
> https://ui.licensing.nvidia.com/login
> - 一般使用者帳號無法登入
> - 需購買 license 後，才能取得企業帳號
2. 線上下載 Flexera software (`.zip`)
3. 在本地端設置 Flexera license server (`setup.bin`)
> - Flexera license server 就是 Apache Tomcat Web Server
> - Flexera license server 無試用版本
- **Web 首頁** http://localhost:8080
- **License 管理介面** http://localhost:8080/licserver
- **Client 端要存取的端點** http://localhost:7070
4. 線上製作 `license.bin`
1. 在 NVIDIA License Portal 建立 license server 組態
2. 將購買的授權(entitlements)，添加到 license server 組態
3. 點選 [DOWNLOAD LICENSE FILE]
> 後端將這些組態，製成 `license.bin`
5. 在本地端的 server，將 license.bin 上傳到 Flexera server
6. 在本地端的 client，將 flexera-server IP & Port 填到 config.txt
- ### 雲端授權 (Per GPU Hour License)
- 按單 GPU 每小時計費
- 例如 Google, [Amazon](https://aws.amazon.com/marketplace/pp/prodview-apbngojlskcyq#pdp-pricing), Microsoft 部署的 Parabricks
- ### 試用授權 (Trial License) (Evaluation License, Eval License, 評估授權)
- 免費使用 3 個月(目前限制是 90 天)，可用於地端或雲端
- ### 研究授權 (Research License)
- 優惠方案(有較高的折扣)，可用於地端或雲端
- 僅限於學術，不得作為商業用途
<br>
<hr>
<br>
## ==各文件的介紹==
> - license server documentation
> https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html
### [[2022 GTC] 你可以在任何地方運行 Parabricks](/TU5IJMcfTD6f_KS9YVH53g#page16：你可以在任何地方運行-Parabricks)
[](https://i.imgur.com/MpVmYQF.jpg)
- **You can run parabricks anywhere:**
你可以在任何地方運行 Parabricks:
- **on-premises** 地端
- **Annual license software** (年度授權軟體)
- **Node locked** (節點鎖定版，單機版，綁訂機器)
- **Floating license** (浮動授權版，多機版，不綁訂機器)
- **Pay per GPU** (按 GPU 付費)
- **EDU Discount** (有教育折扣)
<br>
:::info
:bulb: **[[類似參考] Node Locked 跟 Floating 授權有何不同？](http://ahasoft.blogspot.com/2010/01/x-win32-node-locked-floating.html)**
:::
<br>
- **cloud** 雲端
- **Can run on Google, Amazon, Microsoft**
可以在 Google, Amazon, Microsoft 雲端上執行
- **Working with channels: DNANexus, SevenBridges**
使用渠道：DNANexus、SevenBridges
- **Pay per hour**
按小時付費
- **Some customers are only using the cloud**
一些客戶只使用雲端
- **Some are just using the cloud for peak computation**
另有些客戶只針對高計算量的部份使用雲端
<br>
:::info
:bulb: **SevenBridges**
- [雲計算基因數據分析平台:seven bridges](https://ppfocus.com/0/eddd24ebe.html)
- [生物數據分析公司Seven Bridges入駐Google雲平台](https://kknews.cc/tech/g42pze.html)
- [火視全球|吸金4500萬美元——Seven Bridges Genomics將雲端與基因組學完美結合](https://kknews.cc/tech/38j849o.html)
:::
<br>
### 終端使用者授權合約(EULA) / 授權類型 (License Types)
> END USER LICENSE AGREEMENT FOR PARABRICKS SOFTWARE
> https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/healthcare-parabricks-end-user-license-agreement.pdf
- ### GPU Based License
**“GPU Based License” means a license for on-premise deployment with a fixed duration and licensed based on the maximum number of GPUs authorized per system. As examples, licenses for systems with up to two GPUs and licenses for systems with up to four GPUs.**
“基於 GPU 的許可證”是指具有固定期限的地端部署許可證，並根據每個系統授權的最大 GPU 數量進行許可。 例如，最多具有兩個 GPU 的系統的許可證和最多具有四個 GPU 的系統的許可證。
- ### Per GPU Hour License
**“Per GPU Hour License” means a license for cloud deployment with a fixed duration and licensed based on use per GPU in hourly increments. Hourly licenses are sold in blocks of hours, and any portion of a block not consumed within the authorized period are forfeited as NVIDIA will not provide refunds or credits.**
“按 GPU 小時許可證”是指具有固定期限的雲端部署許可證，並根據每個 GPU 的使用情況以每小時為增量進行許可。 每小時許可證以小時為單位出售，未在授權期限內消耗的任何部分都將被沒收，因為 NVIDIA 不會提供退款或積分(點數)。
- ### Trial License (Evaluation License) 適用授權(評估授權)
**“Trial License” means a license for on-premise or cloud deployment with a fixed duration and licensed for evaluation purpose only.**
“試用許可證”是指用於地端部署或雲端部署的具有固定期限的許可證，僅用於評估目的。
- ### Research License
**“Research License” means a license for on-premise or cloud deployment with a fixed duration and licensed for academic or other non-commercial purposes only.**
“研究許可”是指具有固定期限的地端部署或雲端部署許可，僅用於學術或其他非商業目的。
- **Licenses that are indicated to be subscriptions are inclusive of services, and other licenses are exclusive of services unless otherwise expressly stated in your order.**
除非您的訂單中另有明確說明
<br>
### License Types / GPU Based License
> https://docs.nvidia.com/clara/parabricks/3.7.0/GettingStarted.html#licensing
- ### Node Locked 節點鎖定
**licenses are tied to a specific set of GPUs on a server.**
許可證綁定到伺服器上的一組特定 GPU
> https://forums.developer.nvidia.com/t/planning-to-set-up-the-parabricks-environment/210543
> **Node-lock** - which works similar to the eval license. You provide the **GPU UUIDs** to us and we create a specific license for those GPUs. If you have 8 GPUs, the license would be built exactly to those 8 GPUs and cannot be used anywhere else.
- ### Flexera based 基於 Flexera
**licenses allow for a set amount of GPUs to be used at once using a license server. This will use the Nvidia License Server. You can read more about it here (optional).**
> **https://forums.developer.nvidia.com/t/planning-to-set-up-the-parabricks-environment/210543**
> **Flexera License Server** - This offers more flexibility as the licenses are not directly tied to GPUs. If you buy 8 licenses, you add them to the Flexera LS, and they can be ‘borrowed’ by Parabricks and then returned after your job finishes. So you can share those 8 licenses amongst multiple people using Parabricks.
<br>
<hr>
<br>
## ==License Types / GPU Based License / Flexera License==
> - Flexera License, Debian Package
> https://docs.nvidia.com/clara/parabricks/3.7.0/GettingStarted/FlexeraDebian.html
### Flexera license server
- 就是 web server (Tomcat8, port: 8080)
- Server host ID 必須唯一，可填主機的 MAC address
### 安裝方式
1. **前往 Nvidia license portal 並登錄**
- https://ui.licensing.nvidia.com/login (無帳號可試用)
- 一般使用者帳號無法登入
- 需 **購買 license** 後，才能取得**企業帳號**
2. **線上下載 Flexera software (`NVIDIA-ls-linux-xxxx.xx.x.xxxxxxxx.zip`)**
- 從 NVIDIA License Portal / [software downloads] 頁籤
下載 `NVIDIA-ls-linux-2022.02.0.30886060.zip` (範例)
[](https://i.imgur.com/OVJzw6a.png)
3. **在本地端設置 Flexera software (Flexera License Server, Flexera LS)**
> Flexera License Server 就是 Apache Tomcat Web Server (port 為 8080)
> - **Web 首頁** http://localhost:8080
> - **License 管理介面** http://localhost:8080/licserver
> - **Client 端要存取的端點** http://localhost:7070
1. **安裝 Java JRE** [[doc]](https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html#installing-jre-linux)
- Apache Tomcat Web Server 的網頁是使用 Java 語言撰寫
- 為了能執行 license 管理頁面
2. **安裝 Tomcat8 (Apache Tomcat Web Server)** [[doc]](https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html#installing-running-apache-tomcat-linux)
3. **設定 Flexera software**
- 解壓縮 `NVIDIA-ls-linux-2022.02.0.30886060.zip`
- 執行 `setup.bin`
- [GUI 介面](https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html#installing-nvidia-grid-license-server-software-linux) (不適合 OneAI 安裝，因為沒有 GUI )
`$ sudo ./setup.bin`
- [console 介面](https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html#installing-nvidia-grid-license-server-software-linux-console-mode)
`$ sudo ./setup.bin -i console`
4. **測試 Tomcat**
- 測試首頁
http://localhost:8080
- 測試 license 管理介面 [[doc]](https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html#accessing-license-server-management-interface)
http://localhost:8080/licserver
[](https://i.imgur.com/9ReDizM.png)
[[doc] 如何啟用帳密/或關閉](https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html#securing-license-server)
---
進入管理頁面後
[](https://i.imgur.com/KEfxtqj.png)
- :warning: **注意事項**
- hostname 可以是完整網域名稱，也可以是 IP
- [注意 DHCP，需配置相同 IP](https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html#platform-configuration-requirements)
像這樣，IP 會一直變
- 需留意是否有防火牆擋住 7070
- ==port 8080== 給 admin 管理，**不開放外部存取**
- ==port 7070== 給遠端 client 來存取 server 上的 license
<br>
5. **獲取 server host ID**
> 其實就是 MAC address
> 作為 Flexera LS 的識別碼，隨後會在線上填入資料，製作 `license.bin`
>
[](https://i.imgur.com/L4Fq7xs.png)
<br>
4. **線上製作 `license.bin`**
1. **在 NVIDIA License Portal 建立 license server 組態**
[](https://i.imgur.com/QZyEGQq.png)
<br>
2. **將購買的授權(entitlements)，添加到 license server 組態**
<br>
3. **點選 [DOWNLOAD LICENSE FILE]**
> 後端將這些組態，製成 `license.bin`
>
<br>
4. **之後，如果要更新 license server 組態，請參考：**
- [[修改] 4.4. Adding or Removing Licenses on a License Server](https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html#adding-removing-licenses-on-license-server)
- [[新增] 4.5. Adding Licensed Products to a License Server](https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html#removing-licensed-products-from-license-server)
- [[刪除] 4.6. Removing Licensed Products from a License Server](https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html#removing-licensed-products-from-license-server)
<br>
5. **在本地端的 server，將 license.bin 上傳到 Flexera server**
- #### ==僅能==透過 web UI 上傳 license.bin [[doc]](https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html#installing-a-license)
[](https://i.imgur.com/Lo4BKix.png)
- #### ~~將 license.bin 複製到 Flexera server~~ (更正：==僅能透過 Web UI 上傳==)
```bash
# Copy the license file
# generated from the flexera server setup process
# to installation folder.
$ sudo cp license.bin /usr/local/parabricks/license.bin
```
- #### 查看 host IP 方式，供 client 端使用
```bash
$ hostname -I
192.169.35.141
```
or
```bash
$ apt install -y iproute2
$ ip addr show | grep inet
inet 127.0.0.1/8 scope host lo
inet 192.169.35.141/32 brd 192.169.35.141 scope global eth0
```
- #### 運作機制 [[doc]](https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html#overview-of-nvidia-grid-licensing)
[](https://i.imgur.com/5Wb7GHv.png)
- 類似 pool 概念的總量管制, 不綁定 node.
- client 運行時 (執行 pbrun) , 先跟 license server 借用 gpu 授權, 用完後歸還
- **https://forums.developer.nvidia.com/t/planning-to-set-up-the-parabricks-environment/210543/2?u=tj_tsai**
> **Flexera License Server** - This offers more flexibility as the licenses are not directly tied to GPUs. If you buy 8 licenses, you add them to the Flexera LS, and ***they can be ‘borrowed’ by Parabricks and then returned after your job finishes.** So you can share those 8 licenses amongst multiple people using Parabricks.
- [透過 web UI，查看 license 租借住況](https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html#viewing-all-borrowed-licenses)
<br>
6. **在本地端的 client，將 flexera-server IP & Port 填到 config.txt**
- #### 在 config 檔中，告知 license server 在哪裡
```bash
# Create the license file. <-- ???
$ sudo echo "flexera-server:<SERVER_IP>" >> /usr/local/parabricks/config.txt
```
:::warning
:warning: [NV 文件](https://docs.nvidia.com/clara/parabricks/3.7.0/GettingStarted/FlexeraDebian.html) 應該要
將`"flexera-server:<SERVER_IP>"`
更改為 `"flexera-server:<SERVER_IP:PORT>"`
請參考：
1. [[HackMD] AWS / EC2(VM) / Parabricks 執行環境設定 (使用 Docker Container)](https://hackmd.io/N0HBsUUnTUC7ZXubWNy2Ow)
2. [NV 的 Q6 回覆](https://forums.developer.nvidia.com/t/planning-to-set-up-the-parabricks-environment/210543/4?u=tj_tsai)
Q6: I’m not sure I understand what you’re asking, can you clarify? The License Server is wherever you installed it. So if you installed it on a machine with IP address 10.10.10.10 you would put ==**“flexera-server:10.10.10.10:7070”**== in the command.
:::
- **修改前**
```bash
$ cat /usr/local/parabricks/config.txt
baremetal
x86_64
```
- **修改後 (添加 flexera-server 到 config.txt)**
```bash
$ sudo echo "flexera-server:192.169.35.141:7070" >> /usr/local/parabricks/config.txt
```
```bash
$ cat /usr/local/parabricks/config.txt
baremetal
x86_64
flexera-server:192.169.35.141:7070 <---
```
:::warning
:warning: **注意事項**
1. **檔案位置**
- **`license.bin`**
`/usr/local/parabricks/license.bin`
- **`config.txt`**
`/usr/local/parabricks/config.txt`
<br>
2. **`license.bin` 和 `flexera-server` 資訊不能並存(共存)，否則會有錯誤**
> Cannot use separate license.bin in conjunction with flexera licensing. Either remove the separate license.bin or remove the flexera server line from the installation config.txt
> Exiting...
>
> Could not run fq2bam
> Exiting pbrun ...
>
> i.e. 不能並存，只能擇一
3. **flexera-server 連不上會有錯誤訊息**
> [PB Warning 2022-Apr-08 16:42:50][FlexeraClient.cpp:606] Error: failed server communication: err 0x74000008 sys 0x7: [1,7df,3,0[74000008,7,110001d0]] Generic communications error.
>
> [1,7df,3,0[75000001,7,300101b5]] General data transfer failure. Couldn't connect to server
>
> Could not run fq2bam
> Exiting pbrun ...
:::
<br>
### 參考資料
- [[Nvidia][doc] License Server User Guide](https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html)
- NVIDIA Licensing Center
= NVIDIA Licensing Portal (NVIDIA License Portal )
- Documentation for system administrators that explains how to install, configure, and manage the NVIDIA virtual GPU software license server.
- A **primary** license server, which is actively serving licenses to NVIDIA vGPU software clients
- A **secondary** license server, which acts as a backup for the primary license server
<br>
<hr>
<br>
## ==Q & A==
- ### [[Nvidia][Forums] Planning to set up the Parabricks environment](https://forums.developer.nvidia.com/t/planning-to-set-up-the-parabricks-environment/210543)
- **規則補充**
- 授權類型有兩種
- Node-lock
- 提供 GPU UUID 給 NV 製作 license
- 限定這些綁定的 GPU
- Flexera License Server (Flexera LS)
- 不綁定 GPU、擁有更大的彈性
- 可在多人之間可共享
- Flexera LS 負責租借 GPU 管理
- **Q1: "license server" 是指什麼？**
一個 Apache Tomcat web server
- **Q2: 是否有關於 "license server" 的文件？**
https://docs.nvidia.com/grid/ls/latest/grid-license-server-user-guide/index.html
- **Q3: pbrun 在驗證 license，能否不連網直接驗證？**
- **Node-lock 授權類型**
不需要連網，因為已經燒入 license.bin
(NV 喜歡用 baked into)
- **Flexera License Server 授權類型**
視 Flexera LS 安裝位置所在；
如果在同一台機器上，則不需要連網
- ### 對於 Node-lock，如果購買 8 張 GPU 授權，當某一張 GPU 壞掉並更換新的後，授權是否能重新更新？
- ### pbrun 如果在執行途中 crash，pbrun 是否能歸還 Parabricks 借閱 flag?
<br>
<hr>
<br>
## ==By Diatango==
- ### Study Nvidia Flexara License Server
2022/4/15
[](https://i.imgur.com/jBQlAr3.png)
- ### [[Nvidia][Forums] Planning to set up the Parabricks environment](https://forums.developer.nvidia.com/t/planning-to-set-up-the-parabricks-environment/210543)
2022/4/11
4. 購買 node-lock license 後, 如果更換 GPU 卡片, 需請 NV 重新提供相應的 license.bin
5. 使用 Flexara Server, 如果 client 在運算過程 crash (或其他因素) 而沒有歸還 GPU,
- 可以請 NV 手動協助歸還
- 或是 刪除 Flexara Server 的database, 並再上傳一次 license.bin. 讓 Flexara Server 狀態正確
6. 從 NV 的回覆, 有可能是 Parabricks (client) 可以透過 config.txt 設定 Flexara Server ip 位置 (尚未找到文件說明)
7. 購買 lincese 後, NV 將會提供如何註冊與創建 enterprise account, 以便登入 NV Licensing Portal
8. NV 目前並沒有提供 Flexara License Server 的試用
Sign in with Wallet
Connect another wallet