建置 Flexera License server (private)
===
###### tags: `Parabricks-v3.8`
###### tags: `基因體`, `NVIDIA`, `Clara`, `Parabricks`, `License`, `Flexera License server`, `Flexera LS`
:::success
:golf: **建置目標：四個步驟**
1. [建置 Apache Tomcat Server (Port: 8080)](#1-建置-Apache-Tomcat-Server)
2. [建置 Flexera License Server (Port: 7070)](#2-建置-Flexera-License-Server)
3. [前往官網，根據 GPU 數量需求，配置 `license_xxx.bin`](#3-前往官網，配置-license_xxxbin)
4. [上傳 `license_xxx.bin` 到 Tomcat Server (Flexera License Server)](#4-上傳-license_xxxbin-到-Tomcat-Server)
<br>
**參考文件：**
- Virtual GPU Software License Server - User Guide - Version 2022.02
- 2.3. Installing the NVIDIA vGPU Software License Server on Linux
:::
<br>
[TOC]
<br>
## 1. 建置 Apache Tomcat Server
> Tomcat 網頁使用 Java 語言撰寫，需要編譯 .java 成 .class，然後執行
### 準備 Java 環境
```bash=
# 這樣才會有 openjdk-8-jdk 選項
$ sudo apt update
# 安裝 JDK 1.8
$ sudo apt install -y openjdk-8-jdk
# 測試 JDK 指令
$ javac -version
# 測試 JRE 指令
$ java -version
```
- 預設 openjdk 選項
[](https://i.imgur.com/5yOMDcl.png)
`apt update` 更新後，出現 openjdk-8 選項
[](https://i.imgur.com/ueOQCEZ.png)
<br>
- 測試 JDK 指令
```
$ javac -version
javac 1.8.0_312
```
- 測試 JRE 指令
```
$ java -version
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-8u312-b07-0ubuntu1~18.04-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)
```
- 不同的 java 套件，對應的 JAVA_HOME 位置不一樣
- `sudo apt install -y openjdk-8-jre`
`java`: `/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java`
`JAVA_HOME`: `/usr/lib/jvm/java-8-openjdk-amd64/jre`
<br>
- `sudo apt install -y default-jre`
`java`: `/usr/lib/jvm/java-11-openjdk-amd64/bin/java`
`JAVA_HOME`: `/usr/lib/jvm/java-11-openjdk-amd64/`
<br>
### 準備 Apache Tomcat 套件
```bash=
# 下載 Tomcat 9.X 版
$ wget https://dlcdn.apache.org/tomcat/tomcat-9/v9.0.65/bin/apache-tomcat-9.0.65.tar.gz
```
- 下載 Tomcat 9.X 版
- https://tomcat.apache.org/download-90.cgi
<br>
- https://dlcdn.apache.org/tomcat/tomcat-9/v9.0.65/bin/apache-tomcat-9.0.65.tar.gz
<br>
### 安裝 Tomcat 套件
> 建立使用者 `tomcat` 來跑 Apache Tomcat server
```bash=
# 建立使用者群組：`tomcat`
$ sudo groupadd tomcat
# 建立使用者：`tomcat`，home 目錄為 `/usr/share/tomcat`
$ sudo useradd -d /usr/share/tomcat -g tomcat -M -s /bin/false tomcat
# 建立 home 目錄
$ sudo mkdir -p /usr/share/tomcat
# 將 Tomcat 安裝在 home 目錄
$ sudo tar xvf apache-tomcat-*.tar.gz -C /usr/share/tomcat --strip-components 1
$ ls -ls /usr/share/tomcat
```
- 查看新增的使用者
```
$ cat /etc/group | tail
```
<br>
- 查看安裝目錄 `/usr/share/tomcat`
```bash
$ ls -ls /usr/share/tomcat
```
<br>
### 修正相關檔案權限
> 變更 tomcat 套件的擁有者與權限
```bash=
$ cd /usr/share/tomcat
$ sudo chgrp -R tomcat /usr/share/tomcat
$ sudo chmod -R g+r conf
$ sudo chmod g+x conf
$ sudo chown -R tomcat webapps work temp logs
```
- `/usr/share/tomcat`: Before vs After
- `conf`: Before vs After
<br>
### 啟動 Tomcat 服務
```bash=
# 準備 tomcat 服務的設定檔 `tomcat.service`
$ sudo nano tomcat.service
```
:::warning
檔案：`tomcat.service`
```ini=
[Unit]
Description=Apache Tomcat Server
After=network.target
[Service]
Type=forking
Environment=JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre
Environment=CATALINA_PID=/usr/share/tomcat/temp/tomcat.pid
Environment=CATALINA_HOME=/usr/share/tomcat
Environment=CATALINA_BASE=/usr/share/tomcat
Environment='CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC'
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom'
ExecStart=/usr/share/tomcat/bin/startup.sh
ExecStop=/usr/share/tomcat/bin/shutdown.sh
User=tomcat
Group=tomcat
UMask=0007
RestartSec=15
Restart=always
[Install]
WantedBy=multi-user.target
```
- 除了 `JAVA_HOME` 參數外，其餘都跟文件一樣
- `JAVA_HOME` 如何填寫？
查看 `JAVA_HOME` 所在位置
[](https://i.imgur.com/7D8PQgb.png)
`JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/` (不包含 `bin`)
:::
```bash=3
# 將 `tomcat.service` 複製到 `/etc/systemd/system/` 目錄下
$ sudo cp tomcat.service /etc/systemd/system/
# 啟動 tomcat 服務
$ sudo systemctl daemon-reload
$ sudo systemctl enable tomcat.service
$ sudo systemctl start tomcat.service
```
- 查看 `tomcat.service` 服務狀態
```
$ sudo systemctl status tomcat.service
```
[](https://i.imgur.com/bCaqLqJ.png)
<br>
- 安裝目錄下的相關檔案
```
ubuntu@license-server:/usr/share/tomcat9$ tree
.
├── bin
│ ├── bootstrap.jar
│ ├── catalina-tasks.xml
│ ├── catalina.sh
│ ├── ciphers.sh
│ ├── configtest.sh
│ ├── daemon.sh
│ ├── digest.sh
│ ├── makebase.sh
│ ├── setclasspath.sh
│ ├── shutdown.sh
│ ├── startup.sh
│ ├── tomcat-juli.jar -> ../../java/tomcat9-juli.jar
│ ├── tool-wrapper.sh
│ └── version.sh
├── default.template
├── etc
│ ├── catalina.properties
│ ├── context.xml
│ ├── jaspic-providers.xml
│ ├── logging.properties
│ ├── server.xml
│ ├── tomcat-users.xml
│ └── web.xml
├── lib
│ ├── annotations-api.jar -> ../../java/tomcat9-annotations-api.jar
│ ├── catalina-ant.jar -> ../../java/tomcat9-catalina-ant.jar
│ ├── catalina-ha.jar -> ../../java/tomcat9-catalina-ha.jar
│ ├── catalina-storeconfig.jar -> ../../java/tomcat9-storeconfig.jar
│ ├── catalina-tribes.jar -> ../../java/tomcat9-tribes.jar
│ ├── catalina.jar -> ../../java/tomcat9-catalina.jar
│ ├── el-api.jar -> ../../java/tomcat9-el-api.jar
│ ├── jasper-el.jar -> ../../java/tomcat9-jasper-el.jar
│ ├── jasper.jar -> ../../java/tomcat9-jasper.jar
│ ├── jaspic-api.jar -> ../../java/tomcat9-jaspic-api.jar
│ ├── jsp-api.jar -> ../../java/tomcat9-jsp-api.jar
│ ├── servlet-api.jar -> ../../java/tomcat9-servlet-api.jar
│ ├── tomcat-api.jar -> ../../java/tomcat9-api.jar
│ ├── tomcat-coyote.jar -> ../../java/tomcat9-coyote.jar
│ ├── tomcat-dbcp.jar -> ../../java/tomcat9-dbcp.jar
│ ├── tomcat-i18n-es.jar -> ../../java/tomcat9-i18n-es.jar
│ ├── tomcat-i18n-fr.jar -> ../../java/tomcat9-i18n-fr.jar
│ ├── tomcat-i18n-ja.jar -> ../../java/tomcat9-i18n-ja.jar
│ ├── tomcat-i18n-ru.jar -> ../../java/tomcat9-i18n-ru.jar
│ ├── tomcat-jdbc.jar -> ../../java/tomcat9-jdbc.jar
│ ├── tomcat-jni.jar -> ../../java/tomcat9-jni.jar
│ ├── tomcat-util-scan.jar -> ../../java/tomcat9-util-scan.jar
│ ├── tomcat-util.jar -> ../../java/tomcat9-util.jar
│ ├── tomcat-websocket.jar -> ../../java/tomcat9-websocket.jar
│ └── websocket-api.jar -> ../../java/tomcat9-websocket-api.jar
└── logrotate.template
3 directories, 48 files
```
- 服務單元檔案 (unit file)
> 底下兩個路徑皆相同
>
- /etc/systemd/system/multi-user.target.wants/tomcat9.service
- /lib/systemd/system/tomcat9.service
```unit
$ cat /lib/systemd/system/tomcat9.service
#
# Systemd unit file for Apache Tomcat
#
[Unit]
Description=Apache Tomcat 9 Web Application Server
Documentation=https://tomcat.apache.org/tomcat-9.0-doc/index.html
After=network.target
[Service]
# Configuration
Environment="CATALINA_HOME=/usr/share/tomcat9"
Environment="CATALINA_BASE=/var/lib/tomcat9"
Environment="CATALINA_TMPDIR=/tmp"
Environment="JAVA_OPTS=-Djava.awt.headless=true"
EnvironmentFile=-/etc/default/tomcat9
# Lifecycle
Type=simple
ExecStartPre=+/usr/libexec/tomcat9/tomcat-update-policy.sh
ExecStart=/bin/sh /usr/libexec/tomcat9/tomcat-start.sh
SuccessExitStatus=143
Restart=on-abort
# Logging
SyslogIdentifier=tomcat9
# Security
User=tomcat
Group=tomcat
PrivateTmp=yes
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true
LogsDirectory=tomcat9
LogsDirectoryMode=750
CacheDirectory=tomcat9
CacheDirectoryMode=750
ProtectSystem=strict
ReadWritePaths=/etc/tomcat9/Catalina/
ReadWritePaths=/var/lib/tomcat9/webapps/
[Install]
WantedBy=multi-user.target
```
- Azure VM / 網路設定
[](https://i.imgur.com/GO0TfT6.png)
- 要開啟 8080, 7070 port 來測試
- 實際測試網頁
http://40.74.243.250:8080/ (範例網址)
[](https://i.imgur.com/4oMgFY0.png)
<br>
### 變更服務的 port
修改 Tomcat 服務設定檔：`/usr/share/tomcat/conf/server.xml`
底下範例是將 port 8080 修改為 6060
```xml=62
<!-- A "Connector" represents an endpoint by which requests are received
and responses are returned. Documentation at :
Java HTTP Connector: /docs/config/http.html
Java AJP Connector: /docs/config/ajp.html
APR (HTTP/AJP) Connector: /docs/apr.html
Define a non-SSL/TLS HTTP/1.1 Connector on port 8080
-->
<Connector port="6060" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443" />
```
重啟 Tomcat 服務，並測試瀏覽器是否能透過 port 6060 存取 Tomcat 網頁
```bash
# 重啟 tomcat service
$ sudo systemctl restart tomcat
# 查看 tomcat service 狀態是否是綠燈
$ sudo systemctl status tomcat
```
- 若有設定防火牆，相關設定也要一併修改
<br>
<hr>
<br>
## 2. 建置 Flexera License Server
> 資料來源：
> - 2.3.4. Installing the License Server Software on Linux in Console Mode
<br>
### 準備 Flexera License Server 軟體
- 下載 `NVIDIA-ls-linux-2022.02-2022.02.0.30886060.zip` 軟體
1. 前往 https://ui.licensing.nvidia.com/login
2. 使用企業帳號登入
3. 點擊 **NVIDIA许可门户网站**
4. 依序點擊：
- **SOFTWARE DOWNLOADS**
- **ADDITIONAL SOFTWARE**
- **2022.02 64-bit License Manager for Linux**
- 流程示意圖
[](https://i.imgur.com/8to4fcn.png)
<br>
### 上傳 Flexera License Server 軟體
- 將 `NVIDIA-ls-linux-2022.02-2022.02.0.30886060.zip` 軟體上傳到要安裝的主機
底下為 Azure VM 的操作範例
```bash
$ scp -i license-server_key.pem \
NVIDIA-ls-linux-2022.02-2022.02.0.30886060.zip \
ubuntu@40.74.243.250:/home/ubuntu/
```
<br>
### 安裝 Flexera License Server 軟體
```bash=
$ sudo apt install unzip
$ unzip NVIDIA-ls-linux-2022.02-2022.02.0.30886060.zip
$ cd NVIDIA-ls-linux-2022.02-2022.02.0.30886060
$ chmod +x setup.bin
$ sudo ./setup.bin -i console
```
- 安裝參數
- :::spoiler help 指令說明
```
$ sudo ./setup.bin -h
Preparing to install
Extracting the installation resources from the installer archive...
Configuring the installer for this system's environment...
Launching installer...
Usage: setup [-f <path_to_installer_properties_file> | -options]
(to execute the installer)
where options include:
-? show this help text
-h show this help text
-help show this help text
--help show this help text
-i [gui | console | silent]
specify the user interface mode for the installer
-D<name>=<value>
specify installer properties
-r <path_to_generate_response_file>
Generates response file.
JVM heap size options are only applicable to Installers
-jvmxms <size>
Specify JVM initial heap size.
-jvmxmx <size>
Specify JVM maximum heap size.
The options field may also include the following in case of uninstaller
if it is enabled for Maintenance Mode
-add <feature_name_1> [<feature_name_2 ...]
Add Specified Features
-remove <feature_name_1> [<feature_name_2 ...]
Remove Specified Features
-repair
Repair Installation
-uninstall
Uninstall
notes:
1. the path to the installer properties file may be either absolute,
or relative to the directory in which the installer resides.
2. if an installer properties file is specified and exists, all other
command line options will be ignored.
3. if a properties file named either 'installer.properties' or
<NameOfInstaller>.properties resides in the same directory as the
installer, it will automatically be used, overriding all other command
line options, unless the '-f' option is used to point to another valid
properties file.
4. if an installer properties file is specified but does not exist, the
default properties file, if present, will be used. Otherwise, any
supplied command line options will be used, or if no additional
options were specified, the installer will be run using the default
settings.
```
- 由檔案導入安裝參數
`sudo ./setup.bin -f installer.properties`
- `installer.properties` 範例
```
PORT=6060
author=tj tsai
token=abcdefghijk
```
- 如果目錄下有 `xxx..properties` 檔案，即使沒有 `-f` 也會自動讀取
<br>
- 由指令(`-Dxxx=yyy`)導入安裝參數
```
# 有寫入參數設定，
$ sudo ./setup.bin -DPORT_SPEC=6666 -Dauthor="tj tsai---" -Dtoken=abcdefghijklmnopq
```
```
# 無效參數
$ sudo ./setup.bin --port 6060
```
- **指令參數**會蓋掉**檔案參數**
<br>
- 查看設定參數
```
$ cat /opt/flexnetls/nvidia/installvariables.properties | sort
```
- 結論：底下這些設定，仍無法變更 port 7070
```
--port 6060
-DPORT=6060
-DPORT="PORT: 6060"
-DPORT="#PORT: 6060"
-DPORT_SPEC=6060
-DPORT_SPEC="PORT: 6060"
-DPORT_SPEC="#PORT: 6060"
```
- 自動安裝指令
```
echo -e '\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\nY\n\n/usr/share/tomcat\n\n\n\n' | sudo ./setup.bin -i console
```
自動反安裝指令
```
echo -e '\n' | sudo /opt/flexnetls/nvidia/Change\ License\ Server\ Installation
```
<br>
- [安裝過程] 預設安裝路徑
```
/opt/flexnetls/nvidia
```
- [安裝過程] Choose Firewall Options
[](https://i.imgur.com/2TT9PZ3.png)
- 安裝目錄下的相關檔案
```
ubuntu@license-server:/opt/flexnetls/nvidia$ tree
.
├── Change License Server Installation
├── Change License Server Installation.lax
├── InstallScript.iap_xml
├── Logs
│ └── License_Server_Install_08_05_2022_06_57_11.log
├── console.sh
├── enterprise
│ ├── nvidialsadmin.jar
│ └── nvidialsadmin.sh
├── flexnetls.jar
├── installvariables.properties
├── lib
│ ├── EccpressoAll.jar
│ ├── commons-codec-1.11.jar
│ └── flxBinary.jar
├── local-configuration.yaml
├── producer-settings.xml
├── server
│ ├── configure
│ ├── configure-ls-port.sh
│ ├── flexnetls
│ ├── flexnetls.jar
│ ├── install-functions.sh
│ ├── install-ls-port-service.sh
│ ├── install-systemd.sh
│ ├── producer-settings.xml
│ ├── regid.2009-06.com.flexerasoftware_254cc527-bfc8-4894-a14e-a205aee42205.swidtag
│ └── selfcontainedServer.tgz
├── ui
│ ├── fne.war
│ └── licserver.war
└── uninstaller.jar
5 directories, 27 files
```
- 服務單元檔案 (unit file)
> 底下兩個路徑皆相同
>
- /etc/systemd/system/multi-user.target.wants/flexnetls-nvidia.service
- /etc/systemd/system/flexnetls-nvidia.service
```unit
[Unit]
Description=FlexnetLS Local License Server for nvidia.
After=network.target syslog.target
[Service]
WorkingDirectory=/opt/flexnetls/nvidia
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p ${DATA_DIR} /var/opt/flexnetls/nvidia/logs
ExecStartPre=/bin/chown -R ${FNLS_USER}:${FNLS_GROUP} ${DATA_DIR} /var/opt/flexnetls/nvidia/logs
ExecStartPre=/bin/chmod g+w ${DATA_DIR} /var/opt/flexnetls/nvidia/logs
ExecStartPre=/bin/chmod g+s ${DATA_DIR} /var/opt/flexnetls/nvidia/logs
ExecStart=/usr/bin/java $JVMOPTS $DEFINES -jar flexnetls.jar $OPTIONS --service
ExecStop= /usr/bin/java $DEFINES -jar flexnetls.jar --service-shutdown --logging-threshold=DEBUG
KillMode=process
Restart=on-failure
RestartSec=60
User=flexnetls
SuccessExitStatus=143
TimeoutStopSec=60
StandardOutput=journal
SyslogIdentifier=flexnetls-nvidia
SyslogLevelPrefix=true
[Install]
WantedBy=multi-user.target
```
- Azure VM
- 沒看到有設定 $JVMOPTS, $OPTIONS？
- 查詢啟動指令
```
$ sudo netstat -tulpn | grep 7070
```
- 啟動指令
> /usr/bin/java -server -Xms2g -Xmx2g -XX:CompressedClassSpaceSize=64m -XX:MetaspaceSize=256m -XX:+UseG1GC -XX:NewRatio=3 -XX:MaxGCPauseMillis=75 -XX:G1HeapWastePercent=10 -XX:InitiatingHeapOccupancyPercent=75 -XX:+CMSScavengeBeforeRemark -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED -XX:+ScavengeBeforeFullGC -XX:-OmitStackTraceInFastThrow -Djava.security.egd=file:/dev/./urandom -Dbase.dir=/var/opt/flexnetls/nvidia -jar flexnetls.jar --service
- Azure VM / 網路設定
[](https://i.imgur.com/GO0TfT6.png)
- 要開啟 7070 port 來測試
- 測試 Flexera License 界面
http://40.74.243.250:8080/licserver/
<br>
### 變更服務的 port
1. 將 Flexera License Server 的 port 變更為 6060 (範例)
```
$ sudo nano /opt/flexnetls/nvidia/local-configuration.yaml
```
```yaml=
# local-configuration.
# HTTP listening port. Default is 7070. You can bind to an interface with this syntax: '[127.0.0.1].7070'.
port: 6060
```
2. 重啟 Flexera License Server，需等候約 30 秒才能完成重啟
```
$ sudo systemctl restart flexnetls-nvidia.service
```
3. 修改 pbrun 的 config.txt
```
$ nano /usr/local/parabricks/config.txt
```
```
baremetal
x86_64
flexera-server:20.114.75.34:6060
```
- 執行 pbrun 測試，即可驗證
- 或是透過 API 來驗證
```
$ curl your_ip:6060/api/1.0/instances/
```
底下為操作範例：
```
$ curl http://20.114.75.34:6060/api/1.0/instances/
```
```json
{
"id" : 1,
"serverInstanceId" : "QZ8PPYZXZ997",
"requestId" : "1",
"tenantId" : "",
"pendingDeletion" : false,
"maintenanceEndTime" : "1970-01-01T00:00:00.000Z",
"suspended" : false,
"ready" : true,
"publisherName" : "nvidia",
"lastUpdateTime" : "2022-08-02T05:04:10.070Z",
"failOverRole" : "MAIN",
"lastUpdateHostid" : {
"hostidValue" : "000D3A5E17DD",
"hostidType" : "ETHERNET"
},
"lastServerToServerSyncTime" : "1970-01-01T00:00:00.000Z",
"identityName" : "nvidia-grid",
"lastClientDropTime" : "1970-01-01T00:00:00.000Z",
"lastSyncTime" : "1970-01-01T00:00:00.000Z"
}
```
4. 修改 Tomcat Server 上 licserver 的管理界面
- 變更之前，licserver 無法存取原本的 7070 port，存取會有錯誤訊息
- 修改 port 為 6060
點擊 **License Server Manager** > **Settings**
最後按儲存即生效
5. 參考資料
- [How to change the License Server default TCP/IP communication port number (7070) to a custom value](https://www.l3harrisgeospatial.com/Support/Self-Help-Tools/Help-Articles/Help-Articles-Detail/ArtMID/10220/ArticleID/15103)
<br>
<hr>
<br>
## 3. 前往 NVIDIA 官網，製作 `license_xxx.bin`
1. **再次前往 [NVIDIA License Portal](https://ui.licensing.nvidia.com/)**
2. **製作 `license_xxx.bin`**
- 點擊 **LICENSE SERVERS** > **CREATE SERVER**
[](https://i.imgur.com/2kqx8Jb.png)
<br>
- 查詢主機的 MAC address
```bash=30
# Step 6: check network interfaces and find out the MAC address of the host
$ sudo apt install net-tools
$ ifconfig
$ ifconfig | egrep "[[:alnum:]]{2}(:[[:alnum:]]{2}){5}"
```
- 比如：`02:42:ac:11:00:02`
<br>
- 填寫 **Basic details**
[](https://i.imgur.com/ns5bz7w.png)
- 勾選 **Create legacy server**
- 輸入 **Name**、**Description**、**MAC Address**
- 點擊 **Next: Failover server configuration(optional)**
<br>
- **Failover server configuration(optional)** 設定
- 略過設定，直接點擊 **Next: Select features**
<br>
- **Select features** 設定
[](https://i.imgur.com/S1hrgAl.png)
- 勾選 Entitlements (已購買的授權產品)
- 數量：在 License Server 上想要配置的 license 數量
<br>
- 確認後，即可按建立
[](https://i.imgur.com/Cnl5WbX.png)
<br>
- 建立結果
[](https://i.imgur.com/eNIWWxO.png)
- License Server 所需的 `license_xxx.bin` 已經製作完成
<br>
3. **下載 `license_xxx.bin`**
<br>
- 點擊 **LICENSE SERVERS** > **LIST SERVERS** > 剛建立的 License Server
[](https://i.imgur.com/P7ME1DX.png)
<br>
- 點擊 **Action** > **Download**
[](https://i.imgur.com/iQgd6Vg.png)
<br>
- 確認下載，接著會自動下載 **`license_0242AC110002_08-05-2022-14-10-15.bin`**(範例)
(每次下載，時間戳記都會不同)
[](https://i.imgur.com/Ysnlwbu.png)
<br>
<hr>
<br>
## 4. 上傳 `license_xxx.bin` 到 NVIDIA License Server
1. 上傳 `license_xxx.bin`
- 點擊：**License Management** > **Choose File: `license_xxx.bin`** > **Upload**
<br>
2. 查看 license 可用額度
- 點擊：**Licensed Feature Usage**
<br>
<br>
<hr>
<br>
## 5. pbrun 指令連線測試
### service 不存在
例如 IP 或 port 輸入錯誤，沒有指到正確位址，會有底下的錯誤訊息。
```
$ pbrun germline ...
...
[PB Warning 2022-Aug-02 11:03:30][FlexeraClient.cpp:606] Error: failed server communication: err 0x74000008 sys 0x1c: [1,7df,3,0[74000008,1c,110001d0]] Generic communications error.
[1,7df,3,0[75000001,28,300101b5]] General data transfer failure. Timeout was reached
For technical support visit https://docs.nvidia.com/clara/parabricks/3.8.0/index.html#how-to-get-help
Exiting...
[PB Warning 2022-Aug-02 11:04:32][FlexeraClient.cpp:606] Error: failed server communication: err 0x74000008 sys 0x1c: [1,7df,3,0[74000008,1c,110001d0]] Generic communications error.
[1,7df,3,0[75000001,28,300101b5]] General data transfer failure. Timeout was reached
Could not run fq2bam as part of germline pipeline
Exiting pbrun ...
...
```
<br>
<hr>
<br>
## 6. 反安裝 Flexera License Server 軟體
> 資料來源：7.2. Uninstalling the NVIDIA vGPU Software License Server on Linux
```bash=
$ cd /opt/flexnetls/nvidia/
$ sudo ./Change\ License\ Server\ Installation
```
- :::spoiler 反安裝過程
```
$ sudo ./Change\ License\ Server\ Installation
===============================================================================
License Server (created with InstallAnywhere)
-------------------------------------------------------------------------------
Preparing CONSOLE Mode Uninstallation...
===============================================================================
Uninstall License Server
------------------------
About to uninstall...
License Server
This will remove features installed by InstallAnywhere. It will not remove
files and folders created after the installation.
PRESS <ENTER> TO CONTINUE:
Stopping NVIDIA License Server
Executing NVIDIA License Server's uninstall script
===============================================================================
Uninstalling...
---------------
...
...*
*
*************************
*************************
*************************
*************************
===============================================================================
Uninstall Complete
------------------
All items were successfully uninstalled.
```
:::
:::warning
:warning: 注意事項：
1. 資料夾 `/opt/flexnetls/nvidia` 會只剩 Logs
:::
底下指令無效果
```bash=
$ sudo ./setup.bin -h # 查用法
$ sudo ./setup.bin -uninstall
```
<br>
<hr>
<br>
## 注意事項
### 網路存取問題
- Azure VM
- 無法使用 `ping` 因為 Azure 負載平衡器不允許使用 ICMP 協定 ([詳見說明](https://social.msdn.microsoft.com/Forums/azure/en-US/33248818-743a-473d-997c-48930087cc74/))
- 解法
- ping 是使用 ICPM 協定
- port 是使用 TCP/UDP 協定，可透過 curl, wget 存取網頁服務來判斷
### 管理界面安全性問題
- ### [doc: page6 (14 of 130)] 沒有實作存取控制，預設沒有開放 port 8080
**2.1.3. Network Ports and Management Interface**
- ### [doc: page29 (37 of 130)] 防火牆設定
- ### [doc: page32 (40 of 130)] 登入後 30 分鐘內，他人可免登入
> other users can access the license server through a browser at the same URL without the need to log in for up to 30 minutes after the first successful login.
:warning: 會有登入空窗期
<br>
<hr>
<br>
## 參考資料
- ### [[HackMD] 安裝方式 - 簡易說明](https://hackmd.io/PyY-fPTqTcye1acCRweDgg#%E5%AE%89%E8%A3%9D%E6%96%B9%E5%BC%8F)
- ### [FlexNet Embedded 2018 R2 SP3 License Server Administration Guide](https://help.hcltechsw.com/commerce/9.1.0/install/pdf/FlexNet%20Embedded%20License%20Server%20Administration%20Guide.pdf)
<br>
<hr>
<br>
## 術語
- Flexera LS: [Flexera License Server](https://docs.nvidia.com/clara/parabricks/3.7.0/GettingStarted/FlexeraDebian.html#license-installation)
- FNE Server: [FlexNet Embedded Server](https://www.openlm.com/knowledge-base/what-is-flexnet-embedded/)
<br>
<hr>
<br>
## 錯誤排解
- ### 修改 tomcat.service unit 設定，重啟後有 error
log:
```
$ sudo systemctl restart tomcat
Warning: The unit file, source configuration file or drop-ins of tomcat.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for tomcat.service failed because the control process exited with error code.
See "systemctl status tomcat.service" and "journalctl -xe" for details.
```
[](https://i.imgur.com/UF0nmxh.png)
- 解法：
```bash
$ sudo systemctl daemon-reload
$ sudo systemctl restart tomcat
```
- ### flexnetls-nvidia 服務有錯誤訊息
log:
> Connection refused (Connection refused)
> Connection error: Please make sure the FNE server is up and running
[](https://i.imgur.com/6pRYeEW.png)
[](https://i.imgur.com/JOz1265.png)
- ["Connection error. Please make sure the FNE server is up and running".](https://nvidia.custhelp.com/app/answers/detail/a_id/4111/~/grid-license-server-fails-to-start-due-to-ip-address-change)
> [另一個 link](https://enterprise-support.nvidia.com/s/article/GRID-License-Server-Fails-to-Start-Due-to-IP-Address-Change)
- 實際解法：
- 應該是記憶體給太少，起不來
<br>
- 失敗情況
- 正常情況
- ### 使用 root 安裝 flexnetls-nvidia 會有錯誤
```
# ./setup.bin -i console
...
==============================================================
Check Authorization Level
-------------------------
The current user is not authorized as a sudoer, please authorize or run as different user.
PRESS <ENTER> TO ACCEPT THE FOLLOWING (OK):
```
- ### 沒有 systemd 安裝 flexnetls-nvidia 會失敗
```
$ sudo ./setup.bin -i console
...
Executing NVIDIA License Server Installation Script...
==============================================================
NVIDIA License Server installation failed.
------------------------------------------
License Server Install script failed. Otuput of script:
exitcode: 1
output: systemd does not appear to be in use (java detected)
For more details check the log file /opt/flexnetls/nvidia/Logs/.
PRESS <ENTER> TO EXIT THE INSTALLER:
Starting NVIDIA License Server
==============================================================
NVIDIA License Server installation failed.
------------------------------------------
Failed to start License Server.Output of start script:
exitcode: 1
output:
Press "ENTER" to quit the installer.
PRESS <ENTER> TO EXIT THE INSTALLER:
Opening License Server Port 7070 in Firewall
Starting Tomcat Service
==============================================================
Install Complete
----------------
License Server has been successfully installed to:
/opt/flexnetls/nvidia
PRESS <ENTER> TO EXIT THE INSTALLER:
```
<br>
<hr>
<br>
## 簡易介紹
1. **license.bin vs flexera license**
- Trial License (license.bin, 試用版)
- Per GPU Hour License (公有雲：Azure, AWS)
- GPU Based License
- Node Locked (節點鎖定) (綁定 GPU UUID) (單機版)
- Flexera based (浮動授權) (不綁定 GPU UUID) (多機版)
- Research License
2. **license.bin**
- 特色
- 不限定特定 GPU 數量
- 可在多節點同時使用
- 用法
- 放在根目錄下
- 透過環境變數指定位置
3. **flexera license**
- 特色
- 不綁訂 GPU
- 跟 server 租借 GPU，用完就歸還
- 有一定數量限制
- 用法
- 在 image 範本設置環境變數
- 透過環境變數，指定 Flexera server IP & port
4. **兩者並行**
- license.bin 優先權高於 lexera license
- lexera license 環境變數就會被忽略
Sign in with Wallet
Connect another wallet