# Singularity
## 快速上手
singularity載完可以執行後(注意dockerfile在singularity的限制:root/home之類的、workdir、絕對路徑之類的)
pull在dockerhub上的檔案之後會轉變成SIF檔
開啟互動式環境基本指令
```
singularity shell --writable-tmpfs 你的SIF檔
```
[在singularity遇到root home的問題](https://hackmd.io/magj1td5RIiqhGLoK2avmw#317)
[Singularity 啟用 Fakeroot](https://yylin.io/2020/10/25/singularity-fakeroot/)
## 下載方法
參考:[Linux 安裝 Singularity 3 容器與基本使用教學](https://blog.gtwang.org/linux/singularity-3-installation-and-usage-tutorial/)
### 安裝流程
```
# 更新套件庫
sudo yum update
# 安裝開發者套件
sudo yum groupinstall 'Development Tools'
# 安裝必要套件
sudo yum install libarchive-devel wget openssl-devel libuuid-devel squashfs-tools
```
```shell=
wget https://golang.org/dl/go1.16.3.linux-amd64.tar.gz
tar -C /usr/local/ -xzf go1.16.3.linux-amd64.tar.gz
##路徑加到bashrc 或 /etc/profile
vim /etc/profile
export PATH=$PATH:/usr/local/go/bin
source /etc/profile
go version
# 下載 Singularity 原始碼
git clone https://github.com/sylabs/singularity.git
# 編譯 Singularity
cd singularity
./mconfig
make -C builddir
# 安裝 Singularity
sudo make -C builddir install
# 查詢 Singularity 版本
singularity version
```
>在`./mconfig`時出錯
>```shell=
># 下載 Singularity 原始碼
>git clone https://github.com/sylabs/singularity.git
>
># 編譯 Singularity
>cd singularity
>./mconfig
>make -C builddir
>
># 安裝 Singularity
>sudo make -C builddir install
>```
>出現以下
>
用最新版本的GO成功
## 指令
全部指令
```
$ singularity help
Linux container platform optimized for High Performance Computing (HPC) and
Enterprise Performance Computing (EPC)
Usage:
singularity [global options...]
Description:
Singularity containers provide an application virtualization layer enabling
mobility of compute via both application and environment portability. With
Singularity one is capable of building a root file system that runs on any
other Linux system where Singularity is installed.
Options:
-d, --debug print debugging information (highest verbosity)
-h, --help help for singularity
--nocolor print without color output (default False)
-q, --quiet suppress normal output
-s, --silent only print errors
-v, --verbose print additional information
Available Commands:
build Build a Singularity image
cache Manage the local cache
capability Manage Linux capabilities for users and groups
exec Run a command within a container
help Help about any command
inspect Show metadata for an image
instance Manage containers running as services
key Manage OpenPGP keys
oci Manage OCI containers
plugin Manage singularity plugins
pull Pull an image from a URI
push Upload image to the provided library (default is "cloud.sylabs.io")
remote Manage singularity remote endpoints
run Run the user-defined default command within a container
run-help Show the user-defined help for an image
search Search a Container Library for images
shell Run a shell within a container
sif siftool is a program for Singularity Image Format (SIF) file manipulation
sign Attach a cryptographic signature to an image
test Run the user-defined tests within a container
verify Verify cryptographic signatures attached to an image
version Show the version for Singularity
Examples:
$ singularity help <command> [<subcommand>]
$ singularity help build
$ singularity help instance start
For additional help or support, please visit https://www.sylabs.io/docs/
Information about subcommand can also be viewed with the help comm
```
## 與docker的差別
檔案模式
>Singularity.SIF(singularity image format)
## 注意
Singularity bind mounts /home/$USER, /tmp, and $PWD into your container at runtime.
```
共用地方:/home/$USER, /tmp, $PWD
```
## 指令
可以操作檔案形式有以下
* *.sif
* *.sqsh
* *.img
* directory/* (sandbox format)
* instance://* (local running instance)
* library://* (SIF container hosted on a library)
* docker://* (docker/OCI container on Docker Hub or OCI registry)
* shub://* (container on Singularity Hub)
* oras://* (SIF container on OCI registry...)
### 啟動
安裝完即可使用，無啟動用法
### 版本
`singularity version`
### 基本指令模式
```shell=
singularity [指令] [影像檔位置/可操作之目標]
```
### RUN
Run the user-defined default command within a container
```shell=
singularity run 影像檔位置
singularity run library://sylabsed/examples/lolcow
```
### shell
開啟互動式的 shell 環境
```shell=
singularity shell library://sylabsed/examples/lolcow
```
### 下載及上傳
下載->使用`pull`指令
```shell=
singularity pull library://sylabsed/examples/lolcow
singularity pull docker://godlovedc/lolcow(下載docker image)
```
下載後會變成SIF檔(下載在主機/home裡面)
SIF檔可用RUN啟動或直接啟動 EX:`./lolcow_latest.sif`
上傳->`push`
### 在容器中執行指令
`exec`Run a command within a container
```shell=
singularity exec lolcow_latest.sif cowsay moo
```
->須知道:此指令為用容器中的環境及指令執行檔案
EX
```
singularity exec lammps_openmpi-3.1.4_cmake.sif mpirun --allow-run-as-root -np 4 lmp < in.lj
```
其中in.lj為在容器外部的input檔 mpirun與lmp為容器內部的指令
### 檔案與管線
Singularity 在預設的狀況下會自動綁定 /home/$USER、/tmp 與 $PWD 這三個實體機器上的目錄，也就是說在 Singularity 容器中可以直接存取這三個目錄下的檔案
```shell=
# 建立 $HOME 下的檔案
echo "Hello, Singularity." > $HOME/hostfile.txt
# 於 Singularity 中存取 $HOME 下的檔案
singularity exec lolcow_latest.sif cat $HOME/hostfile.txt
```
>`Hello, Singularity.`
資料除了以檔案傳遞之外，亦可直接透過管線（pipe）的方式，直接導入 Singularity 容器中：
```shell=
# 使用管線傳遞資料進入 Singularity 容器
echo "Hello, Singularity." | singularity exec lolcow_latest.sif cowsay
```
## 自訂 Singularity 影像檔
### sandbox
```
# 建立沙箱目錄
singularity build --sandbox ubuntu/ library://ubuntu
```
這樣可以將 ubuntu 這個 Singularity 影像檔解開，放在一個名稱為 ubuntu 的沙箱目錄中，我們可以使用 `shell`、`exec` 或 `run` 指令來操作這個沙箱目錄，操作時若需要寫入資料，可以加上 `--write` 參數：
```
# 使用沙箱目錄自訂容器環境
singularity shell --writable ubuntu/
```
將環境準備好之後，再將其打包起來，變成一個新的 SIF 檔案：
```
# 將沙箱目錄打包成 SIF 檔案
singularity build new_ubuntu.sif ubuntu
```
### doc(def)
* singularity python可將dockerfile和singularity def互轉
https://singularityhub.github.io/singularity-cli/recipes
另外一種自訂 Singularity 影樣檔的方式是使用定義檔，首先準備如下的定義檔 lolcow.def：
```shell=
BootStrap: library
From: ubuntu:16.04
%post
apt-get -y update
apt-get -y install fortune cowsay lolcat
%environment
export LC_ALL=C
export PATH=/usr/games:$PATH
%runscript
fortune | cowsay | lolcat
%labels
Author GodloveD
```
然後即可以既有的 Singularity 影像檔為基礎，建立新的影像檔：
```
# 以 Singularity 定義檔建立影像檔
sudo singularity build lolcow.sif lolcow.def
```
DEF file 寫法
[Definition Files](https://sylabs.io/guides/3.7/user-guide/definition_files.html)
[Getting Started with Bootstrap](https://singularity.lbl.gov/archive/docs/v2-3/bootstrap-image)
分成兩個部分:header and section
#### header
全部都要以Bootstrap:開頭
eg
```
Bootstrap: docker
From: ubuntu:latest
```
for centos-7其中一種寫法
```
BootStrap: yum
OSVersion: 7
MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/
Include: yum
```
Preferred bootstrap agents
library (images hosted on the Container Library)
docker (images hosted on Docker Hub)
shub (images hosted on Singularity Hub)
oras (images from supporting OCI registries)
scratch (a flexible option for building a container from scratch)
Other bootstrap agents
localimage (images saved on your machine)
yum (yum based systems such as CentOS and Scientific Linux)
debootstrap (apt based systems such as Debian and Ubuntu)
oci (bundle compliant with OCI Image Specification)
oci-archive (tar files obeying the OCI Image Layout Specification)
docker-daemon (images managed by the locally running docker daemon)
docker-archive (archived docker images)
arch (Arch Linux)
busybox (BusyBox)
zypper (zypper based systems such as Suse and OpenSuse)
#### section
* %setup
在build process中，在基本OS安裝後，在hostsystem於容器外執行此類別指令，You can reference the container file system with the `$SINGULARITY_ROOTFS` environment variable in the `%setup` section.
```
%setup
touch /file1 ##建立file1
touch ${SINGULARITY_ROOTFS}/file2 ##建立file2
```
* %files
把外面的檔案CO進去
```
%files
/file1
/file1 /opt ##把file1 COPY 到容器的/opt
```
* %post
相當於dockerfile裡的RUN
* %environment
相當於ENV
* 須注意
* during build: The `%environment` section is written to a file in the container metadata directory. This file is not sourced.建立時需要的變數要另外宣告 EX`PATH=$PATH:/openmpi/bin`
* during runtime: The file in the container metadata directory is sourced.
## singularity 跨機說明
[Singularity and MPI applications](https://sylabs.io/guides/3.4/user-guide/mpi.html)
分成兩種model
Hybrid model、bind model
### hybrid model
說簡單點就是外面的mpi會跟容器內部的mpi連通，就等於是說執行容器內部的mpi一樣
:::info
workflow
1.The MPI launcher (e.g., mpirun, mpiexec) is called by the resource manager or the user directly from a shell.
2.Open MPI then calls the process management daemon (ORTED).
3.The ORTED process launches the Singularity container requested by the launcher command.
4.Singularity instantiates the container and namespace environment.
5.Singularity then launches the MPI application within the container.
6.The MPI application launches and loads the Open MPI libraries.
7.The Open MPI libraries connect back to the ORTED process via the Process Management Interface (PMI).
At this point the processes within the container run as they would normally directly on the host.
注意
裡面外面的mpi版本必須一模一樣
:::
**slurm配singularity容器**
```
$ cat my_job.sh
#!/bin/bash
#SBATCH --job-name singularity-mpi
#SBATCH -N $NNODES # total number of nodes
#SBATCH --time=00:05:00 # Max execution time
mpirun -n $NP singularity exec /var/nfsshare/gvallee/mpich.sif /opt/mpitest
```
其中$NNODES為指定節點數 $NP為指定MPI processes數(=slots)
### bind model
就是讓容器掛載外面的mpi使用，通常適用於容器內無Mpi
:::info
workflow
1.Know where the MPI implementation on the host is installed.
2.Mount/bind it into the container in a location where the system will be able to find libraries and binaries.
編譯軟體的MPI必須和外面的MPI一樣，需知道外面MPI的位置
:::
e.g.
```
#需有--bind指令
$ mpirun -n <NUMBER_OF_RANKS> singularity exec --bind <PATH/TO/HOST/MPI/DIRECTORY>:<PATH/IN/CONTAINER> <PATH/TO/MY/IMAGE> </PATH/TO/BINARY/WITHIN/CONTAINER>
```
## 問題一覽
[在singularity遇到root home的問題](https://hackmd.io/magj1td5RIiqhGLoK2avmw#317)
## singularity 連接外面的infiniband的用法
參考這篇
[Access to /dev/infiniband from user space](https://ask.cyberinfrastructure.org/t/access-to-dev-infiniband-from-user-space/854/3)
要看ib的library裝在哪
**<font color=f100>最主要是/etc/libibverbs.d</font>**
```shell=
# 連接到pbslib 跟 ib lib
# NSCC dgx
singularity exec --bind /opt/pbs,/usr/bin/ibx_devinfo,/usr/lib,/etc/libibverbs.d ubuntu_latest.sif ibv_devinfo
```
可以知道預設是裝在/usr
/usr/lib裡面的東西(可以發現有一些ib-lib)
一些需要的lib的地方及名稱
```
#直接bind /opt/pbs
/opt/pbs/lib/lib*.so*
/usr/lib/libosmcomp.so.3*
/usr/lib/libmlx*.so*
/usr/lib/libib*.so*
/usr/lib/librdmacm.so*
/usr/lib/x86_64-linux-gnu/libnl-route-3.so.200*
```
查看有連到哪個lib
```
apacsc19@dgx4106:~$ ldd /usr/bin/ibv_devinfo
linux-vdso.so.1 (0x00007ffef8bd3000)
libibverbs.so.1 => /usr/lib/libibverbs.so.1 (0x000014a31b407000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014a31b016000)
libnl-route-3.so.200 => /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200 (0x000014a31ada1000)
libnl-3.so.200 => /lib/x86_64-linux-gnu/libnl-3.so.200 (0x000014a31ab81000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x000014a31a962000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x000014a31a75e000)
/lib64/ld-linux-x86-64.so.2 (0x000014a31b826000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014a31a3c0000)
```
## singularity active with infiniband runscript for NSCC
```
#!/bin/bash
ib=/etc/libibverbs.d
for lib in /opt/pbs/lib/lib*.so* /usr/lib/libosmcomp.so.3* /usr/lib/libmlx*.so* /usr/lib/libib*.so* /usr/lib/librdmacm.so* /usr/lib/x86_64-linux-gnu/libnl-route-3.so.200* /lib/x86_64-linux-gnu/libnl-3.so.200 /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu/libdl.so.2 /lib64/ld-linux-x86-64.so.2 /lib/x86_64-linux-gnu/libm.so.6 /usr/lib/libopensm.so*;do
ib="$lib:/usr/lib/$(basename $lib),$ib"
done
singularity exec --bind /opt/pbs,/usr/bin/ibv_devinfo,/usr/lib/libibverbs,$ib /home/users/industry/ai-hpc/apacsc19/scratch/dgx/officialdist ibv_devinfo
```
Sign in with Wallet
Connect another wallet