# GusLab Cluster Manual
## Networking
| HOSTNAME | IP ADDRESS | ZEROTIER VPN |
| :----: | :-----------: | --- |
| amax-59 | 121.48.165.31 | 192.168.192.59 |
| amax-58 | 121.48.165.32 | 192.168.192.58 |
| amax-57 | 121.48.165.30 | 192.168.192.57 |
### Zerotier
Zerotier VPN configured at system boot for every server, available commands are (for admin)
```shell
sudo zerotier-cli status
sudo service zerotier-one restart
```
:::success
Join ZeroTier network (for clients) [documents](https://zerotier.atlassian.net/wiki/spaces/SD/pages/6848513/Join+a+Network)
**Network ID**: c7c8172af1c92117
> #### Step 1
> Request for join from your client
> ```shell
> zerotier-cli join c7c8172af1c92117
> ```
> #### Step 2
> Contact ZeroTier network administrator (huze@sas.upenn.edu) for approval
> [color=blue]
:::
### NFS Mount
configured at system boot in `/etc/fstab`, `/data` directory share among all servers
```
...
# mount nfs shared directories at boot
121.48.165.30:/data /data_57 nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0
121.48.165.31:/data /data_59 nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0
```
### Firewall
in `/etc/iptables/rules.v4`
```
# Generated by iptables-save v1.6.1 on Mon Jul 8 11:30:04 2019
*mangle
:PREROUTING ACCEPT [566822936:2875520714280]
:INPUT ACCEPT [547626555:2866334989910]
:FORWARD ACCEPT [19138076:9180119820]
:OUTPUT ACCEPT [502099743:767408441130]
:POSTROUTING ACCEPT [521239912:776589091473]
:SHADOWSOCKS - [0:0]
COMMIT
# Completed on Mon Jul 8 11:30:04 2019
# Generated by iptables-save v1.6.1 on Mon Jul 8 11:30:04 2019
*nat
:PREROUTING ACCEPT [42:3418]
:INPUT ACCEPT [3:168]
:OUTPUT ACCEPT [359:24033]
:POSTROUTING ACCEPT [338:22857]
:DOCKER - [0:0]
:SHADOWSOCKS - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.19.0.0/16 ! -o br-ab16ed50e605 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o br-8f238847b62d -j MASQUERADE
-A POSTROUTING -s 172.19.0.3/32 -d 172.19.0.3/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.18.0.2/32 -d 172.18.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-ab16ed50e605 -j RETURN
-A DOCKER -i br-8f238847b62d -j RETURN
-A DOCKER -d 127.0.0.1/32 ! -i br-ab16ed50e605 -p tcp -m tcp --dport 8001 -j DNAT --to-destination 172.19.0.3:80
-A DOCKER -d 127.0.0.1/32 ! -i docker0 -p tcp -m tcp --dport 8002 -j DNAT --to-destination 172.17.0.2:80
-A DOCKER ! -i br-8f238847b62d -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.18.0.2:80
-A DOCKER ! -i br-8f238847b62d -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.18.0.2:80
COMMIT
# Completed on Mon Jul 8 11:30:04 2019
# Generated by iptables-save v1.6.1 on Mon Jul 8 11:30:04 2019
*filter
:INPUT ACCEPT [9:549]
:FORWARD DROP [21:1008]
:OUTPUT ACCEPT [1732:564833]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A INPUT -s 121.48.165.0/24 -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth0 -p udp -m udp --dport 9993 -j ACCEPT
-A INPUT -i eth0 -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -p udp -m udp --dport 9993 -j ACCEPT
-A DOCKER -d 172.18.0.2/32 ! -i br-8f238847b62d -o br-8f238847b62d -p tcp -m tcp --dport 80 -j ACCEPT
COMMIT
# Completed on Mon Jul 8 11:30:04 2019
```
in`/etc/iptables/rules.v6`
```
# Generated by ip6tables-save v1.6.1 on Mon Jul 8 11:29:54 2019
*nat
:PREROUTING ACCEPT [2739469:218982750]
:INPUT ACCEPT [2738035:218797929]
:OUTPUT ACCEPT [5819218:481317710]
:POSTROUTING ACCEPT [5781821:474556130]
-A POSTROUTING -s fd42:f5dd:9d0a:3db8::/64 ! -d fd42:f5dd:9d0a:3db8::/64 -m comment --comment "generated for LXD network lxdbr0" -j MASQUERADE
COMMIT
# Completed on Mon Jul 8 11:29:54 2019
# Generated by ip6tables-save v1.6.1 on Mon Jul 8 11:29:54 2019
*filter
:INPUT ACCEPT [776:99365]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [517:93613]
-A INPUT -i lxdbr0 -p tcp -m tcp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 547 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth0 -p udp -m udp --dport 9993 -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth0 -p udp -m udp --dport 9993 -j ACCEPT
-A INPUT -i eth0 -j REJECT --reject-with icmp6-port-unreachable
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth0 -p udp -m udp --dport 9993 -j ACCEPT
-A INPUT -i eth0 -j REJECT --reject-with icmp6-port-unreachable
-A FORWARD -o lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p tcp -m tcp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 547 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -p udp -m udp --dport 9993 -j ACCEPT
-A OUTPUT -p udp -m udp --dport 9993 -j ACCEPT
-A OUTPUT -p udp -m udp --dport 9993 -j ACCEPT
COMMIT
# Completed on Mon Jul 8 11:29:54 2019
```
:::info
Direct IP access of any other port except 22 is not allowed for security concern, however, access for all ports are granted via **ZeroTier VPN**.
##### Example:
start your jupyter notebook with **ZeroTier VPN** IP address
```shell
source activate py3
jupyter notebook --ip=192.168.192.57 --port=8888 --NotebookApp.token=your_token
```
then access it from your browser `http://192.168.192.57:8888`
:::
## Disk and Files
### Mounted Disks
HCP (12 disks in total) mounted at `/data_59/mnt` and `/data_58/mnt`
### Disk Quota
The size of `/home` directory for all non-deamon user has been hard limited to **5G** via `quota`, exceeding this limit will result in a failure in your process thats reads:
:::danger
Disk quota exceeded.
:::
Move large files to `/data` like
```shell
mv ~/.conda /data/your_user_name/.conda
ln -s /data/your_user_name/.conda ~/.conda
```
## Softwares
### Anaconda3
Anaconda3 installed at `/usr/local/anaconda3` and added to PATH.
create your env under your `/home` or `/data` directory or it might be removed by admin
```shell
conda create -p /path/to/your/envs/env_name python=3
```
### Virtualenv
Virtualenv is also available for creating Python envs (not recommended)
### FreeSurfer
FreeSurfer is **not** added to PATH, do it manually or add to your `~/.bashrc`
```shell
export FREESURFER_HOME=/usr/local/freesurfer
source $FREESURFER_HOME/SetUpFreeSurfer.sh
```
### FSL
FSL installed at `/usr/local/fsl/` and added to PATH, ues with a **X11 Forwarding** terminal (MobaXterm in Windows)
```shell
fsl
```
### MRtrix3
```shell
# MRtrix3 PATH automatically generated by set_path script - do NOT modify:
export PATH="/data_59/usr/mrtrix3/bin:$PATH"
```
### Matlab
Matlab installed at `/data_59/usr/MATLAB` and added to PATH, ues with a **X11 Forwarding** terminal (MobaXterm in Windows)
```shell
matlab
```
Sign in with Wallet
Connect another wallet