---
# System prepended metadata

title: GusLab Cluster Manual

---

# GusLab Cluster Manual

## Networking

| HOSTNAME |      IP ADDRESS       | ZEROTIER VPN |
| :----: | :-----------: | --- |
|   amax-59    | 121.48.165.31 | 192.168.192.59 | 
|   amax-58    | 121.48.165.32 | 192.168.192.58 |
|   amax-57    | 121.48.165.30 | 192.168.192.57 |


### Zerotier

Zerotier VPN configured at system boot for every server, available commands are (for admin)
```shell
sudo zerotier-cli status
sudo service zerotier-one restart
```

:::success
Join ZeroTier network (for clients) [documents](https://zerotier.atlassian.net/wiki/spaces/SD/pages/6848513/Join+a+Network)
**Network ID**: c7c8172af1c92117
> #### Step 1
> Request for join from your client
> ```shell
> zerotier-cli join c7c8172af1c92117
> ```
> #### Step 2
> Contact ZeroTier network administrator (huze@sas.upenn.edu) for approval
> [color=blue]
:::

### NFS Mount

configured at system boot in `/etc/fstab`, `/data` directory share among all servers
```
...
# mount nfs shared directories at boot
121.48.165.30:/data     /data_57        nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0
121.48.165.31:/data     /data_59        nfs auto,nofail,noatime,nolock,intr,tcp,actimeo=1800 0 0
```

### Firewall

in `/etc/iptables/rules.v4`
```
# Generated by iptables-save v1.6.1 on Mon Jul  8 11:30:04 2019
*mangle
:PREROUTING ACCEPT [566822936:2875520714280]
:INPUT ACCEPT [547626555:2866334989910]
:FORWARD ACCEPT [19138076:9180119820]
:OUTPUT ACCEPT [502099743:767408441130]
:POSTROUTING ACCEPT [521239912:776589091473]
:SHADOWSOCKS - [0:0]
COMMIT
# Completed on Mon Jul  8 11:30:04 2019
# Generated by iptables-save v1.6.1 on Mon Jul  8 11:30:04 2019
*nat
:PREROUTING ACCEPT [42:3418]
:INPUT ACCEPT [3:168]
:OUTPUT ACCEPT [359:24033]
:POSTROUTING ACCEPT [338:22857]
:DOCKER - [0:0]
:SHADOWSOCKS - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.19.0.0/16 ! -o br-ab16ed50e605 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o br-8f238847b62d -j MASQUERADE
-A POSTROUTING -s 172.19.0.3/32 -d 172.19.0.3/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.18.0.2/32 -d 172.18.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i br-ab16ed50e605 -j RETURN
-A DOCKER -i br-8f238847b62d -j RETURN
-A DOCKER -d 127.0.0.1/32 ! -i br-ab16ed50e605 -p tcp -m tcp --dport 8001 -j DNAT --to-destination 172.19.0.3:80
-A DOCKER -d 127.0.0.1/32 ! -i docker0 -p tcp -m tcp --dport 8002 -j DNAT --to-destination 172.17.0.2:80
-A DOCKER ! -i br-8f238847b62d -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.18.0.2:80
-A DOCKER ! -i br-8f238847b62d -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.18.0.2:80
COMMIT
# Completed on Mon Jul  8 11:30:04 2019
# Generated by iptables-save v1.6.1 on Mon Jul  8 11:30:04 2019
*filter
:INPUT ACCEPT [9:549]
:FORWARD DROP [21:1008]
:OUTPUT ACCEPT [1732:564833]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A INPUT -s 121.48.165.0/24 -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth0 -p udp -m udp --dport 9993 -j ACCEPT
-A INPUT -i eth0 -j REJECT --reject-with icmp-port-unreachable
-A OUTPUT -p udp -m udp --dport 9993 -j ACCEPT
-A DOCKER -d 172.18.0.2/32 ! -i br-8f238847b62d -o br-8f238847b62d -p tcp -m tcp --dport 80 -j ACCEPT
COMMIT
# Completed on Mon Jul  8 11:30:04 2019
```

in`/etc/iptables/rules.v6`
```
# Generated by ip6tables-save v1.6.1 on Mon Jul  8 11:29:54 2019
*nat
:PREROUTING ACCEPT [2739469:218982750]
:INPUT ACCEPT [2738035:218797929]
:OUTPUT ACCEPT [5819218:481317710]
:POSTROUTING ACCEPT [5781821:474556130]
-A POSTROUTING -s fd42:f5dd:9d0a:3db8::/64 ! -d fd42:f5dd:9d0a:3db8::/64 -m comment --comment "generated for LXD network lxdbr0" -j MASQUERADE
COMMIT
# Completed on Mon Jul  8 11:29:54 2019
# Generated by ip6tables-save v1.6.1 on Mon Jul  8 11:29:54 2019
*filter
:INPUT ACCEPT [776:99365]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [517:93613]
-A INPUT -i lxdbr0 -p tcp -m tcp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -i lxdbr0 -p udp -m udp --dport 547 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth0 -p udp -m udp --dport 9993 -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth0 -p udp -m udp --dport 9993 -j ACCEPT
-A INPUT -i eth0 -j REJECT --reject-with icmp6-port-unreachable
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth0 -p udp -m udp --dport 9993 -j ACCEPT
-A INPUT -i eth0 -j REJECT --reject-with icmp6-port-unreachable
-A FORWARD -o lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A FORWARD -i lxdbr0 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p tcp -m tcp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 53 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -o lxdbr0 -p udp -m udp --sport 547 -m comment --comment "generated for LXD network lxdbr0" -j ACCEPT
-A OUTPUT -p udp -m udp --dport 9993 -j ACCEPT
-A OUTPUT -p udp -m udp --dport 9993 -j ACCEPT
-A OUTPUT -p udp -m udp --dport 9993 -j ACCEPT
COMMIT
# Completed on Mon Jul  8 11:29:54 2019
```

:::info
Direct IP access of any other port except 22 is not allowed for security concern, however, access for all ports are granted via **ZeroTier VPN**.

##### Example:
start your jupyter notebook with **ZeroTier VPN** IP address
```shell
source activate py3
jupyter notebook --ip=192.168.192.57 --port=8888 --NotebookApp.token=your_token
```
then access it from your browser `http://192.168.192.57:8888`
:::

## Disk and Files

### Mounted Disks
HCP (12 disks in total) mounted at `/data_59/mnt` and `/data_58/mnt`

### Disk Quota
The size of `/home` directory for all non-deamon user has been hard limited to **5G** via `quota`, exceeding this limit will result in a failure in your process thats reads: 
:::danger
Disk quota exceeded.
:::

Move large files to `/data` like
```shell
mv ~/.conda /data/your_user_name/.conda
ln -s /data/your_user_name/.conda ~/.conda
```

## Softwares

### Anaconda3
Anaconda3 installed at `/usr/local/anaconda3` and added to PATH.

create your env under your `/home` or `/data` directory or it might be removed by admin
```shell
conda create -p /path/to/your/envs/env_name python=3
```

### Virtualenv
Virtualenv is also available for creating Python envs (not recommended)

### FreeSurfer
FreeSurfer is **not** added to PATH, do it manually or add to your `~/.bashrc`
```shell
export FREESURFER_HOME=/usr/local/freesurfer
source $FREESURFER_HOME/SetUpFreeSurfer.sh
```

### FSL
FSL installed at `/usr/local/fsl/` and added to PATH, ues with a **X11 Forwarding** terminal (MobaXterm in Windows)
```shell
fsl
```

### MRtrix3
```shell
# MRtrix3 PATH automatically generated by set_path script - do NOT modify:
export PATH="/data_59/usr/mrtrix3/bin:$PATH"
```

### Matlab
Matlab installed at `/data_59/usr/MATLAB` and added to PATH, ues with a **X11 Forwarding** terminal (MobaXterm in Windows)
```shell
matlab
```

