BeeGFS - HackMD

# BeeGFS ## install ### BeeGFS repo ```bash cat > /etc/yum.repos.d/beegfs_rhel7.repo <<EOF [beegfs] name=BeeGFS 7.1 (rhel7) baseurl=https://www.beegfs.io/release/beegfs_7_1/dists/rhel7 gpgkey=https://www.beegfs.io/release/beegfs_7_1/gpg/RPM-GPG-KEY-beegfs gpgcheck=0 enabled=1 EOF ``` ## 安裝beegfs套件 ### 1. Management server (a1) ```bash yum install -y beegfs-mgmtd beegfs-utils ``` ### 2. Metadata Server (a1 a2 b1 b2) ```bash yum install -y beegfs-meta libbeegfs-ib beegfs-utils ``` ### 3. Storage Server (a1 a2 b1 b2) ```bash yum install -y beegfs-storage libbeegfs-ib beegfs-utils ``` ### 4. Client (all) ```bash yum install -y beegfs-client beegfs-helperd ``` #### 建構ib相關環境 ```bash sed -i 's/^buildArgs=-j8/buildArgs=-j16 BEEGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=\/usr\/src\/ofa_kernel\/default\/include/' /etc/beegfs/beegfs-client-autobuild.conf /etc/init.d/beegfs-client rebuild ``` 若rebuild發生問題, 檢查kernel版本是否正確 ## configure ### 1. Management server (a1) ```bash mkdir /beegfs-mgmtd sed -i 's/storeMgmtdDirectory =/storeMgmtdDirectory = \/beegfs-mgmtd/' /etc/beegfs/beegfs-mgmtd.conf /opt/beegfs/sbin/beegfs-setup-mgmtd -p /beegfs-mgmtd ``` ### 2. Metadata Server (a1 a2 b1 b2) ```bash mkdir /beegfs-meta sed -i 's/sysMgmtdHost =/sysMgmtdHost = a1/' /etc/beegfs/beegfs-meta.conf sed -i 's/storeMetaDirectory =/storeMetaDirectory = \/beegfs-meta/' /etc/beegfs/beegfs-meta.conf /opt/beegfs/sbin/beegfs-setup-meta -p /beegfs-meta -s x -m a1 ``` ### 3. Storage Server (a1 a2 b1 b2) ```bash sed -i 's/sysMgmtdHost =/sysMgmtdHost = a1/' /etc/beegfs/beegfs-storage.conf sed -i 's/storeStorageDirectory =/storeStorageDirectory = \/beegfs-storage/' /etc/beegfs/beegfs-storage.conf ``` ```bash /opt/beegfs/sbin/beegfs-setup-storage -p /beegfs-storage -s x -i x01 -m a1 ``` ### 4. Client (a1 a2 b1 b2) ```bash sed -i 's/sysMgmtdHost =/sysMgmtdHost = a1/' /etc/beegfs/beegfs-client.conf /opt/beegfs/sbin/beegfs-setup-client -m a1 echo "/nfs /etc/beegfs/beegfs-client.conf" > /etc/beegfs/beegfs-mounts.conf ``` ## disable selinux ```bash sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config reboot ``` ## 啟用服務 ### 1. Management server (a1) ```bash systemctl restart beegfs-mgmtd ``` ### 2. Metadata Server (a1 a2 b1 b2) ```bash systemctl restart beegfs-meta ``` ### 3. Storage Server (a1 a2 b1 b2) ```bash systemctl restart beegfs-storage ``` ### 4. helperd (a1 a2 b1 b2) ```bash systemctl restart beegfs-helperd ``` ### 5. client (a1 a2 b1 b2) ```bash systemctl restart beegfs-client ``` ## 查看服務 ### 所有server ```bash beegfs-check-servers ``` ### mgmt service ```bash beegfs-ctl --listnodes --nodetype=mgmt --nicdetails ``` ### storage service ```bash beegfs-ctl --listnodes --nodetype=storage --nicdetails ``` ### metadata service ```bash beegfs-ctl --listnodes --nodetype=meta --nicdetails ``` ### client service ```bash beegfs-ctl --listnodes --nodetype=client --nicdetails ``` ## 刪除node ```bash beegfs-ctl --removenode --nodetype=XXX <node id> ```