# Gitlab Server Maintenance ###### tags: `moxa-linux` `tutorial` `git` `gitlab` [TOC] ## Introduction As gitlab server maintainer, this note would keep gitlab server installation guide and some tips for debugging common gitlab issue. ## Getting Started Before start setting up gitlab server, following items need to be prepared. ### Linux Image Prepare linux image file with extension `.iso` and make a bootable USB drive. I choose `Debian 10.3.0 netinst` as my target OS. Here is a image download path, [debian.org/debian-cd/current/amd64/iso-cd/debian-10.3.0-amd64-netinst.iso](https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-10.3.0-amd64-netinst.iso). Account will be set as `admin`, and password will be `89191230`, so do the root password. ### Packages Preparation Set up root `PATH` first. Edit `/root/.bashrc` (using root privileges, of course) and add the following lines, ```shell su root (passwd: 89191230) vi /root/.bashrc ``` :::info ==export PATH=/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin== ::: Refer to [WiFi issues on Debian 10 - Reddit](https://www.reddit.com/r/debian/comments/ehcpab/debian_10_wifi_is_not_showing_up_anywhere_even/). Before update and upgrade `apt source-list`, edit `/etc/apt/sources.list` and add following lines. ```shell vi /etc/apt/sources.list ``` :::info \# \# deb cdrom:[Official Debian GNU/Linux 10.3.0 \_Buster\_ - Official amd64 NETINST 20200208-12:07]/ buster main #deb cdrom:[Official Debian GNU/Linux 10.3.0 \_Buster\_ - Official amd64 NETINST 20200208-12:07]/ buster main deb http://deb.debian.org/debian/ buster main ==contrib== ==non-free== deb-src http://deb.debian.org/debian/ buster main ==contrib== ==non-free== deb http://security.debian.org/debian-security buster/updates main ==contrib== ==non-free== deb-src http://security.debian.org/debian-security buster/updates main ==contrib== ==non-free== \# buster-updates, previously known as 'volatile' deb http://deb.debian.org/debian/ buster-updates main ==contrib== ==non-free== deb-src http://deb.debian.org/debian/ buster-updates main ==contrib== ==non-free== \# This system was installed using small removable media \# (e.g. netinst, live or single CD). The matching "deb cdrom" \# entries were disabled at the end of the installation process. \# For information about how to configure apt package sources, \# see the sources.list(5) manual. ::: Do `apt source-list` updation, and install all necessary packages. ```shell sudo apt-get update -y sudo apt-get upgrade -y sudo apt-get install vim net-tools -y ``` ### Mount Data Drive No matter hardware configuration of your server, system and data drives should be seperated. If system and data drives are not in the same hard drive, then data drive should be mounted at boot time. In Debian 10, `/etc/fstab` notes a list of filesystems to be mounted at boot time. Before modify `/etc/fstab`, check UUID of data hard drive. ```shell ls -lh /dev/disk/by-uuid/ ``` :::info total 0 lrwxrwxrwx 1 root root 10 Apr 7 12:15 245D-E77E -> ../../sda1 lrwxrwxrwx 1 root root 10 Apr 7 12:15 5fc40e0e-620c-4456-a91c-a08a9bfa9a21 -> ../../sda2 lrwxrwxrwx 1 root root 9 Apr 7 12:15 ==8a6c9bc3-22ed-430a-9fbc-93a717cc4cd6== -> ../../sdb lrwxrwxrwx 1 root root 10 Apr 7 12:15 e4a227b9-7243-432e-bf1b-7f5395838e52 -> ../../sda3 ::: Copy the UUID and edit `/etc/fstab`, ```shell su (passwd: 89191230) vim /etc/fstab ``` :::info \# /etc/fstab: static file system information. \# \# Use 'blkid' to print the universally unique identifier for a \# device; this may be used with UUID= as a more robust way to name devices \# that works even if disks are added and removed. See fstab(5). \# \# \<file system> \<mount point> \<type> \<options> \<dump> \<pass> \# / was on /dev/sda2 during installation UUID=5fc40e0e-620c-4456-a91c-a08a9bfa9a21 / ext4 errors=remount-ro 0 1 \# /boot/efi was on /dev/sda1 during installation UUID=245D-E77E /boot/efi vfat umask=0077 0 1 ==\# /usr/local was on /dev/sdb during installation UUID=8a6c9bc3-22ed-430a-9fbc-93a717cc4cd6 /usr/local ext4 defaults 0 2== \# swap was on /dev/sda3 during installation UUID=e4a227b9-7243-432e-bf1b-7f5395838e52 none swap sw 0 0 ... ::: Then reboot. Data drive will be mounted at boot time. You can check `/etc/mtab` to check currently mounted filesystems. :::warning In this case, we mount data drive to `/usr/local`. ::: ### Docker Engine Refer to [Get Docker Engine - Community for Debian](https://docs.docker.com/install/linux/docker-ce/debian/), go to path https://download.docker.com/linux/debian/dists/buster/pool/stable/amd64/ and download the file with extension `.deb` for Docker engine. Then install the package. ```shell su root (passwd: 89191230) wget https://download.docker.com/linux/debian/dists/buster/pool/stable/amd64/containerd.io_1.2.6-3_amd64.deb wget https://download.docker.com/linux/debian/dists/buster/pool/stable/amd64/docker-ce-cli_19.03.8~3-0~debian-buster_amd64.deb wget https://download.docker.com/linux/debian/dists/buster/pool/stable/amd64/docker-ce_19.03.8~3-0~debian-buster_amd64.deb dpkg -i containerd.io_1.2.6-3_amd64.deb dpkg -i docker-ce-cli_19.03.8~3-0~debian-buster_amd64.deb dpkg -i docker-ce_19.03.8~3-0~debian-buster_amd64.deb ``` After installation, Docker daemon starts automatically. The `docker` group is created but no users are added to it. You need to use `sudo` to run Docker commands. Continue to [Post-installation steps for Linux](https://docs.docker.com/install/linux/linux-postinstall/) to allow non-privileged users to run Docker commands and for other optional configuration steps. ```shell groupadd docker usermod -aG docker $USER ``` ### docker-compose We used `docker-compose` to bring up gitlab server, which is a tool for defining and running multi-container Docker applications. With `docker-compose`, you use a file `docker-compose.yml` to configure your application’s services. Then, with a single command, you create and start all the services from your configuration. #### Package Install ```shell apt-get install docker-compose -y ``` #### docker-compose.yml Get predefined `docker-compose.yml` from git or offline copy-paste anyway. ```shell git clone ssh://git@mhqnetgitlab.moxa.com:10022/ContinuousIntegration/docker.git cd docker/gitlab cp docker-compose.yml /home/admin/install_document/gitlab ``` ### Start Gitlab Server Go to `docker-compose.yml` path and launch the server. ```shell cd /home/admin/install_document/gitlab docker-compose up -d ``` Following message will be shown, :::info Creating admin_redis_1 ... done Creating admin_postgresql_1 ... done Creating admin_gitlab_1 ... done ::: Gitlab web page will be ready in a while. The link will be `server_ip:10080`, e.g. `10.144.41.26:10080`. Users can login with LDAP account or new account after signing up. :::warning First login password is root password 89191230 ::: ## Restore Backup Image ### Get Backup Image Copy backup image by `scp`, ```shell scp -l 16000 /source/path/of/filename_gitlab_backup.tar /dest/path ``` :::warning `-l` is a necessary argument used to limit the bandwidth, else the bandwidth will be occupied while data transferring. ::: ### Move Image to Specified Path ```shell cd /dest/path mv filename_gitlab_backup.tar /usr/local/gitlab/gitlab/backups ``` :::warning According to `docker-compose.yml`, container volume mapping is `/usr/local/gitlab/gitlab:/home/git/data`. And default backup image restore path is `/home/git/data/backups` inside the container. Hence, need to move image to /usr/local/gitlab/gitlab/backups for image restoring. ::: Refer to [Permission Denied when restoring Backup](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3554), need to change directory ownership before restore the image. ```shell docker exec <container-id> chown git:git /home/git/data/backups/filename_gitlab_backup.tar ``` ### Restore Image Before performing a restore make sure the container is stopped and removed to avoid container name conflicts. ```shell docker-compose stop docker-compose run --rm gitlab app:rake gitlab:backup:restore BACKUP=filename 2>&1 | tee log ``` :::warning It may take a couple of time. In the case of 63 GBytes backup image in DELL PowerEdge R340, it takes about 70 minutes to restore. ::: All data will be checked after backup image is restored properly. ## Backup Scripts Preparation Backup needs to be executed as routine. We use `crontab` to do regular task. Details can refer [Linux crontab tutorial](https://blog.gtwang.org/linux/linux-crontab-cron-job-tutorial-and-examples/). :::warning Backup script content may different in each system. It depends on how and where they store backup image. ::: ## Verify Gitlab Server All set! Now need to verify data in gitlab server is correct or not. A simple way to verify is fetching code from new gitlab server, and directly build on local! e.g. `buildroot` in Moxa, details refer to [Buildroot Environment Setup Tutorial](/9-D4MnPbR3Wg1Hv975tfAw) or [Buildroot User Manual](/XEy82_w6SQmVKdh70ELmmw?both). :::warning You can also use Jenkins build online! ::: After checking out the buildroot git respository from new server, edit `install.sh` to modify source path to new server IP. For example, `10.144.41.26` is new server IP address. ```shell cd /home/moxa git clone ssh://git@10.144.41.26:10022/BuildSystem/MoxaBuild.git gitlab-test cd gitlab-test vim install.sh ``` :::info ~~mhqnetgitlab.moxa.com~~:10022 -> ==10.144.41.26==:10022 ::: Execute `install.sh`. ```shell ./install.sh ``` `.config` will be created after making product config. Edit `.config` and do the replacement as above. Fetch target will be replaced to new gitlab server. Fetch source code from new server and build! ```shell ./moxa/fetch_source.sh make ``` If image is built successfully, congrats! New server setting complete. Now you can move gitlab server to IT IDC. :::warning After move server to IT IDC, remember to modify DNS server mapping to new server IP address. ENJOY! ::: ## Some Tips Common usage on gitlab maintain stuff and debugging. ### Restart Gitlab Server When some exceptions occur, restart server would be the easiest way to recover. ```shell cd /home/admin/install_document/gitlab docker-compose restart ``` or ```shell cd /home/admin/install_document/gitlab docker-compose stop docker ps -a docker rm -f <container-id> docker-compose up -d ``` ### Remove Containers If `docker-compose` command is not working, try `docker` command to remove containers. ```shell docker ps -a docker stop <container-id> && docker rm -f <container-id> ``` ### Restart Docker Daemon If both `docker-compose` and `docker` commands are not working, restart docker service. ```shell su (passwd: 89191230) service docker restart ``` ### Looking For Logs ```shell cd /home/admin/install_document/gitlab docker-compose logs ``` ### Check Disk Usage And Directory Size Backup images occupy lots of disk space, to check disk usage and each directory size, you can try following commands. ```shell df -h du -d <depth> -h <path> ``` ## Reference [sameersbn/docker-gitlab/9.4.4](https://github.com/sameersbn/docker-gitlab/tree/9.4.4) [docker-compose.yml@moxa_gitlab](http://mhqnetgitlab.moxa.com:10080/ContinuousIntegration/docker/tree/master/gitlab)