# 2020-03-30 Meeting OPS - Server Resolve

* OPS Maintenance Procedure definieren
* ausgeführte Befehle (nicht in dieser Reihenfolge):
* `df -h`
* `du -hs /var/lib/docker`
* `docker system prune --all --force`
* `du -h --max-depth=1 /var/lib/docker`
* `docker image prune --all`
* `docker volume ls -qf dangling=true | xargs -r docker volume rm`
* `sudo find /var/lib/docker/containers/ -type f -name "*.log" -delete`

* to try
* `sudo docker rm -v $(sudo docker ps -a -q -f status=exited)`
* `sudo docker rmi -f $(sudo docker images -f "dangling=true" -q)`
* `docker volume ls -qf dangling=true | xargs -r docker volume rm`

## References:
[https://lebkowski.name/docker-volumes/](https://lebkowski.name/docker-volumes/)

## Analyse:
* Vermutlich Problem mit Docker und Logs
* Temporäre Lösung: Ordner verschieben zu /data/docker (wurde bei Prod2 schon gemacht)
* Parameter dranhängen:
* Keine langfristige Lösung: /data/docker bei Prod2 schon 94GB groß

Alex wird bei seinem Kollegen nachfragen, ob/wie der das gelöst hat

## Docker Verzeichnis verschieben
- Docker stoppen `systemctl stop docker`
- Docker Verzeichnis kopieren`sudo rsync -aP /var/lib/docker/ /data/docker`
- Data-Directory anpassen `/lib/systemd/system/docker.service`:
Before: `ExecStart=/usr/bin/dockerd -H fd://`
After: `ExecStart=/usr/bin/dockerd -g /data/docker/ -H fd://
`
- Daemon neuladen `systemctl daemon-reload`
- Docker wieder starten `systemctl start docker`
- Ordner löschen oder verschieben `sudo mv /var/lib/docker/ /data/docker_backup`
- Alternativ: `sudo rm /var/lib/docker`
- Ergebnis:

test2@tumwlrt-move-ii-test2:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.9G     0  7.9G   0% /dev
tmpfs           1.6G  150M  1.5G  10% /run
/dev/sda1        20G  2.9G   18G  15% /
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sdb        250G  8.6G  242G   4% /data
tmpfs           1.6G     0  1.6G   0% /run/user/1000

Source: https://www.guguweb.com/2019/02/07/how-to-move-docker-data-directory-to-another-location-on-ubuntu/

## Logs zur Analyse

test2@tumwlrt-move-ii-test2:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.9G     0  7.9G   0% /dev
tmpfs           1.6G  150M  1.5G  10% /run
/dev/sda1        20G   20G   52M 100% /
tmpfs           7.9G     0  7.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sdb        250G  452M  250G   1% /data
tmpfs           1.6G     0  1.6G   0% /run/user/1000
test2@tumwlrt-move-ii-test2:~$ sudo docker system prune -af
sudo: unable to write to /var/lib/sudo/ts/test2: No space left on device
Total reclaimed space: 0B
test2@tumwlrt-move-ii-test2:~$ sudo du -h --max-depth=1 /var/lib/docker
sudo: unable to write to /var/lib/sudo/ts/test2: No space left on device
14G     /var/lib/docker/containers
0       /var/lib/docker/plugins
15G     /var/lib/docker/overlay2
15M     /var/lib/docker/image
52M     /var/lib/docker/volumes
0       /var/lib/docker/trust
144K    /var/lib/docker/network
0       /var/lib/docker/swarm
16K     /var/lib/docker/builder
56K     /var/lib/docker/buildkit
824K    /var/lib/docker/containerd
0       /var/lib/docker/tmp
0       /var/lib/docker/runtimes
28G     /var/lib/docker
test2@tumwlrt-move-ii-test2:~$ docker ps
CONTAINER ID        IMAGE               COMMAND                   CREATED             STATUS              PORTS                    NAMES
0624a97d14cc        1b8ebfb3639b        "/usr/local/bin/mvn-…"    4 months ago        Up 7 weeks          0.0.0.0:8086->8080/tcp   ops_mcc_adcs
220363f1d5ac        4b81b5bdbf11        "docker-entrypoint.s…"    4 months ago        Up 7 weeks          0.0.0.0:8080->443/tcp    ops_mcc_frontend
18fdffe9097a        2536f89f9864        "/usr/local/bin/mvn-…"    4 months ago        Up 7 weeks          0.0.0.0:8084->8080/tcp   ops_mcc_pl
48e65f704816        ops/ops_resq        "/bin/sh -c 'echo \"p…"   5 months ago        Up 7 weeks                                   ops_resq_raw
cba1b9fe8c55        ops/ops_resq        "/bin/sh -c 'echo \"p…"   5 months ago        Up 7 weeks                                   ops_resq_5
c21da9a632be        ops/ops_resq        "/bin/sh -c 'echo \"p…"   5 months ago        Up 7 weeks                                   ops_resq_1
1fdbe9895ab5        dd98af0c774e        "/usr/local/bin/mvn-…"    8 months ago        Up 7 weeks          0.0.0.0:8083->8080/tcp   ops_mcc_eps
9c53b6786566        dffb833b14a9        "/usr/local/bin/mvn-…"    8 months ago        Up 7 weeks          0.0.0.0:8082->8080/tcp   ops_mcc_thm
3b81e518a221        ops/mcc_health      "/usr/local/bin/mvn-…"    9 months ago        Up 7 weeks          0.0.0.0:8088->8080/tcp   ops_mcc_health
db262a4b82cc        8c02b5b821a3        "/usr/local/bin/mvn-…"    9 months ago        Up 7 weeks          0.0.0.0:8085->8080/tcp   ops_mcc_logbook
df889224fab0        c20956d4f7bb        "/usr/local/bin/mvn-…"    9 months ago        Up 7 weeks          0.0.0.0:8089->8080/tcp   ops_mcc_horst
a6a7b2b8973a        9b8becdb15ee        "/usr/local/bin/mvn-…"    9 months ago        Up 7 weeks          0.0.0.0:8091->8080/tcp   ops_mcc_commanding
70c1f7fbab35        0476ec5972ec        "/usr/local/bin/mvn-…"    9 months ago        Up 7 weeks          0.0.0.0:8087->8080/tcp   ops_mcc_com
02ecd5881821        04a9b790653b        "/usr/local/bin/mvn-…"    9 months ago        Up 7 weeks          0.0.0.0:8081->8080/tcp   ops_mcc_cdh
test2@tumwlrt-move-ii-test2:~$ docker image ls
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
ops/mcc_adcs         latest              1b8ebfb3639b        4 months ago        641MB
ops/mcc_frontend     latest              4b81b5bdbf11        4 months ago        1.44GB
ops/mcc_pl           latest              2536f89f9864        4 months ago        642MB
ops/ops_resq         latest              d9f00f2b3160        5 months ago        740MB
ops/mcc_eps          latest              dd98af0c774e        8 months ago        641MB
ops/mcc_thm          latest              dffb833b14a9        8 months ago        643MB
ops/mcc_health       latest              55333c04141b        9 months ago        640MB
ops/mcc_logbook      latest              8c02b5b821a3        9 months ago        641MB
ops/mcc_horst        latest              c20956d4f7bb        9 months ago        642MB
ops/mcc_commanding   latest              9b8becdb15ee        9 months ago        656MB
ops/mcc_com          latest              0476ec5972ec        9 months ago        641MB
ops/mcc_cdh          latest              04a9b790653b        9 months ago        643MB
test2@tumwlrt-move-ii-test2:~$ docker image prune --all
WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B

prod2@tumwlrt-move-ii-prod2:~$ sudo du /data/docker -sh
94G     /data/docker