---
# System prepended metadata

title: WTF is happening on gitlab

---

# WTF is happening on gitlab


As of Joels 1st attempt to push to gitlab we encoutered the git lab error

```
joel.beer@BRIDEVMAC3  ~/Sites/sites/ththjims1803/current   update/screwfix_taleo_scrape ●      git push --set-upstream origin update/screwfix_taleo_scrape
Counting objects: 22, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (22/22), done.
Writing objects: 100% (22/22), 7.42 KiB | 1.48 MiB/s, done.
Total 22 (delta 14), reused 0 (delta 0)
error: remote unpack failed: unable to create temporary object directory
To gitlab.thirtythreebuild.co.uk:thirtythree/ththjims1803.git
 ! [remote rejected] update/screwfix_taleo_scrape -> update/screwfix_taleo_scrape (unpacker error)
error: failed to push some refs to 'git@gitlab.thirtythreebuild.co.uk:thirtythree/ththjims1803.git'
```


Initial research gave us the following

```
error: remote unpack failed: unable to create temporary object directory
    indicates that the Git repository at the     server (not your Git) is out of space, or running into similar server problems,1 or installed incorrectly.
```

## Steps taken
 - metrics cheked
     - build low on memory but useble 
 - lewis ssh- df -h and top found it has memory
 - Rich found sdc data is full on webbuild

```
[10:53:41 root@gitlab01 backup]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        30G  6.1G   24G  21% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G   12K  3.9G   1% /dev/shm
tmpfs           3.9G  417M  3.5G  11% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1       497M   79M  418M  16% /boot
/dev/sdb1        16G   45M   15G   1% /mnt/resource
/dev/sdc        160G  160G   24K 100% /data
tmpfs           797M     0  797M   0% /run/user/1002
tmpfs           797M     0  797M   0% /run/user/1003
```

- 12.03 - appears backups are not being removed. We keep 7 days but are keeping from Sept 6 to present.
- 12.05 - running prune-backups by hand to see if it runs.
- 12.07 - prunk-backups completed. It found all files older than 7 days on server but not in S3 so it did not remove them.
- 12.11 Deep dive into prune back up script and grabbing the S3 back up check command and running it manually. it returned nothing. 
- Current thinking is the path S3 uses is incorrect.
- 12.13 current plan is to manual remove old git lab backups for the inital fix. Then to fix the prune-backup script to point to the correct location on S3
- Last night backup was a partial. So we need to remove old backups and get a full recent backup
- 12.19 - backing up gitlab manually 
- 12.22 - Joel & Lewis test git push with success 


# Summary 

Git lab backups were not being removed so the backup folder filled up. Clearing backups older then 7 days gave up more room and enabled basic git usage again. 
Task in trello created to fix `prune-backups` script so it looks at the right location for the backups.