S3 Backup Integration

The backup container can be used to backup a Postgres, MySQL, or MongoDB database. It will backup the database to a PVC, usually in /backups.

As parto of this project, I want to integrate the ability to move and store backups to an S3 compatible object storage system.

Current Workflow

The current workflow for the backup container is:

1. Backup the database;
2. Restore the backup;
3. Rotate the backup as per the retention policy.

In this scenario the a number of backups are stored locally on a Kubernets PVC. When rotated the oldest backup is removed from the PVC.

This is the current directory structure of a running backup:

bash-4.2$ pwd
/backups
bash-4.2$ ls
daily  weekly
bash-4.2$ ls -lah weekly/
total 12K
drwxrwxrwx. 3 1004440000 root 4.0K Mar 19 01:00 .
drwxrwxrwx. 4 root       root 4.0K Oct 11 07:28 ..
drwxrwxrwx. 2 1004440000 root 4.0K Mar 19 01:00 2023-03-19
bash-4.2$ ls -lah weekly/2023-03-19/
total 37M
drwxrwxrwx. 2 1004440000 root 4.0K Mar 19 01:00 .
drwxrwxrwx. 3 1004440000 root 4.0K Mar 19 01:00 ..
-rw-r--r--. 1 1004440000 root  37M Mar 19 01:00 aries-mediator-db-aries-mediator-agent-wallet_2023-03-19_01-00-00.sql.gz
bash-4.2$ ls -lah daily/
total 16K
drwxrwxrwx. 4 1004440000 root 4.0K Mar 22 01:00 .
drwxrwxrwx. 4 root       root 4.0K Oct 11 07:28 ..
drwxrwxrwx. 2 1004440000 root 4.0K Mar 21 01:00 2023-03-21
drwxrwxrwx. 2 1004440000 root 4.0K Mar 22 01:00 2023-03-22
bash-4.2$ ls -lah daily/2023-03-21/
total 37M
drwxrwxrwx. 2 1004440000 root 4.0K Mar 21 01:00 .
drwxrwxrwx. 4 1004440000 root 4.0K Mar 22 01:00 ..
-rw-r--r--. 1 1004440000 root  37M Mar 21 01:00 aries-mediator-db-aries-mediator-agent-wallet_2023-03-21_01-00-00.sql.gz

Expected Workflow

The desired worklow for the backup container is:

1. Backup the database;
2. Restore the backup;
3. Copy the backup to an S3 compatible storage
4. Rotate the backup as per the retention policy.

In this scenario the there is one backup stored locally on PVC. This backup is the most recent. All other backups are stored in an S3 compatible storage.

The backups are rotated as per the retention policy. This means the oldest backup is removed from the S3 bucket.

5. If multiple .dump or .tar.gz files exist then all of them should be moved to the bucket. Once moved the local copy should be deleted. Only the latest backup should remain on the local /backups file system.

Technical Requirements

• Any S3 related functions should be added to this file and called as needed:

https://github.com/fullboar/backup/blob/main/docker/backup.utils

• The following environment variables should be added. When they the exist the S3 functionality should be enabled:

S3_HOST_URL
S3_BUCKET
S3_ACCESS_KEY
S3_SECRET_ACCESS_KEY

• We don't need to mirror the directory structure above in the s3 object storeage because the file name contains the date already. We can just use

bucket/aries-mediator-db-aries-mediator-agent-wallet_2023-03-19_01-00-00.sql.gz
bucket/aries-mediator-db-aries-mediator-agent-wallet_2023-03-20_01-00-00.sql.gz
bucket/aries-mediator-db-aries-mediator-agent-wallet_2023-03-21_01-00-00.sql.gz