# container-copy
###### tags: `Design`

## Introduction

Copy container was a tool initially created to sync tripleo containers between docker and quay.io once docker started to enforce the daily quota of containers. Since then, some solutions were created to avoid these quota, like the container provider job.
The tool initially was supposed to just copy the containers from docker and rdo to quay. Initially was created in python, using podman v2 rest api, however, due to some limitagions (only worked on podman > 2.6) and because it had to have podman rest api running all the time, and doesn't remove the fact that it was required to prune all the containers locally after it finishes. Another sollution was base the code on the skopeo, that is based on the podman api. This way, the copy of containers is today made in golang code, using the same podman api available on podman.

### Why do I care

Quay today is used as alternative to docker, and since the promotion right now doesn't copy the promoted containers to quay.io, this is required to keep everything in sync.

### When would I look here

When something goes wrong, like for example, when the containers tagged as current-tripleo are too old compared to docker and rdo registries.

### What this does

The copy container check for the job the latest successfull job that build the containers for the list of containers builded, than get the hash code for those containers, and copy from registry.trunk.rdoproject.org to quay.io. This happens every two hours via cron job on the toolbox.
At this point, doesn't matter if the container is promoted or not, it will copy whatever is available at the moment.
If the latest job was already coppied, the tool is smart (well, podman is smart enough) to point that the layers already exist, and it will just keep skipping for the next layer, which make the copy quick, and no need to checks if the container already exists on quay or not. So, you don't need to care to check if the container exist or not before copy it.

After that, another cron job, tag the container with the current-tripleo tag. This is not related to podman it self. The tool check the (several different endpoints for each release) api from DLRN to get the latest current-tripleo tag, and using quay.io api, tag each container.

### Where it is

Right now, this is running in our toolbox machine that can be accessed via ssh on 38.102.83.131.

The logs are available on /root/logs locally, and on the web on http://38.102.83.131/quay/ and http://38.102.83.131/quay/tag

The binary and scripts of cronjob are available right now on /root/copy-quay directory, but soon we will have a dedicated ansible role to deploy it.

### How often it runs

As mentioned before, it runs every 2 hours both the copy, and the tag.

## Toolbox

### Where the code is

Right now the code is available on /root/copy-quay on the toolbox, but it will also be available on [ci-config](https://github.com/rdo-infra/ci-config) repository once https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32938 get merged.

### Deployed and how

The code is on /root/copy-quay, together with the shell script with the call for each release. You can edit the go file for changes and run go build to compile it. It might require some packages, but this is already installed in the toolbox machine, and it's also listed in the patch with the ansible task.

### Log structure

There are four different logs, the logs on /root/logs/release.txt (master, wallaby, victorya, etc) are the logs from the tool running and copying containers. There is also the /root/logs/release-report.html that is the report for the latest time the tool was executed and copied container. As you can notice in the report, it shows all the containers that were executed with the date, the hash and the status for each container, if it was coppied successfully or failed.
Also, under /root/logs/tag you have the same structure but for tagging. The copy and tag of containers are splited although it is done by the same tool.

## User workflow - Copy containers
### Container failing to copy

The first step is check the release-report.hml (where release is master, wallaby, etc) and see the latest hash, and the failing container.

After that, search in the /root/logs/release.txt (where release is the same as in release-report.html) and check the reason of failure. If the failure was related to some timeout, you can either manually run the copy tool to copy only the failing container, or wait for the next run and check if it will still fail.
Of course, wait for the next run might copy a different hash since in this time window, a new hash might be built.

Let's get an example, you check http://38.102.83.131/quay/master-report.html and notice that the container openstack-designate-worker with hash ec48e22f7ecb7ab5150d08a942860b22_x86_64 fail.

At this point, the next step is check http://38.102.83.131/quay/master.txt for the failure reason:

time="2021-07-19T12:00:03Z" level=error msg="Failed to copy container image:  Error in copy the image: Error initializing source docker://trunk.registry.rdoproject.org/tripleomaster/openstack-designate-worker:ec48e22f7ecb7ab5150d08a942860b22_x86_64: error pinging docker registry trunk.registry.rdoproject.org: Get \"https://trunk.registry.rdoproject.org/v2/\": dial tcp 38.102.83.107:443: connect: connection refused"
time="2021-07-19T12:00:03Z" level=info msg="Copying image tripleomaster/openstack-designate-worker:ec48e22f7ecb7ab5150d08a942860b22"

This might happen because we reach the quota for api calls on quay, and you can wait a little bit and try again, or run the copy command to copy this particular container, and this particular hash:

* Log into toolbox as root

ssh root@toolbox

* Enter into copy quay directory

cd /root/copy-quay

* Source the credentials

source /etc/quayrc

* Copy the container
Now, the copy container have a lot of options, you can use --help to check each of those:

./copy-quay --help
Copy a container from one location to another

Usage:
  copy-quay [command]

Available Commands:
  copy        Copy images from source to destiny
  help        Help about any command
  tag         Tag images

Flags:
      --debug                   Enable debug output
      --from-namespace string   Namespace of pushed image (default "tripleomaster")
      --hash string             Hash to be pulled/pushed
  -h, --help                    help for copy-quay
      --job string              Job to collect the list of containers
      --pull-registry string    Registry to pull images from (default "trunk.registry.rdoproject.org")
      --push-hash string        Hash to be pulled/pushed
      --push-registry string    Registry to push images to (default "quay.io")
      --release string          Release (default "master")
      --to-namespace string     Namespace of pushed image (default "tripleomaster")
      --token string            Token to use with quay api
      --zuul-api string         Zuul api endpoint (default "https://review.rdoproject.org/zuul/api/")

Use "copy-quay [command] --help" for more information about a command.

To copy only the openstack-designate-worker container with the hash ec48e22f7ecb7ab5150d08a942860b22_x86_64, you will execute the following:

./copy-quay --from-namespace tripleomaster --to-namespace tripleomaster --release master --token $TOKEN --hash ec48e22f7ecb7ab5150d08a942860b22_x86_64 copy openstack-designate-worker

And the output:

Copying blob 61bd84a9108e skipped: already exists  
Copying blob dc0deb7c6c28 skipped: already exists  
Copying blob c697821d25a3 skipped: already exists  
Copying blob 92bf32900741 skipped: already exists  
Copying blob 1615f362c3b9 skipped: already exists  
Copying blob 782b2b55190b [--------------------------------------] 0.0b / 0.0b
Copying config 7a2d56edb1 [======================================] 11.2KiB / 11.2KiB
Writing manifest to image destination
Storing signatures

The $TOKEN is stored in /etc/quayrc

As you can see, you need to parse the --from-namespace and --to-namespace, this are the namespace from quay and rdo, usually is tripleorelease. In the case above, was tripleomaster, and you also need to tell the release, in this case master. The hash you want to copy, otherwise, it will check the latest hash available. then the copy command, and the container name.

If you do not pass the container after the copy command, it will look for the list of containers in a specific job and copy all the containers.

## User workflow - Tagging

Tagging is similar to copy however, instead of the copy command, you will call tag command, it also check the list of containers from the latest successfull run, and the DLRN api for latest hash.

If you want to tag train8 for example (train have two releases, train that is centos 7 release, and train8 that is centos 8 release) you need to run the following

/copy-quay --from-namespace tripleotraincentos8 --to-namespace tripleotraincentos8 --token $TOKEN --release train8 tag

An the output is something similar to this:

INFO[0000] Promoted hash:  2a68d3b9e14d86b57db95487cadce0e2 
INFO[0000] Job:  periodic-tripleo-ci-build-containers-ubi-8-push-train 
ERRO[0001] URL not found: https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push-train/0641964/logs/containers-successfully-built.log 
INFO[0001] Fetching logs in https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push-train/0641964/logs/containers-successfully-built.log 
INFO[0001] Tagging tripleotraincentos8/centos-binary-swift-proxy-server with tag current-tripleo and manifest sha256:9637f9179d7c6a40e79e53414f3fe8e98f194b3d656e8ae17074b95477c1e040 
"Updated"

INFO[0002] Tagging tripleotraincentos8/centos-binary-swift-proxy-server with tag current-tripleo and manifest sha256:9637f9179d7c6a40e79e53414f3fe8e98f194b3d656e8ae17074b95477c1e040 
"Updated"

Where 2a68d3b9e14d86b57db95487cadce0e2 is the promoted hash, that is retrive via DLRN api. You can chec accessing https://trunk.rdoproject.org/api-centos8-train/api/promotions?promote_name=current-tripleo&limit=1 for example.

Then it check for logs of the latest successfull periodic-tripleo-ci-build-containers-ubi-8-push-train job. The ERRO URL not found is fine, since we check for two different logs for the list of containers due the different ways to build the containers with kolla (old releases).

After that, you can see it started to tag the containers on quay.

If you want to tag only a particular container, you can do so passing the container name after the tag command.

You can also force to tag parsing the --force-hash option.

## Troubleshooting

Add workflows here as new different errors appears

### Docker.io containers missing

Recently there was an issue where docker.io containers for current-tripleo wasn't available, so, in order to copy it from quay.io to docker.io, you can run the following command:

copy-quay --pull-registry quay.io --push-registry docker.io --from-namespace tripleomaster --to-namespace tripleomaster --push-hash 07715e073191d94abc2f35e6b90563f9 --hash 07715e073191d94abc2f35e6b90563f9 --job periodic-tripleo-ci-build-containers-ubi-8-push copy

This will check the job periodic-tripleo-ci-build-containers-ubi-8-push for the list of the containers to be coppied, the --pull-registry is from where you want to pull, in our case quay.io, but it could be from trunk.rdoregistry as well.
The --push-registry is where we want to copy the containers to. and --from-namespace is the namespace, in docker is docker.io/tripleomaster and quay is quay.io/tripleomaster as well, so, --from-namespace and --to-namespace are the same in this case.

The --push-hash is the hash you want to push, because you can put for example current-tripleo here, so that's the reason we have two hashes options and both are the same, we could say, i want the hash 0771 to be pushed as current-tripleo for example, or whatever else you want to.

If you don't pass nothing after the copy, it means it will copy all the containers found in the periodic job, this is just to generate the list of containers.