# Promoter Server - development environment
###### tags: `Documentation`

@ysandeep @akahat @bhagyashris

Please add your notes, bugs, to-do's here while attempting to setup a developer environment for the promotion server.

:::warning
ATTENTION: Make sure dlrn-promoter-service is disabled so it does not consume dlrn hashes and does not conflict with production promoter.

When you run with molecule, you are safe with `staging_setup` flag:
https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/promoter/templates/dlrn-promoter-service.sh.j2#L15
:::

## Folco's notes (automated provisioning in a remote target)
There is an automated way for deploying a test promoter server. This is an alternative for running manual steps.
:::success
The following steps run the provisioning playbook from your local machine and the target instance will deploy promoter plus staging env.
:::
- [ ] Ensure your have ansible/molecule on your local machine/laptop
- [ ] Spawn a new instance on the cloud (centos7.8+), assign a floating IP
- [ ] Add the instance IP to your hosts file in your local machine: `sudo echo 11.111.11.111 instance >> /etc/hosts`
- [ ] Set how you connect into the instance: `sudo vi /etc/ssh/ssh_config`

     Host instance
     HostName 11.111.11.111
     User centos

- [ ] Clone ci-config repo from github.com/rdo-infra/ci-config
- [ ] Run `molecule test -s promoter` on ci-config dir to provision promoter on the instance
- [ ] Run `molecule prepare -s container_push` on ci-config dir to set up staging environment (mock registries) on the instance
:::info
- promoter provisioning will create and deploy with user `promoter` by default, but will connect with `centos` user
- virtualenv will be created in ~/promoter_venv (under promoter user home)
:::

## Wes's first crack at this

* going to attempt to run molecule based integration jobs locally w/o teardown.
* tripleo-ci-promotion-staging-single-pipeline-centos-8
* tripleo-ci-promotion-staging-integration-pipeline-centos-8

## Chandan Kumar's notes
### Pre-requisites
1. Make sure you have fresh CentOS-8 vm
2. ***Donot try on main laptop having podman installed***
### Install and Configure Docker on CentOS-8
1. Adding docker-ce repo

sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo

2. Install docker-ce by skipping broken deps and iptables is needed otherwise docker service will not start

sudo dnf install docker-ce iptables --nobest -y

3. Enable docker service

sudo systemctl enable --now docker

4. Start docker service

sudo systemctl status docker

5. Add user to docker group

sudo usermod -aG docker $USER

6. Disable firewalld for DNS resolution inside Docker containers to work.

sudo systemctl disable firewalld

7. Reboot the system

sudo reboot

8. Make sure docker is working fine

docker pull alpine

9. Enter into docker container

docker run -it --rm alpine /bin/sh

10. Login to docker.io to generate /home/rdouser/.docker/config.json, needed for promoter

docker login

### Install and Running Promoter
1. Install basic packages

sudo dnf -y install git vim python3-virtualenv

2. Install tox

python3 -m pip install --user tox

3. Clone ci-config repo

git clone https://github.com/rdo-infra/ci-config

4. Create a virtualenv

virtualenv promoter_devel

5. Activate venv

source promoter_devel/bin/activate

5. Install promoter

cd ci-config/ci-scripts/dlrnapi_promoter/
pip install -r requirements.txt

### Setting up the stage server
1. Take a look at stage.py

python3 stage.py -h
(promoter_devel) [rdouser@localhost dlrnapi_promoter]$ python3 stage.py -h
usage: stage.py [-h] [--scenes SCENES] [--dry-run]
                [--promoter-user PROMOTER_USER]
                [--stage-config-file STAGE_CONFIG_FILE]
                [--stage-info-file STAGE_INFO_FILE]
                [--db-data-file DB_DATA_FILE]
                [--promoter-config-file PROMOTER_CONFIG_FILE]
                {setup,teardown}

Staging promoter

positional arguments:
  {setup,teardown}

optional arguments:
  -h, --help            show this help message and exit
  --scenes SCENES       Select scenes to create in the environment scenes
                        available dlrn,overcloud_images,containers,registries
  --dry-run             Don't do anything, still create stage-info
  --promoter-user PROMOTER_USER
                        The promoter user
  --stage-config-file STAGE_CONFIG_FILE
                        Config file for stage generation (relative to config
                        dir)
  --stage-info-file STAGE_INFO_FILE
                        Config file for stage results
  --db-data-file DB_DATA_FILE
                        Data file to inject to dlrn server (relative to config
                        dir)
  --promoter-config-file PROMOTER_CONFIG_FILE
                        Config file for promoter on which to base the stage
                        (relative to config dir)

2. Run the stage server

python3 stage.py setup --stage-config-file stage-config.yaml --db-data integration-pipeline.yaml --promoter-config-file staging/CentOS-8/master.ini

With new promoter code use below command:

python3 stage.py  --stage-info-file stage-config.yaml --db-data-file integration-pipeline.yaml setup --release-config CentOS-8/master.yaml

3. You can teardown the environment

python3 stage.py teardown --stage-config-file stage-config.yaml --db-data integration-pipeline.yaml --promoter-config-file staging/CentOS-8/master.ini

Happy Hacking.
Please also go through https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/README.md

## promoter deployment in production - rocky

### Things to fix while deployment
* Issues caught while deployment
- Promoter user is default user
- ssh, user permissons issue while running playbook.
- Promoter user should be passwordless ? (Need discussion with Sagi, Ronelle)
- Rsync issue,permissions (commond: rsync -zahrv --exclude=".tox" --exclude=".venv" /home/centos/ci-config/ promoter@localhost:/home/promoter/ci-config)
- Log server
-

* **promoter server details**
- server is created on upshift under the rhos-stage-dev project
- server ip: 10.0.151.55
- users:
- centos
- promoter
- password: redhat

- [ ] Promoter server requirements
- [ ] A box with 8 GB Ram and 200 GB external volume
- [ ] make sure docker is installed
- [ ] /var/lib/docker needs to be mounted to external volume
- [ ] It is need to avoid no space left error issue
- [x] centos8 vm with dnf update
- [x] Ansible installed and configured
- [x] Docker installed and Docker service up and running
- [x] Make sure git is installed
- [x] Clone the ci-config repo
- [x] virtualenv setup
- [ ] place the secrets at proper place
- [ ] dlrnapi client
- [ ] file server ssh key
- [ ] docker registry
- [ ] quay registry
- [ ] RDO registry
- [ ] Run the dlrn promoter service
- [ ] Make sure
- [ ] promoter target current-tripleo repo dir gets created on trunk.rdoproject.org
- [ ] if manifest is on, make sure image manifests are getting created
- [ ] docker registry have those images pushed
- [ ] image server have correct images
- [ ] Make sure the config and logs files are visible on promoter server
- [ ] Maintain a list who has the access to promoter for manual interevin
- [ ] File server on vm
- [ ] Containers push on quey.io

- NOTE: make sure source and target are saperate in config file.

* **List of Patches of improvements to do the deployment smooth:**
- https://review.rdoproject.org/r/#/c/31695/
- https://review.rdoproject.org/r/#/c/31693/
- https://review.rdoproject.org/r/#/c/31659/

## SUCCESS and next steps!!

* we had a successful ussuri promotion
* one issue to investigate, but not blocking

* On http://10.0.148.74/ promoter
* keep testing here
* enable master, stable/victoria stable/ussuri stable/train c8

* manual steps after infra-setup promoter executes for production
* line 41 /ci-config/ci-scripts/dlrnapi_promoter/dlrn-promoter.sh our up user for images.rdoproject.org
* line 19 set promoter_config_root to "rdo" in dlrn-promoter.sh
* dlrnapi_secret and registry_secret
* ~/.ssh/id_rsa this keys public version is either on images.rdoproject.org OR we reuse the same ssh key from a production server.
* enable / disable git pull's as you see fit

* we'll test all the c8 branches
* amol/bhagyashri continue to work on the container pull/push logging
* setup another internal promoter for OSP 16.2 and 17

## Things must have
- Required attributes for release config
- Validation tests
- Logs, Containers with correct hash, dlrn etc

## Create scripts for improvements:

* find the right hash from the api enpoints
* check wheather api end point is working or not
* get the agg hash and md5 hash and commit hash and distro hash
* wheather images are push to image servers and containers on docker.io, quoy.io
* generate log file for all clients in separate files
* maintain list of all the servers username and ssh keys, pasword.
* maintain the config file as simple as possible and make consistency in the variable names with code.

## Uploading cloud images to quay

Below are the steps to do that.

* Set up a new namespace https://quay.io/organization/testvictoria?tab=repos on Quay Registry
* ci-scripts/container-push/container-push.yml and under task Download and repush container images and under target_registries:
disable docker.io hosts task and enable quay.io

      - host: quay.io
        namespace: "tripleo{{ lookup('env','RELEASE') }}"
        username: "{{ lookup('env','QUAYDOTIO_USERNAME') }}"
        password: "{{ lookup('env','QUAYDOTIO_PASSWORD') }}"
        token: "{{ lookup('env','QUAYDOTIO_TOKEN') }}"
        auth_url: "https://quay.io/api/v1/'"
        schema: "v2_s1"

* in config_environments/rdo/CentOS-8/victoria.yaml set target_namespace: testvictoria
* in ~/registry_secret file set

export QUAYDOTIO_USERNAME='replace-username'
export QUAYDOTIO_PASSWORD='replace-password'
export QUAYDOTIO_TOKEN='replace-token'

* Source ~/registry_secret
* with config_environments/global_defaults.yaml set allowed_clients: registries_client,dlrn_client
* run the promoter at your own RISK: python3 dlrnapi_promoter.py --config-root rdo --release-config CentOS-8/victoria.yaml promoter-all
* The manual container push needs updating, it looks for container-list file.
Things chaged for quay push
* victoria targetnamespace pointing to testvictoria, for ussuri pointing to tripleou

### Steps to setup the promoter server

1. Create vm - c8
2. git pull the ci-config repo: git pull https://github.com/rdo-infra/ci-config
3. Make sure you have device ready /dev/vdb with size > 100 GB
4. Run the ansible playbook : ansible-playbook ~/ci-config/ci-scripts/infra-setup/servers_setup.yml --tag promoter_provisioning
5. Make sure the promoter_provisioning taged role is executed successfully.
6. Make sure the promoter user is created and login to promoter user
7. source the virtual env source ~/promoter_env/bin/activate
8. create ~/registry_secret and ~/dlrnapi_secret file under promoter user and add your scerets there accordingly.
9. line 41 /ci-config/ci-scripts/dlrnapi_promoter/dlrn-promoter.sh our up user for images.rdoproject.org
10. line 19 set promoter_config_root to "rdo" in dlrn-promoter.sh
11. ~/.ssh/id_rsa this keys public version is either on images.rdoproject.org OR we reuse the same ssh key from a production server.
12. enable / disable git pull's as you see fit
13. Copied id_rsa_uploader private key of rdo image server and putted it in current promoter ~/.ssh/ folder (e.g ~/.ssh/id_rsa_uploader)
14. check the dlrn-promoter service status.

## Reviews List
- https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32679
- https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32058
-

### promoter sync - 2021-March-31 agenda

* https://projects.engineering.redhat.com/browse/TRIPLEOCI-375 (copy containers in promoter server)
* Manifest push:
* move the promoter from docker to podman https://hackmd.io/JoufX3RKTsKvOcTtXYEhuA

### promoter sync - 2021-June-22 agenda

* copy containers : https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32938
* do the automation to change the criteria file for promotion
* Suggestion:
* add exclude_job_list var which will exclude the specifided job while doing the promotion and also set the time for the exclude job in the exclude job list once the specified time will off then that job will be considered as criteria job
* add CLI command line tool like tempest-skiplist which will remove or add the jobs from criteria file.

* currently we are off the git pull from the dlrn-script so how can be get rid off this.
* Pending work:
* manifest push - bhagyashris
* add a variable to promoter release config that expliciltiy sets use extended_hash as a boolean - bhagyashris
* do the automation to change the criteria file - not yet assign
* update dns record for promoter.rdoproject.org to 38.102.83.109 - amol
* investigate the more efficient way to test the mainfest work - amol
* Removed promote-images.sh file - https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32679 - amol
* Fix DLRN Aggregate hash path for c7 and c8 - https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32135/11 - amol