owned this note   changed 4 years ago  
Published Linked with GitHub

Promoter Server - development environment

tags: Documentation

@ysandeep @akahat @bhagyashris

Please add your notes, bugs, to-do's here while attempting to setup a developer environment for the promotion server.

ATTENTION: Make sure dlrn-promoter-service is disabled so it does not consume dlrn hashes and does not conflict with production promoter.

When you run with molecule, you are safe with staging_setup flag:
https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/roles/promoter/templates/dlrn-promoter-service.sh.j2#L15

Folco's notes (automated provisioning in a remote target)

There is an automated way for deploying a test promoter server. This is an alternative for running manual steps.

The following steps run the provisioning playbook from your local machine and the target instance will deploy promoter plus staging env.

  • Ensure your have ansible/molecule on your local machine/laptop
  • Spawn a new instance on the cloud (centos7.8+), assign a floating IP
  • Add the instance IP to your hosts file in your local machine: sudo echo 11.111.11.111 instance >> /etc/hosts
  • Set how you connect into the instance: sudo vi /etc/ssh/ssh_config
     Host instance
     HostName 11.111.11.111
     User centos
  • Clone ci-config repo from github.com/rdo-infra/ci-config
  • Run molecule test -s promoter on ci-config dir to provision promoter on the instance
  • Run molecule prepare -s container_push on ci-config dir to set up staging environment (mock registries) on the instance
  • promoter provisioning will create and deploy with user promoter by default, but will connect with centos user
  • virtualenv will be created in ~/promoter_venv (under promoter user home)

Wes's first crack at this

  • going to attempt to run molecule based integration jobs locally w/o teardown.
    • tripleo-ci-promotion-staging-single-pipeline-centos-8
    • tripleo-ci-promotion-staging-integration-pipeline-centos-8

Chandan Kumar's notes

Pre-requisites

  1. Make sure you have fresh CentOS-8 vm
  2. Donot try on main laptop having podman installed

Install and Configure Docker on CentOS-8

  1. Adding docker-ce repo
sudo dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
  1. Install docker-ce by skipping broken deps and iptables is needed otherwise docker service will not start
sudo dnf install docker-ce iptables --nobest -y
  1. Enable docker service
sudo systemctl enable --now docker
  1. Start docker service
sudo systemctl status docker
  1. Add user to docker group
sudo usermod -aG docker $USER
  1. Disable firewalld for DNS resolution inside Docker containers to work.
sudo systemctl disable firewalld
  1. Reboot the system
sudo reboot
  1. Make sure docker is working fine
docker pull alpine
  1. Enter into docker container
docker run -it --rm alpine /bin/sh
  1. Login to docker.io to generate /home/rdouser/.docker/config.json, needed for promoter
docker login

Install and Running Promoter

  1. Install basic packages
sudo dnf -y install git vim python3-virtualenv
  1. Install tox
python3 -m pip install --user tox
  1. Clone ci-config repo
git clone https://github.com/rdo-infra/ci-config
  1. Create a virtualenv
virtualenv promoter_devel
  1. Activate venv
source promoter_devel/bin/activate
  1. Install promoter
cd ci-config/ci-scripts/dlrnapi_promoter/
pip install -r requirements.txt

Setting up the stage server

  1. Take a look at stage.py
python3 stage.py -h
(promoter_devel) [rdouser@localhost dlrnapi_promoter]$ python3 stage.py -h
usage: stage.py [-h] [--scenes SCENES] [--dry-run]
                [--promoter-user PROMOTER_USER]
                [--stage-config-file STAGE_CONFIG_FILE]
                [--stage-info-file STAGE_INFO_FILE]
                [--db-data-file DB_DATA_FILE]
                [--promoter-config-file PROMOTER_CONFIG_FILE]
                {setup,teardown}

Staging promoter

positional arguments:
  {setup,teardown}

optional arguments:
  -h, --help            show this help message and exit
  --scenes SCENES       Select scenes to create in the environment scenes
                        available dlrn,overcloud_images,containers,registries
  --dry-run             Don't do anything, still create stage-info
  --promoter-user PROMOTER_USER
                        The promoter user
  --stage-config-file STAGE_CONFIG_FILE
                        Config file for stage generation (relative to config
                        dir)
  --stage-info-file STAGE_INFO_FILE
                        Config file for stage results
  --db-data-file DB_DATA_FILE
                        Data file to inject to dlrn server (relative to config
                        dir)
  --promoter-config-file PROMOTER_CONFIG_FILE
                        Config file for promoter on which to base the stage
                        (relative to config dir)
  1. Run the stage server
python3 stage.py setup --stage-config-file stage-config.yaml --db-data integration-pipeline.yaml --promoter-config-file staging/CentOS-8/master.ini

With new promoter code use below command:

python3 stage.py  --stage-info-file stage-config.yaml --db-data-file integration-pipeline.yaml setup --release-config CentOS-8/master.yaml
  1. You can teardown the environment
python3 stage.py teardown --stage-config-file stage-config.yaml --db-data integration-pipeline.yaml --promoter-config-file staging/CentOS-8/master.ini

Happy Hacking.
Please also go through https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/README.md

promoter deployment in production - rocky

Things to fix while deployment

  • Issues caught while deployment

    • Promoter user is default user
    • ssh, user permissons issue while running playbook.
    • Promoter user should be passwordless ? (Need discussion with Sagi, Ronelle)
    • Rsync issue,permissions (commond: rsync -zahrv –exclude=".tox" –exclude=".venv" /home/centos/ci-config/ promoter@localhost:/home/promoter/ci-config)
    • Log server

  • promoter server details

    • server is created on upshift under the rhos-stage-dev project
    • server ip: 10.0.151.55
    • users:
      • centos
      • promoter
        • password: redhat

  • Promoter server requirements

    • A box with 8 GB Ram and 200 GB external volume
    • make sure docker is installed
    • /var/lib/docker needs to be mounted to external volume
      • It is need to avoid no space left error issue

  • centos8 vm with dnf update

  • Ansible installed and configured

  • Docker installed and Docker service up and running

  • Make sure git is installed

  • Clone the ci-config repo

  • virtualenv setup

  • place the secrets at proper place

    • dlrnapi client
    • file server ssh key
    • docker registry
    • quay registry
    • RDO registry

  • Run the dlrn promoter service

  • Make sure

    • promoter target current-tripleo repo dir gets created on trunk.rdoproject.org
    • if manifest is on, make sure image manifests are getting created
    • docker registry have those images pushed
    • image server have correct images
    • Make sure the config and logs files are visible on promoter server

  • Maintain a list who has the access to promoter for manual interevin

  • File server on vm

  • Containers push on quey.io

    • NOTE: make sure source and target are saperate in config file.

SUCCESS and next steps!!

  • we had a successful ussuri promotion

    • one issue to investigate, but not blocking

  • On http://10.0.148.74/ promoter

    • keep testing here
      • enable master, stable/victoria stable/ussuri stable/train c8

  • manual steps after infra-setup promoter executes for production

    • line 41 /ci-config/ci-scripts/dlrnapi_promoter/dlrn-promoter.sh our up user for images.rdoproject.org
    • line 19 set promoter_config_root to "rdo" in dlrn-promoter.sh
    • dlrnapi_secret and registry_secret
    • ~/.ssh/id_rsa this keys public version is either on images.rdoproject.org OR we reuse the same ssh key from a production server.
    • enable / disable git pull's as you see fit

  • we'll test all the c8 branches

  • amol/bhagyashri continue to work on the container pull/push logging

  • setup another internal promoter for OSP 16.2 and 17

Things must have

  • Required attributes for release config
  • Validation tests
    • Logs, Containers with correct hash, dlrn etc

Create scripts for improvements:

  • find the right hash from the api enpoints
  • check wheather api end point is working or not
  • get the agg hash and md5 hash and commit hash and distro hash
  • wheather images are push to image servers and containers on docker.io, quoy.io
  • generate log file for all clients in separate files
  • maintain list of all the servers username and ssh keys, pasword.
  • maintain the config file as simple as possible and make consistency in the variable names with code.

Uploading cloud images to quay

Below are the steps to do that.

      - host: quay.io
        namespace: "tripleo{{ lookup('env','RELEASE') }}"
        username: "{{ lookup('env','QUAYDOTIO_USERNAME') }}"
        password: "{{ lookup('env','QUAYDOTIO_PASSWORD') }}"
        token: "{{ lookup('env','QUAYDOTIO_TOKEN') }}"
        auth_url: "https://quay.io/api/v1/'"
        schema: "v2_s1"
  • in config_environments/rdo/CentOS-8/victoria.yaml set target_namespace: testvictoria
  • in ~/registry_secret file set
export QUAYDOTIO_USERNAME='replace-username'
export QUAYDOTIO_PASSWORD='replace-password'
export QUAYDOTIO_TOKEN='replace-token'
  • Source ~/registry_secret
  • with config_environments/global_defaults.yaml set allowed_clients: registries_client,dlrn_client
  • run the promoter at your own RISK: python3 dlrnapi_promoter.py –config-root rdo –release-config CentOS-8/victoria.yaml promoter-all
  • The manual container push needs updating, it looks for container-list file.
    Things chaged for quay push
  • victoria targetnamespace pointing to testvictoria, for ussuri pointing to tripleou

Steps to setup the promoter server

  1. Create vm - c8
  2. git pull the ci-config repo: git pull https://github.com/rdo-infra/ci-config
  3. Make sure you have device ready /dev/vdb with size > 100 GB
  4. Run the ansible playbook : ansible-playbook ~/ci-config/ci-scripts/infra-setup/servers_setup.yml –tag promoter_provisioning
  5. Make sure the promoter_provisioning taged role is executed successfully.
  6. Make sure the promoter user is created and login to promoter user
  7. source the virtual env source ~/promoter_env/bin/activate
  8. create ~/registry_secret and ~/dlrnapi_secret file under promoter user and add your scerets there accordingly.
  9. line 41 /ci-config/ci-scripts/dlrnapi_promoter/dlrn-promoter.sh our up user for images.rdoproject.org
  10. line 19 set promoter_config_root to "rdo" in dlrn-promoter.sh
  11. ~/.ssh/id_rsa this keys public version is either on images.rdoproject.org OR we reuse the same ssh key from a production server.
  12. enable / disable git pull's as you see fit
  13. Copied id_rsa_uploader private key of rdo image server and putted it in current promoter ~/.ssh/ folder (e.g ~/.ssh/id_rsa_uploader)
  14. check the dlrn-promoter service status.

Reviews List

promoter sync - 2021-March-31 agenda

promoter sync - 2021-June-22 agenda

  • copy containers : https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32938

  • do the automation to change the criteria file for promotion

    • Suggestion:
      • add exclude_job_list var which will exclude the specifided job while doing the promotion and also set the time for the exclude job in the exclude job list once the specified time will off then that job will be considered as criteria job
      • add CLI command line tool like tempest-skiplist which will remove or add the jobs from criteria file.

  • currently we are off the git pull from the dlrn-script so how can be get rid off this.

  • Pending work:

Select a repo