Nightly releases and Continous Integration for DUNE DAQ Software

### Nightly releases and Continous Integration for DUNE DAQ Software DUNE DAQ/SC Meeting Pengfei Ding Fermilab April 26, 2021 --- <style> .reveal { font-size: 24px; } .blue { color: blue; } .green { color: green; } .red { color: red; } </style> ## Introduction (I) * Continuous Integration (CI) practices: * **commit often** -- developers integrate code into share repository frequently (multiple times a day); * **automatically built/tested** each integration will be verified by automated build and tests; * Goal and benifits of CI: * ensure the most up-to-date (and tested) code is reflected and maintained; * prevent backups and delays when multiple devs are ongoing. * Continuous deployment (CD): * Ongoing delivery of features as they are updated, tested and ready for release. --- <img src="https://i.imgur.com/EuNf5BE.png" alt="CI drawing" style="width:600px;"/> --- ## Introduction to CI and nightly releases (II) * Nightly releases: * A step close to CD; * the develop release is built/tested/deployed nightly; * it will be the base for each repo's CI in the next day; * **underlying requirement:** releated PRs in multiple repos should be merged to develop on the same day. * **Benifits:** * For developers -- ensuring their code can be successfully integrated into the release on a daily basis; * For release managers -- ensuring the release is in good-standing and ready to be freezed and released. * Overall: Faster iterations, accelerated delivery, increased visibility of bugs and issues, low-risk releases. --- ### Nightly releases (I) * **Building:** - Contains the **HEAD of develop branch** across all DAQ repositories; - built on GitHub cloud node at 2:00am (CDT); - build log and pre-built release (if the build succeeds) is saved for 90 days on GitHub; * **Deployment:** - the release name is in the form of `NYY-MM-DD`; - the release will be deployed to `/cvmfs/dunedaq-development.opensciencegrid.org`; - a soft link named `last_successful` will ponit to the most recent successfully built nightly release. * **Usage:** - Just as a typical frozen release, - use `-r` with the patth to the `nightly` directory in cvmfs ```sh dbt-create.sh –r /cvmfs/dunedaq-development.opensciencegrid.org/nightly <NYY-MM-DD> workdir ``` --- ### Nightly releases (II) More details about how nightly releases are made: * Release Definition: * the same way as we defined, e.g. `dunedaq-v2.4.0`; * in the release manifest file `daq-release/configs/dunedaq-develop/release_manifest.sh` * list of external packages and their versions; * list of DAQ packages for the release. * Building a docker image: - built by a daily cronjob running on a server with cvmfs access; - based on `dunedaq/sl7-minimal:latest` image; - adding a layer of disk mirror containing external packages from cvmfs; - adding a layer of pre-built develop release; - the image is tagged as `dunedaq/sl7-minimal:dev` (**reused tag**); - and a **unique tag** `dunedaq/sl7-minimal-nightly:NYY-MM-DD`. --- ### Docker images - GitHub Repo: [daq-docker](https://github.com/DUNE-DAQ/daq-docker) - `sl7-minimal` contains the `Dockerfile` for making `dunedaq/sl7-minimal:latest` image; - any updates to the repo will trigger dockerhub to rebuild this image; - `sl7-ci` contains the `Dockerfile` and scripts to make the `dunedaq/sl7-minimal:dev` image; - it is refreshed nightly by a cronjob on a Fermilab server with access to cvmfs. - Dockerhub image: [dunedaq/sl7-minimal](https://hub.docker.com/repository/docker/dunedaq/sl7-minimal) - Dockerhub image: [dunedaq/sl7-minimal-nightly](https://hub.docker.com/repository/docker/dunedaq/sl7-minimal-nightly) --- ### Nightly releases (III) - Artifact generation: - a GitHub Action workflow in the `daq-release` repo; - run one hour after the docker image was built - checks out the develop branch of `daq-release` and `daq-buildtools`; - sets up a working directory with only external packages from `dunedaq-develop` release; - checks out develop branches of all DAQ packages and build the release; - uploads the pre-built release as ”artifacts” of the CI job. --- ### Demo (I): nightly GitHub Action for `daq-release` - Workflow status page: [Nightly workflow](https://github.com/DUNE-DAQ/daq-release/actions) - [Latest workflow run](https://github.com/DUNE-DAQ/daq-release/actions/runs/762687473) <img src="https://i.imgur.com/gSITupq.png" alt="CI drawing" style="width:600px;"/> --- ### Demo (II): nightly GitHub Action for `daq-release` - [Workflow configuration file](https://github.com/DUNE-DAQ/daq-release/blob/develop/.github/workflows/nightly.yml) ```yaml= # This is a basic workflow that is manually triggered name: Nightly workflow # Controls when the action will run. Workflow runs when manually triggered using the UI # or API. on: #push: #branches: [ develop ] schedule: - cron: "0 7 * * *" workflow_dispatch: # Inputs the workflow accepts. inputs: name: # Friendly description to be shown in the UI instead of 'name' description: 'Manual start the CI workflow' # Default value if no value is explicitly provided default: 'Yes' # Input has to be provided for the workflow to run required: true # A workflow run is made up of one or more jobs that can run sequentially or in parallel jobs: # This workflow contains a single job called "greet" bulid_the_develop_release: # The type of runner that the job will run on runs-on: ubuntu-latest container: image: dunedaq/sl7-minimal:dev env: DBT_AREA_FILE: "dbt-settings" defaults: run: shell: bash # Steps represent a sequence of tasks that will be executed as part of the job steps: # Runs a single command using the runners shell - name: Start the CI run: echo "Start the CI anyway ${{ github.event.inputs.name }}" - name: Checkout daq-buildtools uses: actions/checkout@v2 with: repository: DUNE-DAQ/daq-buildtools path: daq-buildtools - name: Checkout daq-release uses: actions/checkout@v2 with: repository: DUNE-DAQ/daq-release path: daq-release - name: setup dev area and checkout code run: | source $GITHUB_WORKSPACE/daq-buildtools/dbt-setup-env.sh dbt-create.sh -r $GITHUB_WORKSPACE/daq-release/configs dunedaq-develop dev cd dev cp $GITHUB_WORKSPACE/daq-release/configs/dunedaq-develop/release_manifest.sh . $GITHUB_WORKSPACE/daq-release/scripts/checkout-package.sh -f ./release_manifest.sh -a -b develop -o sourcecode - name: setup build env and build the dev release run: | source $GITHUB_WORKSPACE/daq-buildtools/dbt-setup-env.sh cd $GITHUB_WORKSPACE/dev export DBT_AREA_ROOT=$PWD export DBT_AREA_FILE="dbt-settings" source ${DBT_AREA_ROOT}/${DBT_AREA_FILE} setup_ups_product_areas || true setup -B python ${dune_python_version} || true source ${DBT_AREA_ROOT}/${DBT_VENV}/bin/activate setup_ups_products dune_devtools || true setup_ups_products dune_systems || true setup_ups_products dune_externals || true export DBT_INSTALL_DIR=${DBT_AREA_ROOT}/install export DBT_SETUP_BUILD_ENVIRONMENT_SCRIPT_SOURCED=1 dbt-build.sh --install nightly_tag="N$(date +%y-%m-%d)" mkdir -p $GITHUB_WORKSPACE/nightly/${nightly_tag} $GITHUB_WORKSPACE/daq-release/scripts/create-ups-products-area.sh -t $GITHUB_WORKSPACE/nightly/${nightly_tag}/packages cp $GITHUB_WORKSPACE/daq-release/configs/dunedaq-develop/dbt-build-order.cmake $GITHUB_WORKSPACE/nightly/${nightly_tag} cp $GITHUB_WORKSPACE/daq-release/configs/dunedaq-develop/pyvenv_requirements.txt $GITHUB_WORKSPACE/nightly/${nightly_tag} $GITHUB_WORKSPACE/daq-release/scripts/upsify-daq-pkgs.py -w $GITHUB_WORKSPACE/dev -i -o $GITHUB_WORKSPACE/nightly/${nightly_tag}/packages echo "PROGRESS: created ups products for DAQ packages." echo " dune_daqpackages=(" >> ${DBT_AREA_ROOT}/${DBT_AREA_FILE} echo "PROGRESS: updating dbt-settings.sh under /releases/dunedaq-develop" cd /releases/dunedaq-develop/packages find . -type d -name "*.version"|grep -v ups|sed 's/\.\//"/g'|sed 's/\.version/ e19:prof\"/g'|tr '/' ' '>> ${DBT_AREA_ROOT}/${DBT_AREA_FILE} echo ")">> ${DBT_AREA_ROOT}/${DBT_AREA_FILE} sed -i 's,.*/cvmfs/dunedaq.open," /cvmfs/dunedaq-development.opensciencegrid.org/nightly/'"$nightly_tag"'/externals,' ${DBT_AREA_ROOT}/${DBT_AREA_FILE} sed -i 's,.*/cvmfs/dunedaq-development.open," /cvmfs/dunedaq-development.opensciencegrid.org/nightly/'"$nightly_tag"'/packages,' ${DBT_AREA_ROOT}/${DBT_AREA_FILE} mv /releases/dunedaq-develop/externals $GITHUB_WORKSPACE/nightly/${nightly_tag} mv ${DBT_AREA_ROOT}/${DBT_AREA_FILE} $GITHUB_WORKSPACE/nightly/${nightly_tag}/dbt-settings.sh cd $GITHUB_WORKSPACE/nightly tar zcvf dunedaq-${nightly_tag}.tar.gz ${nightly_tag} rm -rf ${nightly_tag} - name: upload nightly tarball uses: actions/upload-artifact@v2 with: name: nightly release path: ${{ github.workspace }}/nightly - name: upload build log file uses: actions/upload-artifact@v2 with: name: build_log path: ${{ github.workspace }}/dev/log ``` --- ### CI for DAQ packages - Runs inside `dunedaq/sl7-minimal:dev` docker container; - Use the **latest develop/nightly release; - Only build **individual DAQ package**; - CI build is triggered by: - Push to develop; - Pull request to develop; - Manual trigger; - Daily at 2:00am (CDT); --- ### CI workflow triggered by Pull request <img src="https://i.imgur.com/8LlUitg.png" alt="CI drawing" style="width:600px;"/> --- ### CI workflow status for latest commit merged into develop <img src="https://i.imgur.com/SO24De5.png" alt="CI drawing" style="width:600px;"/> --- ### Manually trigger a CI workflow <img src="https://i.imgur.com/vSm5vR6.png" alt="CI drawing" style="width:600px;"/> --- ### Some details of the CI workflow - It sets up a working directory using the develop release first; - it *unsetup* the package itself before building; - the build log and pre-build package is saved for 90 days on GitHub; ```yaml= name: build-develop # Controls when the action will run. Workflow runs when manually triggered using the UI # or API. on: push: branches: [ develop ] paths-ignore: - 'docs/**' pull_request: branches: [ develop ] schedule: - cron: "0 7 * * *" workflow_dispatch: # A workflow run is made up of one or more jobs that can run sequentially or in parallel jobs: Build_against_dev_release: # The type of runner that the job will run on runs-on: ubuntu-latest container: image: dunedaq/sl7-minimal:dev defaults: run: shell: bash env: DBT_AREA_FILE: "dbt-settings" # Steps represent a sequence of tasks that will be executed as part of the job steps: # Runs a single command using the runners shell - name: Checkout daq-buildtools uses: actions/checkout@v2 with: repository: DUNE-DAQ/daq-buildtools path: daq-buildtools - name: Checkout daq-release uses: actions/checkout@v2 with: repository: DUNE-DAQ/daq-release path: daq-release - name: setup dev area run: | source $GITHUB_WORKSPACE/daq-buildtools/dbt-setup-env.sh dbt-create.sh -r /releases dunedaq-develop dev - name: checkout package for CI uses: actions/checkout@v2 with: path: ${{ github.repository }} - name: setup build env and build the dev release run: | export REPO=$(echo '${{ github.repository }}' | awk -F '/' '{print $2}') source $GITHUB_WORKSPACE/daq-buildtools/dbt-setup-env.sh cd $GITHUB_WORKSPACE/dev dbt-setup-build-environment || true setup_repo_env="SETUP_$REPO" [[ -z ${!setup_env_repo+x} ]] || unsetup $REPO cp -pr $GITHUB_WORKSPACE/DUNE-DAQ/$REPO $GITHUB_WORKSPACE/dev/sourcecode dbt-build.sh --install $GITHUB_WORKSPACE/daq-release/scripts/upsify-daq-pkgs.py -w $GITHUB_WORKSPACE/dev -o /tarballs - name: upload UPS tarball and log file uses: actions/upload-artifact@v2 with: name: ups_package path: /tarballs - name: upload build log file uses: actions/upload-artifact@v2 with: name: build_log path: ${{ github.workspace }}/dev/log ``` --- ### Instructions of setting CI workflow for a DAQ package 1. Go to the `Actions` tab of the GitHub repo; 2. Click "set up this workflow" using the template `dunedaq-develop CI for C++` 3. Commit the workflow YAML file without change into the directory of `REPO/.github/workflows` 4. A second template which rebuilds the whole release is also available. **Note: These two workflows are not to be changed by developers but release mangers.** <img src="https://i.imgur.com/EzkAojZ.png" alt="CI drawing" style="width:600px;"/> --- ## Live demo - Let's enable CI for [`cmdlib`](https://github.com/DUNE-DAQ/cmdlib) --- ### Future improvements * **Additional workflows:** * doing clang-format linting; * running unit tests and generating code coverage reports. * **Addtional templates for developers to use:** * adding integration tests. * Automating other parts of release making procedure (for release managers): * generating/publishing documentations (done); * making pypi repositories for python modules we use; * building external dependencies. * **Self-hosted runner machines for CD:** * A server at CERN or Fermilab with cvmfs access; * cron job with kerneros enabled for publishing to cvmfs. --- ### Conclusion * Initial nighly releases and CI for packages are ready (feedbacks are extremely welcome); * Additional features are on the way: * CI workflows for linting and code coverage reporting will soon be enabled; * template for developers to add integration tests; * Please make your package's develop branch **always build** at the end of day.