---
# System prepended metadata

title: INUITS - EESSI

---

# INUITS - EESSI

## Lessons learned

(TODO Hafsa: what worked, what didn't work, good/bad things, ...)

* Readme was really helpful to understand the software layer and to set up the EESSI bot components.
* The regular meetings and discussions helped a lot with the progress and the notes after every meeting were also helpful to keep track of everything.
* The reviews on the PR were always detailed and clear (they were also explained in the meetings) which made it easy to make changes accordingly.
* most challenging: git/GitHub
* most learned: Git, merge conflicts, Python

---------------------------------------------------------------------------------------------------

## Next meetings

- ...

---------------------------------------------------------------------------------------------------

## Sync meeting Mon 2 Jan 2023

11:00 CET - 15:00 PKT

- OK for Hafsa, Thomas, Kenneth

### Meeting agenda

- PR #131: very close to being finished
- PR #132: looks OK
- PR #136: we should avoid repeating the same code, and implement a `retry` function instead?
- things to improve in bot
    - make sure that it's clear what the status is of a build (still running vs bot crashed, etc.)
    - support a way to ask the bot for a status update
    - status page of all bot instances

---------------------------------------------------------------------------------------------------

## Sync meeting Wed 28 Dec 2022

10:00 CET - 14:00 PKT

- OK for Hafsa, Thomas - Kenneth unsure

### Meeting agenda

- PR #130
    - Suggest to add a comment that config is read to raise an exception early when the event handler starts. Line 120 event handler and line 577 in job manager.
    - Remove new log method from job manager
    - (Thomas) create issue about reading config multiple times (see https://github.com/EESSI/eessi-bot-software-layer/pull/130#pullrequestreview-1230996141)
- PR #131
    - issue #125 (dedicated log method)
    - Do not read config in log (event handler). Write an __init__ method which is similar to the one for the job manager. The PyGHee needs to be initialized though.
    - Addressing failing tests "KeyError" (see https://github.com/EESSI/eessi-bot-software-layer/actions/runs/3790115473/jobs/6444477781)
        - Deposit a basic app.cfg in tests directory
        - `[job_manager]`
        - `log_path = some_relative_path` (without a leading `/`)
        - Or create that file before initializing the job manager instance.
- PR #132
    - issue #20 (retry communication with GitHub if first attempt failed)
    - A few minor changes should be implemented. See PR comments.
    - Suggest to add addressed issues in PR description and mention that one case of communicating with GitHub is improved.
- issue #30 (handle connection error to GitHub gracefully)
    - pick any other communication with GitHub (e.g., tools/pr_comments.py) and add `try/except` blocks and also `tries` loop
- next issues
    - issue #110 (make log method print function name that called it)



---------------------------------------------------------------------------------------------------

## Sync meeting Wed 21 Dec 2022

10:00 CET - 14:00 PKT

- OK for Kenneth, Hafsa, Thomas

### Meeting agenda

- PR #116 merged
- PR #128 is replaced by PR #130 (for issue #112)
    - validating whether all required config settings are provided (like in Thomas' PR #85) can be done in a separate PR
    - see PR review for PR #130
    - also use `read_config` in job manager
- next issues
    - issue #125 (dedicated log method)
    - issue #110 (make log method print function name that called it)
    - issue #30 (handle connection error to GitHub gracefully)
    - issue #20 (retry communication with GitHub if first attempt failed)

---------------------------------------------------------------------------------------------------

## Sync meeting Mon 19 Dec 2022

09:00 CET - 13:00 PKT

- OK for Kenneth, Hafsa, Thomas

### Meeting agenda

- PR #116 (added comment in pr for showing running state of the jobs)
    - only missing docstring in `process_running_jobs` function, then good to merge
- PR #123 (improve start of the app)
    - merged!
- PR #124
    - merged!
- PR #128 for issue #112
    - see suggestions in PR review
- next issues
    - issue #125 (dedicated log method)
    - issue #110 (make log method print function name that called it)
    - issue #30 (handle connection error to GitHub gracefully)
    - issue #20 (retry communication with GitHub if first attempt failed)

---------------------------------------------------------------------------------------------------

## Sync meeting Mon 12 Dec 2022

11:00 CET - 15:00 PKT

- OK for Kenneth, Hafsa, Thomas

### Meeting agenda

- PRs
    - #116: add missing docstrings
    - #123: see PR review
    - #124: see PR review
- next issues
    - #112 (generic read_config function)
    - #125 log method

---------------------------------------------------------------------------------------------------

## Sync meeting Wed 07 Dec 2022

14:00 CET - 18:00 PKT

- OK for Kenneth, Hafsa, Thomas

### Meeting agenda

- Merged PR #83 PR comments
- PR #116 added comment in pr for showing running state of the jobs
    - addresses issue #27
    - determines running jobs and adds row(s) to PR comment
    - missing: only add row once
        - for this the comment body needs to be scanned
- Next issues to look at
    - issue #9 improve information printed at start of event handler (and job manager)
        - for now only change the eessi-bot-software-layer code (later changes maybe ported to PyGHee)
    - issue #97 (permission to trigger builds)
    - issue #112 (generic read_config function)
 

---------------------------------------------------------------------------------------------------

## Sync meeting Mon 05 Dec 2022

10:00 CET - 14:00 PKT

- OK for Kenneth, Hafsa, Thomas

### Meeting agenda

- EESSI monthly update meeting
- Merged PR #80 argparse
- Merged PR #84 username
- PR #83 PR comments
    - closed PR #82
    - Conflicts are resolved
    - few minor issues will be resolved (Thomas forgot to submit the review and Hafsa couldn't see them)
- issue #27 print more information about state changes of jobs
    - when a job starts running a line to the PR comment is added
    - 1st step: determine running jobs and log information (not harmful to log everytime the job manager runs the main loop)
        - maybe need to run squeue or parse its output for 'RUNNING'
        - also consider to update/change current function for obtaining current jobs
            - seems not needed as state is already included in returned data structure
        - probably need to define a new function, for example, determine_running_jobs(self, current_jobs)
        - plus a second function which processed running jobs
    - 2nd step: update PR comment once
    - bonus 1: update every hour (or make update frequency configurable)
    - bonus 2: report about time left for the job

- Next issues to look at
    - issue #9 improve information printed at start of event handler (and job manager)
        - for now only change the eessi-bot-software-layer code (later changes maybe ported to PyGHee)
 
---------------------------------------------------------------------------------------------------

## Sync meeting Wed 30 Nov 2022

10:00 CET - 14:00 PKT

- OK for Kenneth, Hafsa, Thomas

### Meeting agenda

- EESSI monthly update meeting tomorrow
- PR #80 argparse
    - fix unusued import, then ready to be merged
- PR #83 PR comments
    - close PR #82
    - try to address comments in PR (more generic function, docstrings)
    - add logging output to figure out why the more generic function doesn't work
- PR #84 username
    - looks fine, can be merged as is
- Next issues to look at
    - issue #9 improve information printed at start of event handler (and job manager)
        - for now only change the eessi-bot-software-layer code (later changes maybe ported to PyGHee)
    - issue #27 print more information about state changes of jobs
        - when a job starts running a line to the PR comment is added

-------------------------------------
--------------------------------------------------------------

## Sync meeting Mon 28 Nov 2022

10:00 CET - 14:00 PKT

- OK for Kenneth, Hafsa, Thomas

### Meeting agenda

- PR #80 argparse
    - https://github.com/EESSI/eessi-bot-software-layer/pull/80
    - few minor changes to make, then ready to merge PR
- draft PR #82
    - addresses issue #32, function for identifying PR comment to be updated
        - https://github.com/EESSI/eessi-bot-software-layer/issues/32
    - avoid getting commits into PR which don't belong to the PR by using the following procedure
    ```
    git clone https://github.com/YOUR_GH_ACCOUNT/eessi-bot-software-layer identify-pr-comment
    cd identify-pr-comment
    git branch identify-pr-comment
    git checkout identify-pr-comment
    ```
    - discussed draft PR
        - add parameter for search_pattern
        - move `return None` to correct indentation level
            - capturing such unintended changes could be covered by some tests?
        - rename (?) file `github.py` to `pr_comments.py` to avoid confusion with `connections/github.py`
        - add missing `import`
- until next meeting:
    - try to finish two PRs as they are
    - if time, either add tests for them or find a new issue (looked a bit into issues but didn't find an obvious one to continue)

---------------------------------------------------------------------------------------------------

## Sync meeting Thu 24 Nov 2022

10:00 CET - 14:00 PKT

- OK for Kenneth, Hafsa, Thomas

### Meeting agenda

- PR #78: should be good to merge
    - https://github.com/EESSI/eessi-bot-software-layer/pull/78
- draft PR #80 to rework option parser
    - https://github.com/EESSI/eessi-bot-software-layer/pull/80
- next issue for Hafsa: https://github.com/EESSI/eessi-bot-software-layer/issues/32

---------------------------------------------------------------------------------------------------

## Sync meeting Mon 21 Nov 2022

10:00 CET - 14:00 PKT

- OK for Kenneth, Hafsa (Thomas excused)

### Meeting agenda

- problem with bot tripping over unknown jobs fixed by cleaning up 'jobs' directory
- PR #78
    - run_cmd needs a log_file argument to make sure logging is done in correct file
    - enhance existing test for run_cmd to check log_file functionality
- issue #25
    - open draft PR for parse_common_args
    - ArgumentParser.parse_args will fail if it finds unknown options => problem for parse_common_args...

---------------------------------------------------------------------------------------------------

## Sync meeting Wed 16 Nov 2022

10:00 CET - 14:00 PKT

- OK for Thomas, Kenneth, Hafsa

### Meeting agenda

- problems for Hafsa with connecting to the AWS cluster
    - test with another SSH server?
    - Kenneth will look into getting Hafsa a VSC account to access login.hpc.ugent.be
- PR #63 merged
- PR #70 adds flake8 check in CI (issue #64)
    - should be almost ready to go now, only small style fixes needed still
- next:
    - issue #33: follow-up by Thomas
    - issue #61: run_cmd everywhere => Hafsa
- after:
    - issue #25: dedicated parse_args function => Hafsa
        - incl. tests for the parse_args function(s)
    - issue #32: function to locate PR comment to update => Hafsa
    - issue #9: cleaner 'start' method for app => Hafsa
    - issue #20: retry communication with GitHub if it fails => Hafsa
    - issue #38: add unit tests for existing functions => Hafsa
        - this can be done by only touching files in test/ directory
        - for `task/build.py` functions:
            - should be easy: `create_pr_dir`, `get_build_env_cfg`, `create_metadata`,
            - a bit more difficult (talks to GitHub): `download_pr`, `setup_pr_in_arch_job_dir`
            - mroe difficult (talks to Slurm): `submit_job`, `submit_build_jobs`

---------------------------------------------------------------------------------------------------

## Sync meeting Mon 14 Nov 2022

10:00 CET - 14:00 PKT

- OK for Thomas, Kenneth, Hafsa

### Meeting agenda

- working on issue #33
    - see PR #63 (fix_bug branch)
- next issues (use a separate branch!)
    - issue #25: dedicated parse_args function
    - issue #32: function to locate PR comment to update
    - issue #61: run_cmd everywhere
    - issue #64: flake8 (Thomas)

---------------------------------------------------------------------------------------------------

## Sync meeting Wed 9 Nov 2022

14:00 CET - 18:00 PKT

- OK for Thomas, Kenneth, Hafsa

### Meeting agenda

- done
    - run_cmd PR merged
    - refactoring PR merged
    - started new PR to make bot ignore non-bot jobs
- for future PRs: start a new branch from latest `main` branch
  ```
  git checkout main
  git pull origin main
  
  # clean up old branches
  git branch -d refactoring
  
  # create new branch for new PR
  git checkout -b example_branch
  git add ...
  git commit ...
  git push YOUR_FORK example_branch
  ```
- next
    - issue #25: dedicated parse_args function
    - issue #32: function to locate PR comment to update
    - issue #61: run_cmd everywhere

---------------------------------------------------------------------------------------------------

## Sync meeting Mon 7 Nov 2022

10:00 CET - 14:00 PKT

- OK for Thomas, Kenneth, Hafsa

### Meeting agenda

- follow-up on current issues + PRs
    - 1) finish PR #56 (run_cmd)
    - 2) open PR for additional refactoring (issue #53)
    - 3) open PR for issue #33: avoid that bot crashes on non-bot jobs
    - then follow up on priority:high issues
- Thomas
    - look into new PR on top of main for `deploy` support
        - new PR for deploy (+ eessi-upload-* script?)
            - or maybe to new EESSI/tools repo?
        - new PR for resubmit
    - get status from Jakob on issue #6 (error reporting)

---------------------------------------------------------------------------------------------------

## Sync meeting Thu 3 Nov 2022

11:00 CET - 15:00 PKT
 
- OK for Thomas, Kenneth, Hafsa

### Meeting agenda

- EESSI meeting: 14:00 CET - 18:00 PKT
- run_cmd PR
- next issues

---------------------------------------------------------------------------------------------------

## Sync meeting Wed 2 Nov 2022

10:00 CET - 14:00 PKT

- OK for Kenneth, Thomas, Hafsa

### Meeting agenda

- follow up on `run_cmd` PR (https://github.com/EESSI/eessi-bot-software-layer/pull/56)
    - code style fixes, test for `run_cmd`, suggestions in PR review
- prepare slides on progress on bot for EESSI monthly meeting (Thu 3 Nov, 14:00 CET - 18:00 PKT)
    - https://docs.google.com/presentation/d/125xb6892Sn5FY-JzDgNjTXeGDzYSpUGJzPyN_MJ0lwE
    - see (current) slide 9-10

---------------------------------------------------------------------------------------------------

## Sync meeting Fri 28 Oct 2022

10:00 CEST - 13:00 PKT

### Meeting agenda

- PR #52 is merged \o/
- code style check added
    - see https://github.com/EESSI/eessi-bot-software-layer/pull/54
- tests/CI
    - see https://github.com/EESSI/eessi-bot-software-layer/pull/55
- next issues:
    - implementation of run function (#44)
        - TODO:
            - fix code style issues
                - check locally with `flake8` command, install with `pip install`
            - add test for `run_cmd` function
                - check locally with `./test.run` (install `pytest` first with `pip install`)
            - tackle PR review remarks
    - avoid crash for non-bot jobs(#33)

---------------------------------------------------------------------------------------------------

## Sync meeting Wed 26 Oct 2022

10:00 CEST - 13:00 PKT

### Meeting agenda

- problems with smee.io being unstable
    - alternatives?
        - ngrok + localtunnel
            - see https://docs.github.com/en/developers/apps/getting-started-with-apps/setting-up-your-development-environment-to-create-a-github-app#step-1-start-a-new-smee-channel
- discuss recent PRs
    - refactor: https://github.com/EESSI/eessi-bot-software-layer/pull/52
        - PR is almost ready, see remarks
- next issues to work on:
    - implementation of dedicated run function (https://github.com/EESSI/eessi-bot-software-layer/issues/44)
- let Hafsa present progress on the bot during next EESSI meeting?

---------------------------------------------------------------------------------------------------

## Sync meeting Mon 24 Oct 2022

11:00 CEST - 14:00 PKT

### Meeting agenda

- discuss recent PRs
    - recfactor: https://github.com/EESSI/eessi-bot-software-layer/pull/52
- look at next issues to work on
    - after refactor: dedicated `run_cmd` function https://github.com/EESSI/eessi-bot-software-layer/issues/44

---------------------------------------------------------------------------------------------------

## Sync meeting Wed 19 Oct 2022

11:00 CEST - 14:00 PKT

### Meeting agenda

- changes to rename event handler were done
    - PR opened https://github.com/EESSI/eessi-bot-software-layer/pull/50
- preparing to refactor, see https://github.com/EESSI/eessi-bot-software-layer/issues/36
- fixing hardcoding of `--hold` (see https://github.com/EESSI/eessi-bot-software-layer/issues/35)

---------------------------------------------------------------------------------------------------

## Sync meeting Mon 17 Oct 2022

13:30 CEST - 16:30 PKT

### Meeting agenda

- go over worflow again to develop & test the bot
- questions by Hafsa
    - Why do jobs need to be submitted with hold?
        - let event handler submit jobs, but ensure that job manager is in control over when jobs start
        - that way, job can only start running after job manager has seen it
        - + we will have a clear record in the job manager log for each job (start/running/done)
        - workaround for lack of Slurm accounting database in CitC cluster
        - job manager can also run a single loop iteration + with a filter for a specific job ID
            - see extra command line options for job manager
    - currently working on:
        - understanding the event manager log file + corresponding code
        - looking into issue #37 (rename for event manager) + other issues marked with "good first issue" label

---------------------------------------------------------------------------------------------------

## Sync meeting Fri 14 Oct 2022

14:00 CEST - 17:00 PKT

### Meeting agenda

- updates by Hafsa
    - questions on README from EESSI/eessi-bot-software-layer repository
        - which value should be used for
            - cvmfs_customizations
                - should be set empty ({})
            - jobs_base_dir + jobs_ids_dir
                - $HOME can't be used, should be absolute path
    - get bot working in Hafsa's account on CitC cluster
        - success, see https://github.com/Hafsa-Naeem/software-layer/pull/1
    - TODO Hafsa
        - check log files (pyghee.log + eessi_bot_job_manager.log) to have a clear view what is being done
        - try to replicate the demo using a new test PR
            - use this URL: https://github.com/Hafsa-Naeem/software-layer/compare/main...EESSI:software-layer:add-CaDiCaL-9.3.0?expand=1
        - update README (https://github.com/EESSI/eessi-bot-software-layer/issues/47)

---------------------------------------------------------------------------------------------------

## Sync meeting Mon 10 Oct 2022

14:00 CEST - 17:00 PKT

### Meeting agenda

- questions by Hafsa
- things to do with Hafsa
    - set up account on CitC cluster using GitHub account
    - demo of bot with PR
- next steps
    - set up bot according to README, see https://github.com/EESSI/eessi-bot-software-layer

- notes
    - demo of bot done, see recorded meeting
        - TOD Hafsa: try and set up the bot using documentation in README file + replicate demo
    - account for Hafsa @ CitC has been set up
        - log in with `ssh hafsa@3.250.220.9`
        - more info at https://github.com/EESSI/hackathons/tree/main/2022-01/citc
    - several issues have been opened, labels have been added, see https://github.com/EESSI/eessi-bot-software-layer/issues
        - issues with "difficulty:easy" + "good first issue" are good starting points