Ursaverance Knowledgeshare Notes

# Ursaverance Knowledge Share ## Reference Materials - [Perseverance Skills](https://github.com/anaconda-distribution/perseverance-skills/tree/main) - [Perseverance Backlog List (Jira)](https://anaconda.atlassian.net/jira/software/c/projects/PKG/boards/222/backlog?issueLimit=100) - [`aggregate`](https://github.com/AnacondaRecipes/aggregate) repo ## Session Notes ### Session 7 (recorded, November 16, 2023) #### Participants - Lorenzo Pirritano (Perseverance) - Bianca Henderson (Ursa) - Ken Odegard (Ursa) #### What Happened / What Was Learned? - Wrapping up dependencies necessary and verifying that they are available on the relevant channel(s) - Identifying which steps of the build process can be automated (i.e., pressing "Y" for confirming things on the command line) * * * ### Session 6 (recorded, November 9, 2023) #### Participants - Lorenzo Pirritano (Perseverance) - Bianca Henderson (Ursa) - Ken Odegard (Ursa) #### What Happened / What Was Learned? - All the previous open PRs for the packages need to be reviewed by at least 2 team members and merged. The packages needs to be deployed before being used as dependencies by other packages. **Featured package:** [`pytest-bdd`](https://github.com/AnacondaRecipes/pytest-bdd-feedstock) - Start from this [Jira Ticket](https://anaconda.atlassian.net/browse/PKG-2697), follow the [procedure](https://github.com/anaconda-distribution/perseverance-skills/blob/main/sections/02_Package_building/01_How_tos/Creating_new_packages.md#creating-new-packages) and add it as a submodule to `aggregate`. **Local rebuild package:** [`allure-python`](https://github.com/AnacondaRecipes/allure-python-feedstock) - Find out if `allure-python` builds locally - We saw how to use a recipe that's not ready yet (using a commit ID (for the dependency) from a GitHub PR as the channel and putting that in a `abs.yaml` file) * * * ### Session 5 (recorded, August 28, 2023) #### Link to recording - [video recording](https://anaconda.zoom.us/rec/share/y1X7rYyS7ZZWAwpJdNN_k6AYFfv_m3BK1WtsWjP7pJ3EiT2CiLvKWxPquZNU4RcK.r4a_Zz--FiSt5vZ0) - Passcode: 2c*jrNs+ #### Participants - Lorenzo Pirritano (Perseverance) #### What Happened / What Was Learned? **Featured package:** [`behave`](https://github.com/AnacondaRecipes/behave-feedstock) - Picked up from where we left things last time with [Jira ticket for `behave`](https://anaconda.atlassian.net/browse/PKG-2635) - I try to build locally `behave`, and I find out that it requires `parse` and `parse_type`, both not on `defaults`. - I follow the [procedure](https://github.com/anaconda-distribution/perseverance-skills/blob/main/sections/02_Package_building/01_How_tos/Creating_new_packages.md#creating-new-packages) for both and add them as submodules to `aggregate`. - There is a [PR open](https://github.com/AnacondaRecipes/behave-feedstock/pull/2) for this, waiting for all the dependencies to be built and uploaded. - I open two Jira tickets for each of them: - https://anaconda.atlassian.net/browse/PKG-2695 - https://anaconda.atlassian.net/browse/PKG-2696 - I proceed to build one by one `parse` and `parse_type` **Featured package:** [`parse`](https://github.com/AnacondaRecipes/parse-feedstock) - This has no blockers and it builds locally - The [PR is here](https://github.com/AnacondaRecipes/parse-feedstock/pull/2) and it will need 2 approvals - [Jira Ticket](https://anaconda.atlassian.net/browse/PKG-2695) **Featured package:** [`parse_type`](https://github.com/AnacondaRecipes/parse_type-feedstock) - This package depends on `parse` as well: I can build it locally but on a PR it will fail until `parse` will be deployed. - [Jira Ticket](https://anaconda.atlassian.net/browse/PKG-2696) - [Open PR](https://github.com/AnacondaRecipes/parse_type-feedstock/pull/2), it fails as it requires `parse` to be deployed. **Local rebuild package:** [`behave`](https://github.com/AnacondaRecipes/behave-feedstock) - The build succeed because all the dependencies are found locally, I can now try to build again [`allure-python`](https://github.com/AnacondaRecipes/allure-python-feedstock) **Local rebuild package:** [`allure-python`](https://github.com/AnacondaRecipes/allure-python-feedstock) - The build fails because [`pytest-bdd`](https://github.com/AnacondaRecipes/pytest-bdd-feedstock) is missing - On the next session I will need to add it to the `aggregate` repo as submodule and build it. - I create a [Jira Ticket](https://anaconda.atlassian.net/browse/PKG-2697) as a starting point for the next time. * * * ### Session 4 (August 18, 2023) #### Participants - Lorenzo Pirritano (Perseverance) - Ken Odegard (Ursa) - Bianca Henderson (Ursa) ### What Happened / What Was Learned? **Featured package:** [`nose2`](https://github.com/conda-forge/nose2-feedstock) - [ ] [[PKG-2590] nose2 0.13.0 #2](https://github.com/AnacondaRecipes/nose2-feedstock/pull/2) <--- this PR will get merged - Each PR tends to have: - Link to theJira ticket - dev-url - conda-forge - dependencies - Speed of build can sometimes depend on the time of day (morning EU time tends to be faster since US folks are asleep) - [Uploading packages to `default` (instructions)](https://github.com/anaconda-distribution/perseverance-skills/blob/main/sections/02_Package_building/01_How_tos/Uploading_packages_to_defaults.md) - `nose2` is done (its [Jira ticket](https://anaconda.atlassian.net/browse/PKG-2590) set to `Done`, and also this step could be automated with a hook after the artifacts are verified to be online). - Going back to the `allure-python-feedstock` with this [Jira ticket](https://anaconda.atlassian.net/browse/PKG-2589) - We fork the feedstock from `conda-forge` and add it as [new submodule to aggregate](https://github.com/AnacondaRecipes/allure-python-feedstock). - We fix the `meta.yaml` divergences from `conda-forge` with the linter (`conda lint -v allure-python-feedstock`). Mainly Python build things (`setuptools`, `wheels`), Python version selectors, the `about` section. - It's a multioutput recipe so there's a bit of editing to be done. - The dependency `nose2` is not available for `py3.11`, we need to keep this in mind for later. - We try to build and we discover that `behave>=1.2.5` is another dependency that is not on Anaconda `default`. - Again, same procedure: create a [Jira ticket](https://anaconda.atlassian.net/browse/PKG-2635) for `behave`, add it as a dependency to the [allure-python Jira ticket](https://anaconda.atlassian.net/browse/PKG-2589) - Find out what is the latest version of `behave` that we could build and update the Jira ticket with it (`1.2.6`). - Anyway the changes to the `allure-python` are meaningful, commit a draft, push and open a [draft PR](https://github.com/AnacondaRecipes/allure-python-feedstock/pull/1) **Question:** What parts do you see being the most automatable? - Prefect stuff: * Prefect creates a folder with the name of the feedstock * The directory in there corresponds to the PR number * All of the folders within that PR number directory needs to reflect that we can build things for all platforms - Updating `aggregate` once the package is on `main` (could maybe be done with a Makefile) - Double checking dependencies; potentially these could be extracted from the `setup()` call in `setup.py`, and in other ways for other common building systems, at least for Python packages. - Creating feedstock repos, with all their specific permissions etc. * * * ### Session 3 (August 11, 2023) #### Participants - Lorenzo Pirritano (Perseverance) - Ken Odegard (Ursa) - Bianca Henderson (Ursa) ### What Happened / What Was Learned? **Featured package:** [`allure-python`](https://github.com/conda-forge/allure-python-feedstock) - This package is multi-output so Lorenzo will need to check first that all the dependencies are on Anaconda `default` channel - Linter that gets run locally is the same one that gets run on the CI system (Lorenzo runs the linter in a dedicated `anaconda-linter` conda env) - Conda-build is run from `aggregate` directory because it picks up the `conda_build_config.yaml` configuration from there and propogates it - We want to use specific versions of `numpy` so those are pinned to each Python version; the versions are picked up via a `{{ numpy }}` var (the [`zip_key`](https://docs.conda.io/projects/conda-build/en/latest/resources/variants.html#special-variant-keys) variant key was discussed in relation to this; it works similarly to [`zip()` in Python](https://docs.python.org/3/library/functions.html#zip)) - `imagecodecs` is a C++ library, which tends to have lots of specific dependencies that are required for building - Before saying that a package is available on all of the platforms, we need to make sure all of the dependencies are also available for all; in order to do this we have to force checking of compatibility by indicating that it's built for `noarch` - Related to "noarch": `s390x` and `ppc64le` (PowerPC) are finicky platforms to build for - [Conda `noarch` package blog post](https://www.anaconda.com/blog/condas-new-noarch-packages) - Packages are built on the [Prefect](https://discourse.prefect.io/) system; scripts from Zeus (via `ssh`) need to be run - When there are dependency issues to resolve, a package is considered "on hold" while the SE figures out the dependency resolution - Each feedstock PR requires 2 approvals - 4 - 5 packages are usually being worked on per week, on average (per each Perseverance SE) - Lorenzo referred to [`02_Package_building/01_How_tos/Creating_new_packages.md`](https://github.com/anaconda-distribution/perseverance-skills/blob/main/sections/02_Package_building/01_How_tos/Creating_new_packages.md) in the [Perseverance Skills repo](https://github.com/anaconda-distribution/perseverance-skills/tree/main) **Featured package:** [`nose2`](https://github.com/conda-forge/nose2-feedstock) - [Jira ticket `PKG-2590`](https://anaconda.atlassian.net/browse/PKG-2590) - [nose2 0.10.0 PR](https://github.com/AnacondaRecipes/nose2-feedstock/pull/2) under review - Feedstock PR branch name has the ticket number (in the case above, `PKG-2590`) in it and that auto-updates Jira - [`nose2 0.10.0` upstream](https://github.com/nose-devs/nose2/tree/0.10.0) * * * ### Session 2 (August 3, 2023) #### Participants - Ryan Keith (Perseverance) - Ken Odegard (Ursa) - Bianca Henderson (Ursa) ### What Happened / What Was Learned? **Featured package:** [`pandas`](https://github.com/AnacondaRecipes/pandas-feedstock) We watched Ryan run some tests using `pandas` and `pytest` (`pandas` doesn't allow `pytest` to run tests directly on it, `pytest` has to be invoked via `pandas`) - If internals submodule is compiled, that's why `pandas` has to run the tests vs package builders being able to run `pytest` independently - Could `conftest` be copied over so that `pyproject.toml` can in theory pick it up? * * * ### Session 1 (July 20, 2023) #### Participants - Ryan Keith (Perseverance) - Ken Odegard (Ursa) - Bianca Henderson (Ursa) ### What Happened / What Was Learned? We watched Ryan build a package: - First, Ryan checked out [aggregate](https://github.com/AnacondaRecipes/aggregate) - Packaging requests can be made in multiple ways (Slack channel (which creates a Jira ticket), sometimes PRs and PR comments) - CLI GitHub manual: https://cli.github.com/manual/