Try   HackMD

Ursaverance Knowledge Share

Reference Materials

Session Notes

Session 7 (recorded, November 16, 2023)

Participants

  • Lorenzo Pirritano (Perseverance)
  • Bianca Henderson (Ursa)
  • Ken Odegard (Ursa)

What Happened / What Was Learned?

  • Wrapping up dependencies necessary and verifying that they are available on the relevant channel(s)
  • Identifying which steps of the build process can be automated (i.e., pressing "Y" for confirming things on the command line)

Session 6 (recorded, November 9, 2023)

Participants

  • Lorenzo Pirritano (Perseverance)
  • Bianca Henderson (Ursa)
  • Ken Odegard (Ursa)

What Happened / What Was Learned?

  • All the previous open PRs for the packages need to be reviewed by at least 2 team members and merged. The packages needs to be deployed before being used as dependencies by other packages.

Featured package: pytest-bdd

Local rebuild package: allure-python

  • Find out if allure-python builds locally
  • We saw how to use a recipe that's not ready yet (using a commit ID (for the dependency) from a GitHub PR as the channel and putting that in a abs.yaml file)

Session 5 (recorded, August 28, 2023)

Participants

  • Lorenzo Pirritano (Perseverance)

What Happened / What Was Learned?

Featured package: behave

Featured package: parse

Featured package: parse_type

  • This package depends on parse as well: I can build it locally but on a PR it will fail until parse will be deployed.
  • Jira Ticket
  • Open PR, it fails as it requires parse to be deployed.

Local rebuild package: behave

  • The build succeed because all the dependencies are found locally, I can now try to build again allure-python

Local rebuild package: allure-python

  • The build fails because pytest-bdd is missing
  • On the next session I will need to add it to the aggregate repo as submodule and build it.
  • I create a Jira Ticket as a starting point for the next time.

Session 4 (August 18, 2023)

Participants

  • Lorenzo Pirritano (Perseverance)
  • Ken Odegard (Ursa)
  • Bianca Henderson (Ursa)

What Happened / What Was Learned?

Featured package: nose2

  • [PKG-2590] nose2 0.13.0 #2 <–- this PR will get merged
  • Each PR tends to have:
    • Link to theJira ticket
    • dev-url
    • conda-forge
    • dependencies
  • Speed of build can sometimes depend on the time of day (morning EU time tends to be faster since US folks are asleep)
  • Uploading packages to default (instructions)
  • nose2 is done (its Jira ticket set to Done, and also this step could be automated with a hook after the artifacts are verified to be online).
  • Going back to the allure-python-feedstock with this Jira ticket
  • We fork the feedstock from conda-forge and add it as new submodule to aggregate.
  • We fix the meta.yaml divergences from conda-forge with the linter (conda lint -v allure-python-feedstock). Mainly Python build things (setuptools, wheels), Python version selectors, the about section.
  • It's a multioutput recipe so there's a bit of editing to be done.
  • The dependency nose2 is not available for py3.11, we need to keep this in mind for later.
  • We try to build and we discover that behave>=1.2.5 is another dependency that is not on Anaconda default.
  • Again, same procedure: create a Jira ticket for behave, add it as a dependency to the allure-python Jira ticket
  • Find out what is the latest version of behave that we could build and update the Jira ticket with it (1.2.6).
  • Anyway the changes to the allure-python are meaningful, commit a draft, push and open a draft PR

Question:
What parts do you see being the most automatable?

  • Prefect stuff:
    • Prefect creates a folder with the name of the feedstock
    • The directory in there corresponds to the PR number
    • All of the folders within that PR number directory needs to reflect that we can build things for all platforms
  • Updating aggregate once the package is on main (could maybe be done with a Makefile)
  • Double checking dependencies; potentially these could be extracted from the setup() call in setup.py, and in other ways for other common building systems, at least for Python packages.
  • Creating feedstock repos, with all their specific permissions etc.

Session 3 (August 11, 2023)

Participants

  • Lorenzo Pirritano (Perseverance)
  • Ken Odegard (Ursa)
  • Bianca Henderson (Ursa)

What Happened / What Was Learned?

Featured package: allure-python

  • This package is multi-output so Lorenzo will need to check first that all the dependencies are on Anaconda default channel
  • Linter that gets run locally is the same one that gets run on the CI system (Lorenzo runs the linter in a dedicated anaconda-linter conda env)
  • Conda-build is run from aggregate directory because it picks up the conda_build_config.yaml configuration from there and propogates it
  • We want to use specific versions of numpy so those are pinned to each Python version; the versions are picked up via a {{ numpy }} var (the zip_key variant key was discussed in relation to this; it works similarly to zip() in Python)
  • imagecodecs is a C++ library, which tends to have lots of specific dependencies that are required for building
  • Before saying that a package is available on all of the platforms, we need to make sure all of the dependencies are also available for all; in order to do this we have to force checking of compatibility by indicating that it's built for noarch
  • Related to "noarch": s390x and ppc64le (PowerPC) are finicky platforms to build for
  • Conda noarch package blog post
  • Packages are built on the Prefect system; scripts from Zeus (via ssh) need to be run
  • When there are dependency issues to resolve, a package is considered "on hold" while the SE figures out the dependency resolution
  • Each feedstock PR requires 2 approvals
  • 4 - 5 packages are usually being worked on per week, on average (per each Perseverance SE)
  • Lorenzo referred to 02_Package_building/01_How_tos/Creating_new_packages.md
    in the Perseverance Skills repo

Featured package: nose2

Session 2 (August 3, 2023)

Participants

  • Ryan Keith (Perseverance)
  • Ken Odegard (Ursa)
  • Bianca Henderson (Ursa)

What Happened / What Was Learned?

Featured package: pandas

We watched Ryan run some tests using pandas and pytest (pandas doesn't allow pytest to run tests directly on it, pytest has to be invoked via pandas)

  • If internals submodule is compiled, that's why pandas has to run the tests vs package builders being able to run pytest independently
  • Could conftest be copied over so that pyproject.toml can in theory pick it up?

Session 1 (July 20, 2023)

Participants

  • Ryan Keith (Perseverance)
  • Ken Odegard (Ursa)
  • Bianca Henderson (Ursa)

What Happened / What Was Learned?

We watched Ryan build a package:

  • First, Ryan checked out aggregate
  • Packaging requests can be made in multiple ways (Slack channel (which creates a Jira ticket), sometimes PRs and PR comments)
  • CLI GitHub manual: https://cli.github.com/manual/
Last changed by 
 
Bianca Henderson
0
416

Read more

Build Tools Team Sync Meeting Notes

XYZ (team)

Mar 13, 2025
Conda Plugins News Bulletin, Q4 '22 Edition

(updated week of Nov 14 2022) Current state of the world Plugins infrastructure has merged &#x1F389; Subcommand plugins are possible to develop and use w/ conda Virtual packages are implemented via plugins officially Subcommand tutorial repo is ready (including non-Python examples) The libmamba solver hook + plugin have been merged! We have a dedicated tab for plugins on our Planning Board

Nov 17, 2022
Read more from Bianca Henderson
Published on HackMD