---
tags:
- bot-meeting-notes
---

# conda-forge bot/infra meetings

first friday of every month at 14:00 UTC

## 2024-11-01

Attendees:

- WV
- MB
- SC
- KZ

Agenda:

- MRB:
- Next steps for bot (report link: https://drive.google.com/file/d/15KzxbqXieC99N_I0I0K4MlTU-wy9BbTC/view?usp=sharing)
1. Do push updates for PR json + labels
2. Build out queue of PRs to be made
3. Make PRs as events come in via the queue
- Potential new infra brought up by WV
- https://temporal.io/
- https://workers.cloudflare.com/
- Moving automerge out of gha for individual feedstocks to central location

- SC:
- Combining Pulumi w/ 1password for secret managing
- Need to figure out necessary scopes per service
- What would be a good candidate for the first workflow to try?
- CIRUN_API_KEY

- JRG(via KZ):
- Zulip vote has passed, invite link incoming

- WV:
- Python bindings for rattler are underway,
- wishes welcome
- rattler-solver for conda part of that (collab w/ JRG)
- parallel linking is a separate issue
- sharded repodata is being rolled out on prefix

## 2024-10-04

Attendees:

- YT
- JRG
- WV
- KZ

Agenda:

- WV:
- Version bumping for recipe v1
- Duplicate version/URL detection in the bot?
- YT: update_sources will write the detected version to countyfair, then the migrator jobs will take process the new versions, and then the autotick will execute the new migration data.
- WV: The version is kept, but the detected URL is not. Maybe it would be nice to keep around to simplify the process.
- KZ: Some docs at https://github.com/regro/cf-scripts/?tab=readme-ov-file#current-bot-jobs-and-structure
- YT: Some opportunities for optimization of some steps
- KZ: Consider constraints of resources
- KZ:
- Going through the backlog of broken version updates.
- JRG:
- Raw links in status page now available for broken version updates and migration details.
- Many errors can be explained with a handful

## 2024-09-06

Attendees:

- JRG
- YT
- MB
- ED
- KZ

Agenda:

- YT:
- Big queue of PRs awaiting reviews about adding abstractions for local debugging
- Suggested order is marked as (Git-n) with n={1..}
- Example: https://github.com/regro/cf-scripts/pull/2948
- Working on adding integration tests to the bot to at least make sure that it can start and work. Tests are basic but sufficient to detect 'obvious' issues that required reverts in the past.
- ED: Where do the mockup repos live?
- YT: Fake orgs and repos set up separately, suffixed with -staging.
- MRB: Can we have the repos embedded in regro itself to reduce the spread of orgs?
- YT: Trying to avoid interfering with the production environment.
- MRB: With one more bot account it should be ok.
- Is it helpful to add review comments after a PR has been merged?
- MRB: Yes, please do.
- JRG:
- Almost done with the CZI grant, ~70h left. Will prioritize Pulumi secret management. Any other tasks should be bumped?
- MRB:
- Yes for secrets
- Mirroring: Wolf's prototype and layers, are full .conda artifacts stored?
- JRG: Yes
- Delegate to original source for non-mirrorable content
- DO use redirections if possible
- "OCI middleware to provide static-like downloads so conda can use it blindly"
- MRB to KZ: how's the conda-build Lazy Index PR going?
- https://github.com/conda/conda-build/pull/5363
- KZ: Almost ready, just received feedback to improve code coverage.
- MRB: Performance in conda-forge feedstocks?

## 2024-07-02

Attendees:

- JRG
- ED
- WV
- NM
- KZ
- NM

Agenda:

- WV: Working on rattler-build integrations across conda-smithy, staged-recipes
- JRG: Main challenges?
- WV: A lot of moving pieces across regro / conda-forge, unclear relationship, API is basically frozen, dependencies on `main`. Trying to modernize projects as they are being worked on (ruff, pyproject.toml).
- e.g., https://github.com/conda-forge/webservices-dispatch-action
- ED: do we have dev standards for these things? i.e., if app, use lock files, data model = pydantic, etc.
- pathlib, not os.path

- WV: sovereign tech fund "reduce tech debt" PRs have kind of rotted because they didn't get merged in. Also, some of the PRs feel like automated tooling changes (os.path -> Pathlib)
- cool diagrams they have at https://github.com/neighbourhoodie/conda-smithy/wiki/bots-bots-bots. would be nice if they can contribute those.

## 2024-06-07

Attendees:

- JRG
- MRB
- YT
- KZ
- NM

Agenda:

- YT: Local debugging of migration runners so it's not as tied to the Github infra.
- Git logic needs to be decoupled from the other logic so this refactor is possible. Using OO approach to replace the Git logic with a DryRun logic. Some parts already merged, but still some more needed.
- Question about maintenance bus factors, and how to help with the bot tasks / reviews.
- MRB: Help welcome, take into account maintenance model. Sometimes quick patches are needed instead of a perfect solution.
-
- NM: rattler-build in conda-smithy
- Received PR comments from Isuru. What to do after the (eventual) approval. Staged-recipes integration?
- MRB recommends starting integration tests with an existing feedstock.

- KZ: Problem in openmpi5 migration? https://github.com/conda-forge/libnetcdf-feedstock/pull/192#issuecomment-2149885644
- Documented at https://github.com/regro/cf-scripts/issues/2694 and should be fixed by now

- MRB: https://github.com/regro/conda-forge-feedstock-check-solvable can use rattler under the hood now

- JRG: More verbose logs coming soon
- https://github.com/conda-forge/conda-forge-ci-setup-feedstock/pull/310
- https://github.com/conda-forge/conda-smithy/pull/1947
- https://github.com/conda/conda-build/pull/5344/

- MRB: Status on conda-build using the new conda Index object?
- KZ: In progress, check https://github.com/conda/conda-build/pull/5363

## 2024-04-05

Attendees:
- JRG
- MRB
- YT
- KZ

Agenda:

- YT: Changes in pydantic schema PR
- JRG: Summary of progress in managing Github teams with Pulumi
- MRB: Same but for repo secrets

## 2024-03-01

Attendees:
- JRG
- MRB
- KZ

Agenda:

- JRG: new pydantic docs for conda-forge.yml
- JRG: move orga docs to community
- JRG+KZ: reorg docs into new Diataxis-based framework
- MRB: what to do with libcfgraph
- JRG: conda-forge-metadata still defaults to libcfgraph; switch to OCI+anaconda combo
- MRB: need to check who else is using that repo
- JRG: Python import maps still unavailable elsewhere; we need the files-to-artifact DB somewhere.
- (We talked about mongodb free tier, or a regular SQL server.)

## 2024-02-02

Attendees
- JRG
- MRB
- Bela Stoyan

Agenda:

- JRG: Add latest updates for conda-forge.org/packages
- MRB: Use atom feed for /feedstocks commits: https://github.com/conda-forge/feedstocks/commits/main.atom
- JRG: [Github Actions runners for Apple Silicon](https://github.com/conda-forge/conda-forge.github.io/issues/1781)
- JRG: [Make Travis CI opt-in](https://github.com/conda-forge/conda-forge.github.io/issues/1875)
- Bela: Debugging improvements for autotickbot.
- https://github.com/regro/cf-scripts/pull/2131
- What kind of problems are usually found while working with the bot?
- Recipes not being parsable by the bot. We try to expose some of these errors to the status page but they don't always make it. Need local debugging by running the bot code against the problematic recipe.
- Improvement: test the recipe parsing.
- Global state of the bot gets in the way of graph traversal
- Very tricky to debug
- Debugging via notebook in regro/cf-graph-countyfair
- "I didn't get a version update"
- Parsing issues
- Upstream changed source URL and we don't know about that novel schema
- Checking URLs for existence (e.g. new version) sometimes fails with curl, but not wget, or requests
- Solvability of the dependencies. It got a little better, but still not perfect.
- A few times a month, they need to reset some metadata in the bot database. If you accidentally push a tag with confusing syntax that is mistakingly taken as too far in the future, the bot will remember it even if deleted in the repo, so folks have to manually submit a PR in the graph to clear it up.
- This would be nice to have as an admin-requests PR-workflow.
- What's the biggest time sink while debugging issues in cf-script?
- Setting up, updating the repos and metadata, loading up the repo
- 6 months ago, an admin command allows to update version in-feedstock. Pulls bot code. Note that the webservices server is too slow to run the code itself, so instead it makes GHA run it on behalf of the user with a dance of webhooks and repository_dispatch payloads. See:
- https://github.com/conda-forge/conda-forge-webservices/blob/1c7f17858912ea94f0f2db4106d740152b5ea6f8/conda_forge_webservices/commands.py#L385
- https://github.com/conda-forge/conda-forge-webservices/blob/1c7f17858912ea94f0f2db4106d740152b5ea6f8/conda_forge_webservices/commands.py#L835
- https://github.com/conda-forge/webservices-dispatch-action/blob/e17bfa1/webservices_dispatch_action/version_updater.py
- Idea: devise automation to debug a migration via issue-title admin-command. e.g. `@conda-forge-admin, please debug migration XYZ`
- Wolf: rattler-build and conda-forge: availability soonish.
- MRB: 1st question: Define availability. Just compatibility for recipe.yaml, or also bot integrations.
- WV: Tier 1 integration would be to recognize recipe.yaml presence to run for recipe.yaml. Autotick-bot compatibility would come after.
- MRB: Editing the YAML is easier, yes, but there's a lot of hardcoded logic and assumptions about meta.yaml and might be not trivial. Needs to ensure that both formats receive the same output.
- WV: Render output format will also be different. The names in-tarball will be different too (instead of info/meta.yaml it'll be info/recipe.yaml, and so on).
- MRB: There's some Jinja replacement done for Jinja functions like `compiler()` stuff, which are ignored in the output graph. Also some complexity with run_exports and pinnings.
- MRB: Rendering happens for each .ci_support/ file, so there's some solving happening, but partially.
- The code is difficult to follow in some places.
- Python bindings would come very handy
- MRB: Does rattler understand recipe/conda_build_config.yaml files?
- WV: Yes, to a degree. No selectors are taken into account. Considering changing it to have a 2-stage parsing to get platforms and some fields only, and the rest will come in a second phase.
- MRB: This might get in the way of conda-smithy rerendering processing and migrators. If not compatible, it might require two versions of the same migrator for each format. It'd be easier to have rattler understand CBC as we move forward.
- CBC.yaml can run arbitrary Python in the selectors (it's an eval after all). We need to be careful with things like [os.environ calls](https://github.com/search?q=org%3Aconda-forge%20path%3Arecipe%2Fconda_build_config.yaml%20os.environ&type=code)
- MRB: Idea: Generate meta.yaml from recipe.yaml to make the bots work initially.

## 2024-01-12

Attendees:
- JRG
- MRB
- Hind
- Uwe

Agenda:

- JRG: web updates
- OCI: mamba in process
- Uwe: rattler build work is coming, lots of bot changes, will be working on it

## 2023-11-03

Attendees:
- JRG
- MRB

Agenda:

- JRG: Status of GPU CI ([demo](https://github.com/conda-forge/cf-autotick-bot-test-package-feedstock/pull/466)) and TOS.
- JRG: [Dashboard prototype](https://github.com/Quansight-Labs/cf-infra-docs/pull/12)
- Needs incidents info as JSON
- JRG: [Provenance info](https://conda-metadata-app.streamlit.app/?q=pkgs%2Fmain%2Flinux-64%2Faiohttp-3.8.5-py39h5eee18b_0.conda) in conda-metadata-app and [implementation for conda-forge](https://github.com/conda-forge/conda-smithy/pull/1577)
- MRB: libcfgraph only relied on for pypi_mappings after the run_exports refactor in regro/check-solvable
- JRG: Plans for conda-forge.org revamp?
- Ask core about opinions when a conda-forge.github.io PR is inplace
- Ok to use netlify for PR previews only
- Migrate blog as proof of concept migration (careful with URLs)
- Keep sphinx docs around for now

## 2023-09-01

Attendees:
- Jaime Rodríguez-Guerra
- Vini
- Hind
- Eric
- MRB

Agenda:
- Jaime: Potential SDG application for a Scaleway-based Apple Silicon CI. Amit is looking into it to assess whether it's feasible.
- Apple licensing
- MacStadium no public CI
- Matt: New recipe format in the bot
- Main point: Parsing recipe format to add nodes in the graph
- Dependencies on conda-build
- Might be good timing to start anew
- Good option for CZI grant (rattler-build + conda-forge?)
- Discussion about OCI and community standardization:
- Mamba has made some advances in download API
- conda can use https://github.com/conda/conda/pull/13054
- Probably good idea to use a CEP here
-

## 2023-07-06

Attendees:

- Jaime
- Matt
- Vini
- Hind

Agenda:

- [ ] (JRG) opt-in CI access control in admin-requests
- https://github.com/conda-forge/admin-requests/issues/752
- [ ] (JRG & Hind) OCI staging
- https://github.com/conda-forge/conda-forge-ci-setup-feedstock/pull/208#issuecomment-1573586190 (and following comments)
- Matt: Does the repodata allow for external URLs?
- Jaime: We need to think about this because it looks like it does not and I thought it did.
- [ ] (Vini) Questions about conda-forge-database
- Matt: Better use a flat table than the recursive tree structure
- [ ] (Matt) Updates on the MongoDB work on cf-graph-countyfair
- Uses https://github.com/regro/cf-scripts/blob/master/conda_forge_tick/lazy_json_backends.py and https://github.com/regro/cf-scripts/blob/master/conda_forge_tick/lazy_json_backups.py
- Only thing left: backups somewhere else. Free MongoDB only gives us ~6min of backups.
- Each snapshot is 25MB.

## 2023-03-03

Attendees:

- Jaime
- Eric
- Matt
- Vini

Notes:

- depfinder: problem is because we're fnmatching on a bunch of stuff that includes the full path
- https://github.com/conda/grayskull/issues/441 this one?
-
- DEBUG depfinder:reports.py:168 ******found ignore match for name: /usr/share/miniconda3/envs/test/conda-bld/praw_1677855233037/work/praw/reddit.py
- problem 1: we have an env named "test" so the fnmatch for "*/test/*"
- problem 2:

bot background:
- biggest thing that's happened in bot land in the last 2 years are that the bot will now update recipe with dependencies from grayskull. or depfinder actually. it does both. grayskull is broken cause grayskull has a bug. marcelo hasn't had a chance to fix it. things related to carets in the deps which conda doesn't understand. and thene xtra requirements syntax for pip-land. so that's all broken. depfinder broke which we think we just figured out.
- https://github.com/conda/grayskull/issues/441 this one?
- second big change - slow process - started before bot meetings stopped - completed it in th past week . refactored a lot of the internals of the bots datamodel sothat everythign can run in parallel. much more responsive now cause its on cron jobs. before that was done - vini you were working on the version stuff - if you made any changes to the json blob - one json blob per feedstock - this comes from the github API - what happened was that if you wanted to run diff parts of the bot in parallel they'd all edit the file simultaneously and the whole thing woudl fall apart. that single PR-json file and extracted the different elements into different files. jobs that collect versions, jobs that update PRs, jobs that update fedstocks and remake metadata and some other jobs, those ar ethe main three, all of those jobs can run in parallel now. what happens now is that when the bot starts up it grabs the current state. does use a consistent data model at a given time. if we were to switch to a DB and we allowed updates on the fly, the bot would potentially see a changing data model as it runs. now you just DL a git repo so it all works. that's cut out a huge amount of latency, even though CF has grown by ~2x. now the bot runs on a cycle that takes ~2 hours which is down from 3 which is down from even 4-5 hours not too far ago.
- V: some stuff is lingering around. graph, date. another job can start while the prev job is running.
- M: bot always had this deadman switch. current job kicks off the next one, as opposed to the cron job. sometimes that step of kicking off the next one fails so we have a cron job that sees if one is running and starts one if its not. switched to GH concurrency feature in their workflows and use the github cron syntax for kicking things off. now you'll see dup jobs coming in. that cron job kicks off a new one but concurrency of 1 means that it doesn't actually run. bot status badges dont mean anything anymore. when github does the cancellation, it doesn't know that we actually are working fine.
- M: With Eric's help we moved things back into one repo instead of splitting into two. cf-autotick-bot was merged with cf-scripts
- V: knew we had migrations and admin-requests. noticed on [J infra source of truth](https://cf-infra-docs.netlify.app/docs/reference/infrastructure/) there is a [curator app](https://cf-infra-docs.netlify.app/docs/reference/infrastructure/services/#bot-accounts). is this new?
- M: Security discussion. not transcribing.
-
- M: Bot latency is the biggest issue right now. "Half a day" in practice (30 mins min, plus the CI job itself - 2h-ish). Cutting the time down is the biggest impact on users. Splitting tasks in separate jobs could also work (version updates different from migrations).
- V: How are tasks splitted in version updates?
- M: Some modulo operation on the feedstock name hash (?)
- M: chaindb still used for dependency tracking (?), which requires xonsh. If we remove that, we can remove xonsh completely.

## 2023-04-07

**cf-graph-countyfair** - working data for the bot. what versions its found, all of the nodes, etc.

**libcfgraph** - indexes all of our artifacts so we can access them quickly
- really useful for a whole bunch of tasks / stuff

https://hackmd.io/63JsxC1GSWWXX3IClySLtA

move cf-graph into the DO database
- postgres

easiest place to start is the version update loop
next place to go is PR update loop
- issue here is that sometimes the bot has to go all the way down
- two layers of PR data
- list of prs that the bot has made
- github pr info something something etag
-

Action items:

- [ ] Webservices split: strategy
- [ ] Swap one bot task to use a postgres backend