---
tags: [post-mortem]
---
# 2022-05 release incident of conda 4.13.0/conda-build 3.21.{8,9}

Authors: Jannis Leidel (@jezdez, Anaconda), Mark Harfouche(@hmaarrfk)

Between May 26 and May 31, users of conda-build and conda version 4.13.0 may have experienced errors when attempting to build new packages. The most visible effect of this incident was the spurious failures of conda-forge's build infrastructure on windows. The issue has now been resolved with the release of conda-build version 3.21.9. In this brief, we describe the technical details in hopes avoiding such widespread incidences in the future.

## Incident

The new [conda 4.13.0 release](https://github.com/conda/conda/releases/tag/4.13.0) removed a lot of legacy Python support code, mostly related to Python 2.7 which has not been supported for a while.
This change was done over a [number of commits](https://github.com/conda/conda/pull/11364) and reviewed closely to reduce the fallout.

Despite checking the conda-build code base thoroughly for imports of the removed code in conda, an inline import for the `conda.common.compat.itervalues` function [slipped through the cracks](https://github.com/conda/conda-build/blob/1ed8da062f123e1fbca604df26b064b267a3d094/conda_build/utils.py#L2085).

A few isolated things that happened:

- The code path with the broken import was NOT successfully triggered as part of the conda or conda-build continuous integration.
- The conda 4.13.0 release landed on defaults first, just before a long weekend, without a conda-forge release yet.
- The feedstock maintenance tool conda-smithy (heavily used by conda-forge feedstocks) used flexible channel priority and mixed channels (defaults and conda-forge).
- Incompatible flags between conda & mamba completely breaking mamba (silent exit) which made debugging reports against mamba particularly hard.
- Repodata patching was needed to restrict to previous conda, but mamba continues to respect an unpatched `repodata_record.json`.

As a result, the incident cascaded into the conda-forge community where it was first discovered (among many others) when Numpy tried to build a new version. The Numpy release process got the conda 4.13.0 release from defaults, without prior code review/testing through the regular conda-forge workflow, which is their preferred stack.

It stands to reason that conda-forge's build system (e.g. the conda feedstock) also wouldn't have caught the conda-build bug, even if conda 4.13.0 would have been released there first.

## Mitigation

1. Fix the underlying import error in https://github.com/conda/conda-build/pull/4482
2. Patching repodata for defaults and conda-forge so that older versions of conda-build don’t unexpectedly break with conda 4.13.0 (remains in place)
3. Patch conda-build 3.21.8 in conda-forge with bugfix
4. Releasing conda-build 3.21.9 with the bugfix
5. Build conda-build 3.21.9 to defaults and conda-forge
7. Updating conda-smithy to use strict channel priorities
8. Released mamba 0.24.0 and micromamba 0.24.0 that fixes incompatibilities with conda

## Recommendations

- conda and conda-build **releases in tandem** to reduce time window for potential cascading incidents
- strict, easy to understand **version compatibility** between conda and conda-build
- **continuously pay down tech debt** for conda and friends (in contrast to huge code removals) to reduce surface area for compatibility issues
- new regular and predictable **release process** (also see CEP drafts for: [release schedule](https://github.com/conda-incubator/ceps/pull/26), [conda version](https://github.com/conda-incubator/ceps/pull/25) and [deprecation policy](https://github.com/conda-incubator/ceps/pull/27))
- release coordination of conda and conda-build (and mamba?) for both defaults AND conda-forge (**new cross-organizational conda release team**, led by rotating release manager)
- **new nightly integration testing** between conda, conda-build and mamba canary releases to catch bugs earlier
- **strict channel policy for conda community channels** to reduce fallout on maintainers
- reenable tests in conda and conda-build feedstock to **increase test coverage during builds**

## Timeline <small>(UTC)</small>

| Date | Time | Action | Link |
| -----| ---- | ------ | ---- |
| 2022-05-19 | 16:57 | conda 4.13.0 tagged on Github, https://github.com/conda/conda/releases/tag/4.13.0 |
| 2022-05-19 | 17:02 | conda 4.13.0 feedstock PR opened, https://github.com/AnacondaRecipes/conda-feedstock/pull/6 |
| 2022-05-19 | 17:06 | conda 4.13.0 package build requested for defaults internally at Anaconda | |
| 2022-05-19 | 23:18 | conda 4.13.0 autotick-bot opens PR for conda-forge, https://github.com/conda-forge/conda-feedstock/pull/165 |
…
| 2022-05-26 | 17:07 | Anaconda-internal status request for conda 4.13.0 package build | |
| 2022-05-26 | 23:09 | conda 4.13.0 released on defaults | |
…
| 2022-05-27 | 01:09 | Bug filed in conda-build regarding a regression following the removal of Python 2.7 specific code in conda which was used in conda-build, https://github.com/conda/conda-build/issues/4481 |
| 2022-05-27 | 02:08 | Bugfix PR opened, https://github.com/conda/conda-build/pull/4482 |
| 2022-05-27 | 06:15 | Numpy feedstock maintainers on conda-forge try to build new version, blocked by conda-build bug, https://github.com/conda-forge/numpy-feedstock/pull/272 |
| 2022-05-27 | 06:47 | Bugfix filed in conda-forge feedstock by Numpy maintainers, https://github.com/conda-forge/conda-build-feedstock/pull/176 |
| 2022-05-27 | 08:17 | Bug filed in conda-forge repo for cascading install issue https://github.com/conda-forge/conda-forge.github.io/issues/1762) finding that due to a channel | priority |
| 2022-05-27 | 14:09 | Bugfix PR approved, https://github.com/conda/conda-build/pull/4482 |
| 2022-05-27 | 14:36 | Bugfix filed in conda-forge feedstock, https://github.com/conda-forge/conda-build-feedstock/pull/176 |
| 2022-05-27 | 15:54 | Bugfix PR merged, https://github.com/conda/conda-build/pull/4482 |
| 2022-05-27 | 16:19 | conda-build release PR opened https://github.com/conda/conda-build/pull/4483) and draft release created |
| 2022-05-27 | 17:04 | Repodata patch merged for conda-forge to prevent the issue from spreading, https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/pull/271 |
| 2022-05-28 | 03:10 | Repodata patch for defaults merged to prevent the issue from spreading, https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/pull/271 |
...
| 2022-05-30 | 16:12 | First patch for conda-smithy to switch conda-forge feedstock to strict channel priority opened, replaced by second patch, https://github.com/conda-forge/conda-smithy/pull/1630 |
| 2022-05-30 | 19:12 | Second PR opened for conda-smithy to switch conda-forge feedstock to strict channel priority, https://github.com/conda-forge/conda-smithy/pull/1631 |
| 2022-05-30 | 19:30 | Second PR merged for conda-smithy to switch conda-forge feedstock to strict channel priority,, https://github.com/conda-forge/conda-smithy/pull/1631 |
| 2022-05-31 | 10:09 | conda-build 3.21.9 tagged on GitHub, https://github.com/conda/conda-build/releases/tag/3.21.9 |
| 2022-05-31 | 11:31 | conda-build 3.21.9 defaults feedstock PR opened, https://github.com/AnacondaRecipes/conda-build-feedstock/pull/15 |
| 2022-05-31 | 15:36 | conda-build 3.21.9 defaults feedstock PR merged, https://github.com/AnacondaRecipes/conda-build-feedstock/pull/15 |
| 2022-05-31 | 11:48 | conda-build 3.21.9 conda-forge feedstock PR opened, https://github.com/conda-forge/conda-build-feedstock/pull/177 |
| 2022-05-31 | 13:22 | conda-build 3.21.9 conda-forge feedstock PR merged, https://github.com/conda-forge/conda-build-feedstock/pull/177 |