2022-05 release incident of conda 4.13.0/conda-build 3.21.{8,9}

--- tags: [post-mortem] --- # 2022-05 release incident of conda 4.13.0/conda-build 3.21.{8,9} Authors: Jannis Leidel (@jezdez, Anaconda), Mark Harfouche(@hmaarrfk) Between May 26 and May 31, users of conda-build and conda version 4.13.0 may have experienced errors when attempting to build new packages. The most visible effect of this incident was the spurious failures of conda-forge's build infrastructure on windows. The issue has now been resolved with the release of conda-build version 3.21.9. In this brief, we describe the technical details in hopes avoiding such widespread incidences in the future. ## Incident The new [conda 4.13.0 release](https://github.com/conda/conda/releases/tag/4.13.0) removed a lot of legacy Python support code, mostly related to Python 2.7 which has not been supported for a while. This change was done over a [number of commits](https://github.com/conda/conda/pull/11364) and reviewed closely to reduce the fallout. Despite checking the conda-build code base thoroughly for imports of the removed code in conda, an inline import for the `conda.common.compat.itervalues` function [slipped through the cracks](https://github.com/conda/conda-build/blob/1ed8da062f123e1fbca604df26b064b267a3d094/conda_build/utils.py#L2085). A few isolated things that happened: - The code path with the broken import was NOT successfully triggered as part of the conda or conda-build continuous integration. - The conda 4.13.0 release landed on defaults first, just before a long weekend, without a conda-forge release yet. - The feedstock maintenance tool conda-smithy (heavily used by conda-forge feedstocks) used flexible channel priority and mixed channels (defaults and conda-forge). - Incompatible flags between conda & mamba completely breaking mamba (silent exit) which made debugging reports against mamba particularly hard. - Repodata patching was needed to restrict to previous conda, but mamba continues to respect an unpatched `repodata_record.json`. As a result, the incident cascaded into the conda-forge community where it was first discovered (among many others) when Numpy tried to build a new version. The Numpy release process got the conda 4.13.0 release from defaults, without prior code review/testing through the regular conda-forge workflow, which is their preferred stack. It stands to reason that conda-forge's build system (e.g. the conda feedstock) also wouldn't have caught the conda-build bug, even if conda 4.13.0 would have been released there first. ## Mitigation 1. Fix the underlying import error in https://github.com/conda/conda-build/pull/4482 2. Patching repodata for defaults and conda-forge so that older versions of conda-build don’t unexpectedly break with conda 4.13.0 (remains in place) 3. Patch conda-build 3.21.8 in conda-forge with bugfix 4. Releasing conda-build 3.21.9 with the bugfix 5. Build conda-build 3.21.9 to defaults and conda-forge 7. Updating conda-smithy to use strict channel priorities 8. Released mamba 0.24.0 and micromamba 0.24.0 that fixes incompatibilities with conda ## Recommendations - conda and conda-build **releases in tandem** to reduce time window for potential cascading incidents - strict, easy to understand **version compatibility** between conda and conda-build - **continuously pay down tech debt** for conda and friends (in contrast to huge code removals) to reduce surface area for compatibility issues - new regular and predictable **release process** (also see CEP drafts for: [release schedule](https://github.com/conda-incubator/ceps/pull/26), [conda version](https://github.com/conda-incubator/ceps/pull/25) and [deprecation policy](https://github.com/conda-incubator/ceps/pull/27)) - release coordination of conda and conda-build (and mamba?) for both defaults AND conda-forge (**new cross-organizational conda release team**, led by rotating release manager) - **new nightly integration testing** between conda, conda-build and mamba canary releases to catch bugs earlier - **strict channel policy for conda community channels** to reduce fallout on maintainers - reenable tests in conda and conda-build feedstock to **increase test coverage during builds** ## Timeline <small>(UTC)</small> | Date | Time | Action | Link | | -----| ---- | ------ | ---- | | 2022-05-19 | 16:57 | conda 4.13.0 tagged on Github, https://github.com/conda/conda/releases/tag/4.13.0 | | 2022-05-19 | 17:02 | conda 4.13.0 feedstock PR opened, https://github.com/AnacondaRecipes/conda-feedstock/pull/6 | | 2022-05-19 | 17:06 | conda 4.13.0 package build requested for defaults internally at Anaconda | | | 2022-05-19 | 23:18 | conda 4.13.0 autotick-bot opens PR for conda-forge, https://github.com/conda-forge/conda-feedstock/pull/165 | … | 2022-05-26 | 17:07 | Anaconda-internal status request for conda 4.13.0 package build | | | 2022-05-26 | 23:09 | conda 4.13.0 released on defaults | | … | 2022-05-27 | 01:09 | Bug filed in conda-build regarding a regression following the removal of Python 2.7 specific code in conda which was used in conda-build, https://github.com/conda/conda-build/issues/4481 | | 2022-05-27 | 02:08 | Bugfix PR opened, https://github.com/conda/conda-build/pull/4482 | | 2022-05-27 | 06:15 | Numpy feedstock maintainers on conda-forge try to build new version, blocked by conda-build bug, https://github.com/conda-forge/numpy-feedstock/pull/272 | | 2022-05-27 | 06:47 | Bugfix filed in conda-forge feedstock by Numpy maintainers, https://github.com/conda-forge/conda-build-feedstock/pull/176 | | 2022-05-27 | 08:17 | Bug filed in conda-forge repo for cascading install issue https://github.com/conda-forge/conda-forge.github.io/issues/1762) finding that due to a channel | priority | | 2022-05-27 | 14:09 | Bugfix PR approved, https://github.com/conda/conda-build/pull/4482 | | 2022-05-27 | 14:36 | Bugfix filed in conda-forge feedstock, https://github.com/conda-forge/conda-build-feedstock/pull/176 | | 2022-05-27 | 15:54 | Bugfix PR merged, https://github.com/conda/conda-build/pull/4482 | | 2022-05-27 | 16:19 | conda-build release PR opened https://github.com/conda/conda-build/pull/4483) and draft release created | | 2022-05-27 | 17:04 | Repodata patch merged for conda-forge to prevent the issue from spreading, https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/pull/271 | | 2022-05-28 | 03:10 | Repodata patch for defaults merged to prevent the issue from spreading, https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/pull/271 | ... | 2022-05-30 | 16:12 | First patch for conda-smithy to switch conda-forge feedstock to strict channel priority opened, replaced by second patch, https://github.com/conda-forge/conda-smithy/pull/1630 | | 2022-05-30 | 19:12 | Second PR opened for conda-smithy to switch conda-forge feedstock to strict channel priority, https://github.com/conda-forge/conda-smithy/pull/1631 | | 2022-05-30 | 19:30 | Second PR merged for conda-smithy to switch conda-forge feedstock to strict channel priority,, https://github.com/conda-forge/conda-smithy/pull/1631 | | 2022-05-31 | 10:09 | conda-build 3.21.9 tagged on GitHub, https://github.com/conda/conda-build/releases/tag/3.21.9 | | 2022-05-31 | 11:31 | conda-build 3.21.9 defaults feedstock PR opened, https://github.com/AnacondaRecipes/conda-build-feedstock/pull/15 | | 2022-05-31 | 15:36 | conda-build 3.21.9 defaults feedstock PR merged, https://github.com/AnacondaRecipes/conda-build-feedstock/pull/15 | | 2022-05-31 | 11:48 | conda-build 3.21.9 conda-forge feedstock PR opened, https://github.com/conda-forge/conda-build-feedstock/pull/177 | | 2022-05-31 | 13:22 | conda-build 3.21.9 conda-forge feedstock PR merged, https://github.com/conda-forge/conda-build-feedstock/pull/177 |