2022-05 release incident of conda 4.13.0/conda-build 3.21.{8,9}

Authors: Jannis Leidel (@jezdez, Anaconda), Mark Harfouche(@hmaarrfk)

Between May 26 and May 31, users of conda-build and conda version 4.13.0 may have experienced errors when attempting to build new packages. The most visible effect of this incident was the spurious failures of conda-forge's build infrastructure on windows. The issue has now been resolved with the release of conda-build version 3.21.9. In this brief, we describe the technical details in hopes avoiding such widespread incidences in the future.

Incident

The new conda 4.13.0 release removed a lot of legacy Python support code, mostly related to Python 2.7 which has not been supported for a while.
This change was done over a number of commits and reviewed closely to reduce the fallout.

Despite checking the conda-build code base thoroughly for imports of the removed code in conda, an inline import for the conda.common.compat.itervalues function slipped through the cracks.

A few isolated things that happened:

• The code path with the broken import was NOT successfully triggered as part of the conda or conda-build continuous integration.
• The conda 4.13.0 release landed on defaults first, just before a long weekend, without a conda-forge release yet.
• The feedstock maintenance tool conda-smithy (heavily used by conda-forge feedstocks) used flexible channel priority and mixed channels (defaults and conda-forge).
• Incompatible flags between conda & mamba completely breaking mamba (silent exit) which made debugging reports against mamba particularly hard.
• Repodata patching was needed to restrict to previous conda, but mamba continues to respect an unpatched repodata_record.json.

As a result, the incident cascaded into the conda-forge community where it was first discovered (among many others) when Numpy tried to build a new version. The Numpy release process got the conda 4.13.0 release from defaults, without prior code review/testing through the regular conda-forge workflow, which is their preferred stack.

It stands to reason that conda-forge's build system (e.g. the conda feedstock) also wouldn't have caught the conda-build bug, even if conda 4.13.0 would have been released there first.

Mitigation

1. Fix the underlying import error in https://github.com/conda/conda-build/pull/4482
2. Patching repodata for defaults and conda-forge so that older versions of conda-build don’t unexpectedly break with conda 4.13.0 (remains in place)
3. Patch conda-build 3.21.8 in conda-forge with bugfix
4. Releasing conda-build 3.21.9 with the bugfix
5. Build conda-build 3.21.9 to defaults and conda-forge
6. Updating conda-smithy to use strict channel priorities
7. Released mamba 0.24.0 and micromamba 0.24.0 that fixes incompatibilities with conda

Recommendations

• conda and conda-build releases in tandem to reduce time window for potential cascading incidents
• strict, easy to understand version compatibility between conda and conda-build
• continuously pay down tech debt for conda and friends (in contrast to huge code removals) to reduce surface area for compatibility issues
• new regular and predictable release process (also see CEP drafts for: release schedule, conda version and deprecation policy)
• release coordination of conda and conda-build (and mamba?) for both defaults AND conda-forge (new cross-organizational conda release team, led by rotating release manager)
• new nightly integration testing between conda, conda-build and mamba canary releases to catch bugs earlier
• strict channel policy for conda community channels to reduce fallout on maintainers
• reenable tests in conda and conda-build feedstock to increase test coverage during builds

Timeline (UTC)