--- tags: [post-mortem] --- # 2023-03 Release Issues for conda 23.3.0 and conda-build 3.24.0 Authors: Ken Odegard (@kenodegard, Anaconda); Daniel Holth (@dholth, Anaconda) In preparation for the March 2023 releases of conda and conda-build, several issues arose. These issues were brought about by three factors: the addition of new dependencies in conda 23.3.0, the import side-effects of conda, and a lack of canary testing. Additionally, we have been facing difficulties with downstream tests when `libsolv>0.7.23` is utilized. ## Isolated Mode Conda-build is still affected by historical artifacts from a time when it was part of conda's source code and used to package itself. This is apparent in how conda is called within conda-build using subprocesses and as a module (`python -m conda`) rather than as an executable. The main problem with invoking conda in this manner is that the current working directory (which contains the source code) is added to the `PYTHONPATH`, causing the new source code to be executed instead of treating it like any other package. This issue was initially identified and addressed in conda-build 3.23.0 (2022-11-16) (refer to [conda-build#4604](https://github.com/conda/conda-build/pull/4604) and [conda-build#4625](https://github.com/conda/conda-build/pull/4625)) by introducing an environment variable (`_CONDA_BUILD_ISOLATED_MODE=1`) that could be set before running conda-build. However, setting environment variables prior to running conda-build has proven difficult due to the build systems used for both the `main` and `conda-forge` channels. This issue remains unresolved. ### Recommendations - Define and customize all recipes exclusively in `meta.yaml` (see [conda-build#4816](https://github.com/conda/conda-build/pull/4816)) instead of requiring a special conda-build invocation. - Push all source code down into a `src` directory (see [conda#12542](https://github.com/conda/conda/issues/12542)) or modify the recipe to check out conda in a subdirectory ([conda-feedstock#204](https://github.com/conda-forge/conda-feedstock/pull/204)). ## `packaging` Dependency Conda 23.3.0 introduced deprecation helper functions that relied on `packaging`. These helper functions were imported in `conda/__init__.py` to deprecate `conda.another_to_unicode`. As a result, `packaging` became an import side-effect that suffered from the same isolated mode issue discussed earlier: ```pycon Traceback (most recent call last): File "/opt/conda/lib/python3.10/runpy.py", line 187, in _run_module_as_main mod_name, mod_spec, code = _get_module_details(mod_name, _Error) File "/opt/conda/lib/python3.10/runpy.py", line 146, in _get_module_details return _get_module_details(pkg_main_name, error) File "/opt/conda/lib/python3.10/runpy.py", line 110, in _get_module_details __import__(pkg_name) File "/home/conda/feedstock_root/build_artifacts/conda_1678979738526/work/conda/__init__.py", line 10, in <module> from .deprecations import deprecated File "/home/conda/feedstock_root/build_artifacts/conda_1678979738526/work/conda/deprecations.py", line 9, in <module> from packaging.version import parse, Version ModuleNotFoundError: No module named 'packaging' ``` The conda-build 3.24.0 release included `packaging` as a dependency (see [conda-build#4443](https://github.com/conda/conda-build/pull/4443)). After reviewing our options, we decided to release conda-build first, followed by conda. This approach ensured that the dependency would be resolved during the build process, thereby avoiding any need for patching. Although the release of conda-build 3.24.0 allowed us to move forward with the release of conda 23.3.0 on `conda-forge`, we encountered issues when packaging conda for the `main` channel. This was due to stale images that were not refreshed until the following week, causing further delays. ### Recommendations - Relying on ordered releases makes us dependent on the specifics of the various build systems. Do not rely on ordered releases (i.e., don't release one in order to release the other). - Continue working towards only building packages in isolated mode. (No code from a newly-packaged conda should run during conda-build's packaging phase. Packaged code should run during the test phase.) ## `conda.auxlib.packaging` and `conda.__version__ == None` We encountered a problem where `conda.__version__` would be `None` during certain parts of the build process, causing a parsing error: ```pycon Traceback (most recent call last): File "/opt/conda/lib/python3.10/runpy.py", line 187, in _run_module_as_main mod_name, mod_spec, code = _get_module_details(mod_name, _Error) File "/opt/conda/lib/python3.10/runpy.py", line 146, in _get_module_details return _get_module_details(pkg_main_name, error) File "/opt/conda/lib/python3.10/runpy.py", line 110, in _get_module_details __import__(pkg_name) File "/home/conda/feedstock_root/build_artifacts/conda_1679676579485/work/conda/__init__.py", line 10, in <module> from .deprecations import deprecated File "/home/conda/feedstock_root/build_artifacts/conda_1679676579485/work/conda/deprecations.py", line 266, in <module> deprecated = DeprecationHandler(__version__) File "/home/conda/feedstock_root/build_artifacts/conda_1679676579485/work/conda/deprecations.py", line 29, in __init__ self._version = parse(version) File "/opt/conda/lib/python3.10/site-packages/packaging/version.py", line 52, in parse return Version(version) File "/opt/conda/lib/python3.10/site-packages/packaging/version.py", line 195, in __init__ match = self._regex.search(version) TypeError: expected string or bytes-like object ``` The issue was caused by `conda.auxlib.packaging.get_version` which attempted to detect the current version from `conda/.version` and `git describe`, and if both failed, it would return `None`. This has been possible for a long time but was not a problem in the past because we didn't need to parse the current version when importing from conda. To fix this, we decided to intercept any bad versions by including a patch. The patch sets the version to "0.0.0.dev0+placeholder" if a `TypeError` is raised during parsing. Here is the patch: ```diff diff --git a/conda/deprecations.py b/conda/deprecations.py index 68ca1a7ad..c514fa7e1 100644 --- a/conda/deprecations.py +++ b/conda/deprecations.py @@ -25,8 +25,10 @@ class DeprecationHandler: :param version: The version to compare against when checking deprecation statuses. """ - if not isinstance(version, Version): + try: self._version = parse(version) + except TypeError: + self._version = parse("0.0.0.dev0+placeholder") def __call__( self, ``` ### Recommendations - Stop using `setup.py` and `auxlib` for packaging (see [conda#12508](https://github.com/conda/conda/issues/12508)) ## `conda-forge`, `libsolv` pinning, and downstream tests During the conda 23.1.0 (2023-01-18) release we ran into problems with downstream tests when packaging on `conda-forge`. At the time we circumvented the issue by pinning `boa`'s dependency on `libsolv` (see [boa-feedstock#61](https://github.com/conda-forge/boa-feedstock/pull/61)). This issue reemerged as the pin added to `boa` was removed (see [boa-feedstock#69](https://github.com/conda-forge/boa-feedstock/pull/69)). The fix applied was to pin the necessary `libsolv` version in the `conda-forge` config (see [`conda-forge.yml`'s `remote_ci_setup`](https://github.com/conda-forge/conda-feedstock/pull/202#discussion_r1150394671)). ### Recommendations - Fix downstream testing for `noarch` packages (see [conda-build#4832](https://github.com/conda/conda-build/pull/4832)). ## Regression in `conda.core.sudir_data.SubdirData` A regression was identified in `conda.core.sudir_data.SubdirData` breaking `conda-index`, a package necessary to update all CDN-cached or non-anaconda.org channels. While the regression was fixed and merged on March 21st, conda 23.3.0 had already been cut and tagged on March 15th resulting in the regression being released to `main` and `conda-forge` on March 29th. The regression was severe enough to warrant a patch release so conda 23.3.1 was cut and tagged on March 29th and released to `main` and `conda-forge` on March 30th. ### Recommendations - Downstream projects need to run tests against conda-canary to detect breakages ahead of releases. ## Missing JLAP Submodule After the release of conda 23.3.0, it was discovered that the new jlap submodule was missing. The reason for this was that `auxlib` was unable to detect submodules without an `__init__.py` file, and so setuptools did not package them. (When we test from unpackaged conda, Python finds the module without issues). To remedy the situation, `__init__.py` files were added where they were missing. An initial attempt to remedy the issue was proposed via a conda recipe patch but this was rejected since additional issues needed to be addressed (see [Regression in `conda.core.sudir_data.SubdirData`](https://hackmd.io/Okuttg00RAyeXgGz13ANVg#Regression-in-condacoresudir_dataSubdirData) above). ### Recommendations - A release branch will be created a week prior to release week, and feature work will be frozen on that branch. Only bugfixes and other necessary fixes will be allowed on the release branch. During this period, the latest canary builds will be installed by the release managers (and optionally other maintainers) and daily usage will be tested. - Add a comparison between the main and release branches as part of the release checklist. Have fixes been merged? - Test packages before uploading to `main` and `conda-forge` channels. - Consider implementing an [Andon](https://en.wikipedia.org/wiki/Andon_(manufacturing)) system allowing anyone to formally request a release delay for quality purposes. - The project has switched to a setuptools alternative. This should avoid setuptools' classically error-prone package search behavior. - When doing first-party packaging, be willing to release a patch release with a new version number instead of adding a `.patch` to a conda recipe. ## Communication There continues to be confusion over who the release managers are for any given release. This confusion leads to conflicting messaging over who needed to know what and who to communicate with when issues arose. It was easy to know how to communicate with the release managers, on the #conda-release slack channel, but it was not clear whether those messages had been read or understood, or whether the release manager had gone home for the day. The chat was not an effective way to communicate that the SubdirData regression, and not the jlap packaging issue, was more necessary to include in a patch release. Furthermore, this March release happened to include both a conda and conda-build release. The choice to do a conda-build release was a last minute decision and to lower the release burden we opted to have different release managers for conda and conda-build. Unfortunately, having different release managers did not help and only made things more confusing for everyone involved. Communication was further complicated by the Europe-USA timezone differences with the US-based release managers not being available when Europe-based colleauges came online in their respective mornings and continued debugging release issues from the previous day. The release ownership structure made it more difficult for available persons to act. Stress from time pressure also made communication more difficult. ### Recommendations - Conda and conda-build will be released in tandem moving forward. - The same release managers (a.k.a., pilot and co-pilot) will be in charge of both the conda and conda-build releases. - The release managers will announce themselves at the start of the release process to both the Conda Community's public Matrix channel [#conda-release](https://matrix.to/#/#conda-release:matrix.org) and Anaconda's internal Slack channel #conda-release. ### Alternatives - The SubdirData fix was merged about a week before an earlier tagged commit, without the fix, was uploaded for distribution. A release branch would not solve this communication problem. - We will probably stabilize conda's packaging within the next few months, avoiding the difficult packaging activity that characterized the week of this release. - Merges to main tend to slow down naturally during and after a release. If we are able to do better continuous integration and testing across projects we might be able to trust the recent main branch. - Implement more formalized testing beyond run-pytest-before-merge (all agree). - Ask conda developers, Anaconda employees or users to `conda install conda-canary/label/dev::conda` for daily work? - We will always need patch releases from time to time. What is a reasonable QA cost? Can we get better at making patch releases? - Stop using threaded Slack conversations in the `#conda-release` channel.