--- tags: [post-mortem] --- # 2023-05 Release Issues and Notes for `conda 23.5.0` and `conda-build 3.25.0` **Authors:** - Bianca Henderson (@bhenderson, Anaconda) - Ken Odegard (@kenodegard, Anaconda) - Srivas Venkatesh (@svenkatesh, Anaconda) This document discusses the various details related to preparations and problems surrounding the May 2023 releases of [ 23.5.0](https://github.com/conda/conda/issues/12625) and [conda-build](https://github.com/conda/conda-build/issues/4859). * * * ## Raising More Awareness About Upcoming Releases We need to reach out to relevant parties about any new upcoming release, _especially_ conda-forge maintainers. [`conda/communications` repo](https://github.com/conda/communications) should be used for reviewing announcement wording. We did a good job with Perseverance Team, but we could be better about communication with Ursa and other conda folks. Things need to be more visible somehow. We should take more advantage of the Release channel on Matrix. * * * ## `FileNotFound` Error During Tests, Related to New `repodata` / `repofetch` Work This issue was first described by Jaime in [this Slack thread](https://anaconda.slack.com/archives/C049N05A2R2/p1684416551587939) as a possible race condition on Thursday, May 18th. ### Resolved? ✅ Turns out the problem wasn’t a race condition, but a cache directory path that changes during unit tests but wouldn't change during a normal conda run. [Removing the memoization](https://github.com/conda/conda/pull/12690/commits/91751559d3925836465b9da4cb5b60b954831dc1) after checking that it is not called very many times (~12 times) during a run seems to have fixed the problem! ### Discussions Jaime filed a [corresponding pull request](https://github.com/conda/conda-libmamba-solver/pull/211) in the [`conda-libmamba-solver` repo](https://github.com/conda/conda-libmamba-solver) on Thursday, May 18th. Test failure examples can be found in [this run](https://github.com/conda/conda-libmamba-solver/actions/runs/5014258672/jobs/8988306902). Daniel filed a corresponding conda issue titled **["Conda can try to use a missing temporary file when `repodata.json` was not found, and `{}` was returned instead"](https://github.com/conda/conda/issues/12689)** in order to address this problem. The description in the issue states: > The new `RepodataFetch` feature tries to replace the index file with a temporary file, for locking reasons, in `gateways/repodata/RepodataFetch.fetch_latest()`. However the underlying `RepoInterface` might not write to that file, *e.g.* when it returns `{}` in place of a `404` or possibly in "offline" mode (which uses the cache even when it would normally be expired). The problem, at its root, seems to be about resetting the context (*i.e.* we can’t memoize the `cache` directory) * * * ## Automated / Manual Testing Plan for Releases @sven6002 is in the release "jumpseat" and is working on a manual testing plan for releases. [An initial issue](https://github.com/conda/conda/issues/12663) was filed. A pairing session occurred on Thursday, May 18th where Bianca, Ken, and Srivas manually tested all [changelog items](https://github.com/conda/conda/pull/12685/files#diff-06572a96a58dc510037d5efa622f9bec8519bc1beab13c9f251e97e657a9d4ed) in the latest canary builds which included all of the latest work (JLAP, deprecations, etc.) on both Mac/Linux and Windows. ### Mac/Linux & Windows - Manual Testing #### Deprecations Warning Testing * Mark `conda_env` dependencies listed below as pending deprecation. (#12492) * `get_pip_version` ``` PYTHONWARNINGS=d python >>> import conda_env.pip_util >>> conda_env.pip_util.get_pip_version() >>> <stdin>:1: DeprecationWarning: conda_env.pip_util.get_pip_version is deprecated and will be removed in 23.9. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/svenkatesh/miniconda3/envs/scratch/lib/python3.11/site-packages/conda/deprecations.py", line 68, in inner return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ TypeError: get_pip_version() missing 1 required positional argument: 'prefix' ``` * `pip_util.PipPackage` ``` >>> c<stdin>:1: PendingDeprecationWarning: conda_env.pip_util.PipPackage is pending deprecation and will be removed in 24.3. ``` * `pip_util.installed` ``` <stdin>:1: PendingDeprecationWarning: conda_env.pip_util.installed is pending deprecation and will be removed in 24.3. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/svenkatesh/CondaCode/conda/conda/deprecations.py", line 68, in inner return func(*args, **kwargs) TypeError: installed() missing 1 required positional argument: 'prefix' ``` * `_canonicalize_name` ``` >>> conda_env.pip_util._canonicalize_name() <stdin>:1: PendingDeprecationWarning: conda_env.pip_util._canonicalize_name is pending deprecation and will be removed in 24.3. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/svenkatesh/CondaCode/conda/conda/deprecations.py", line 68, in inner return func(*args, **kwargs) TypeError: _canonicalize_name() missing 1 required positional argument: 'name' ``` * `add_pip_installed` ``` >>> conda_env.pip_util.add_pip_installed() <stdin>:1: PendingDeprecationWarning: conda_env.pip_util.add_pip_installed is pending deprecation and will be removed in 24.3. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/svenkatesh/CondaCode/conda/conda/deprecations.py", line 68, in inner return func(*args, **kwargs) TypeError: add_pip_installed() missing 2 required positional arguments: 'prefix' and 'installed_pkgs' ``` * `env.load_from_directory` ``` >>> conda_env.env.load_from_directory() <stdin>:1: PendingDeprecationWarning: conda_env.env.load_from_directory is pending deprecation and will be removed in 24.3. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/svenkatesh/CondaCode/conda/conda/deprecations.py", line 68, in inner return func(*args, **kwargs) TypeError: load_from_directory() missing 1 required positional argument: 'directory' ``` * Mark `conda.auxlib.packaging` for deprecation in 24.3.0. (#12509) ``` >>> import conda.auxlib.packaging /Users/svenkatesh/CondaCode/conda/conda/auxlib/packaging.py:65: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives from distutils.command.build_py import build_py <stdin>:1: PendingDeprecationWarning: conda.auxlib.packaging is pending deprecation and will be removed in 24.3. Use a modern build systems instead, see https://packaging.python.org/en/latest/tutorials/packaging-projects#creating-pyproject-toml for more details. ``` * Mark `conda.testing` as pending deprecation. (#12676) * `integration.get_conda_list_tuple` ``` >>> conda.testing.integration.get_conda_list_tuple() <stdin>:1: PendingDeprecationWarning: conda.testing.integration.get_conda_list_tuple is pending deprecation and will be removed in 24.3. Use __conda.core.prefix_data.PrefixData().get()__ instead. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/svenkatesh/CondaCode/conda/conda/deprecations.py", line 68, in inner return func(*args, **kwargs) TypeError: get_conda_list_tuple() missing 2 required positional arguments: 'prefix' and 'package_name` ``` * `encode_for_env_var` ```>>> conda.testing.encode_for_env_var() <stdin>:1: PendingDeprecationWarning: conda.testing.encode_for_env_var is pending deprecation and will be removed in 24.3. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/svenkatesh/CondaCode/conda/conda/deprecations.py", line 68, in inner return func(*args, **kwargs) TypeError: encode_for_env_var() missing 1 required positional argument: 'value' ``` * `integration.temp_chdir` ``` >>> conda.testing.integration.temp_chdir() <stdin>:1: PendingDeprecationWarning: conda.testing.integration.temp_chdir is pending deprecation and will be removed in 24.3. Use `monkeypatch.chdir` instead. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/svenkatesh/CondaCode/conda/conda/deprecations.py", line 68, in inner return func(*args, **kwargs) File "/Users/svenkatesh/.conda/envs/conda/lib/python3.10/contextlib.py", line 281, in helper return _GeneratorContextManager(func, args, kwds) File "/Users/svenkatesh/.conda/envs/conda/lib/python3.10/contextlib.py", line 103, in __init__ self.gen = func(*args, **kwds) TypeError: temp_chdir() missing 1 required positional argument: 'target_dir' ``` * Mark `python -m conda_env` as pending deprecation. Use `conda env` instead. (#12492) ``` /Users/svenkatesh/.conda/envs/conda/lib/python3.10/site-packages/conda/auxlib/packaging.py:68: __DeprecationWarning__: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives from distutils.command.build_py import build_py usage: __main__.py [-h] {create,export,list,remove,update,config} ... positional arguments: {create,export,list,remove,update,config} create Create an environment based on an environment definition file. If using an environment.yml file (the default), you can name the environment in the first line of the file with 'name: envname' or you can specify the environment name in the CLI command using the -n/--name argument. The name specified in the CLI will override the name specified in the environment.yml file. Unless you are in the directory containing the environment definition file, use -f to specify the file path of the environment definition file you want to use. export Export a given environment list List the Conda environments remove Remove an environment update Update the current environment based on environment file config Configure a conda environment options: -h, --help Show this help message and exit. conda commands available from other packages (legacy): env ``` * Mark `python -m conda_env.cli.main` as pending deprecation. Use `conda env` instead. (#12492) ``` /Users/svenkatesh/.conda/envs/conda/lib/python3.10/runpy.py:86: PendingDeprecationWarning: __main__ is pending deprecation and will be removed in 24.3. Use `conda env` instead. exec(code, run_globals) usage: main.py [-h] {create,export,list,remove,update,config} ... positional arguments: {create,export,list,remove,update,config} create Create an environment based on an environment definition file. If using an environment.yml file (the default), you can name the environment in the first line of the file with 'name: envname' or you can specify the environment name in the CLI command using the -n/--name argument. The name specified in the CLI will override the name specified in the environment.yml file. Unless you are in the directory containing the environment definition file, use -f to specify the file path of the environment definition file you want to use. export Export a given environment list List the Conda environments remove Remove an environment update Update the current environment based on environment file config Configure a conda environment options: -h, --help Show this help message and exit. conda commands available from other packages: doctor - A subcommand that displays environment health report conda commands available from other packages (legacy): env ``` ### Enhancements * Add `conda doctor` subcommand plugin. (#474) ``` (scratch) C:\Users\svenkatesh\miniconda3>conda doctor -h C:\Users\svenkatesh\miniconda3\envs\scratch\Lib\site-packages\conda\gateways\connection\adapters\ftp.py:20: DeprecationWarning: 'cgi' is deprecated and slated for removal in Python 3.13 import cgi usage: conda doctor [-h] [-v] [-n ENVIRONMENT | -p PATH] Display a health report for your environment. options: -h, --help show this help message and exit -v, --verbose generate a detailed environment health report Target Environment Specification: -n ENVIRONMENT, --name ENVIRONMENT Name of environment. -p PATH, --prefix PATH Full path to environment location (i.e. prefix). ``` * Add Python 3.11 support. (#12256) ``` active environment : base active env location : C:\Users\svenkatesh\miniconda3\envs\scratch shell level : 2 user config file : C:\Users\svenkatesh\.condarc populated config files : conda version : 23.5.18.0.dev53 conda-build version : not installed python version : 3.11.3.final.0 virtual packages : __archspec=1=x86_64 __win=0=0 base environment : C:\Users\svenkatesh\miniconda3\envs\scratch (writable) conda av data dir : C:\Users\svenkatesh\miniconda3\envs\scratch\etc\conda conda av metadata url : None channel URLs : https://repo.anaconda.com/pkgs/main/win-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/r/win-64 https://repo.anaconda.com/pkgs/r/noarch https://repo.anaconda.com/pkgs/msys2/win-64 https://repo.anaconda.com/pkgs/msys2/noarch package cache : C:\Users\svenkatesh\miniconda3\envs\scratch\pkgs C:\Users\svenkatesh\.conda\pkgs C:\Users\svenkatesh\AppData\Local\conda\conda\pkgs envs directories : C:\Users\svenkatesh\miniconda3\envs\scratch\envs C:\Users\svenkatesh\.conda\envs C:\Users\svenkatesh\AppData\Local\conda\conda\envs platform : win-64 user-agent : conda/23.5.18.0.dev53 requests/2.29.0 CPython/3.11.3 Windows/10 Windows/10.0.22621 administrator : False netrc file : None offline mode : False ``` * Add `conda list --reverse` to return a reversed list of installed packages. (#11954) ``` List installed packages in a conda environment. Options: positional arguments: regex List only packages matching this regular expression. options: -h, --help Show this help message and exit. --show-channel-urls Show channel urls. Overrides the value given by `conda config --show show_channel_urls`. --reverse List installed packages in reverse order. -c, --canonical Output canonical names of packages only. -f, --full-name Only search for full names, i.e., ^<regex>$. --full-name NAME is identical to regex '^NAME$'. --explicit List explicitly all installed conda packages with URL (output may be used by conda create --file). --md5 Add MD5 hashsum when using --explicit. -e, --export Output explicit, machine-readable requirement strings instead of human-readable lists of packages. This output may be used by conda create --file. -r, --revisions List the revision history. --no-pip Do not include pip-only installed packages. Target Environment Specification: -n ENVIRONMENT, --name ENVIRONMENT Name of environment. -p PATH, --prefix PATH Full path to environment location (i.e. prefix). Output, Prompt, and Flow Control Options: --json Report all output as json. Suitable for using conda programmatically. -v, --verbose Use once for info, twice for debug, three times for trace. -q, --quiet Do not display progress bar. conda commands available from other packages: doctor - A subcommand that displays environment health report ``` #### Run by automation test cases on GitHub-CI * Switch from `setup.py` to `pyproject.toml` and use [Hatchling](https://pypi.org/project/hatchling/) for our build system. (#12509) * Optimize which Python modules get imported during `conda activate` calls to make it faster. (#12550) * Add `conda_cli` fixture to replace `conda.testing.helpers.run_inprocess_conda_command` and `conda.testing.integration.run_command`. (#12592) * Add `tmp_env` fixture to replace `conda.testing.integration.make_temp_env`. (#12592) * Add `path_factory` fixture to replace custom prefix logic like `conda.testing.integration._get_temp_prefix` and `conda.testing.integration.make_temp_prefix`. (#12592) * Refactor the way that the `Activator` classes are defined in `conda/activate.py`. (#12627) * Warn about misconfiguration when signature verification is enabled. (#12639) ### Bug fixes #### Run by automation test cases on GitHub-CI * `conda clean` no longer fails if we failed to get the file stats. (#12536) * Provide fallback version if `conda.deprecations.DeprecationHandler` receives a bad version. (#12541) * Avoid `TypeError` when non-string types are written to the index cache metadata. (#12562) * `conda.core.package_cache_data.UrlsData.get_url` no longer fails when `package_path` has `.conda` extension. (#12516) ### Windows Specific Testing with `conda-canary` * Ensure the default value for `defaults` includes `msys2` when `context.subdir` is `win-*` on non-Windows platforms. (#12555) * Stop pre-converting paths to Unix style on Windows in `conda.sh`, so that they are prefix replaceable upon installation, which got broken by #12509. It also relies on `cygpath` at runtime, which all `msys2`/`cygwin` bash versions on Windows should have available. (#12627) ![msys2](https://hackmd.io/_uploads/S1liVOHSh.png) ### Docs * Change the README example from IPython Notebook and NumPy to PyTorch. (#12579) ![PyTorch](https://hackmd.io/_uploads/HJU56WFSh.png) * Discuss options available to properly configure mirrored channels. (#12583, #12641) * Add `flake8-docstrings` to `pre-commit`. (#12620) ![Other Features](https://hackmd.io/_uploads/r1_7SuHHh.png) ### Other * Update retry language in flexible solve and `repodata` logs to be less ominous. (#12007) ![PYTORCH](https://hackmd.io/_uploads/SJh0VdrSh.png) * Improve `repodata` / `subdir_data` programming interface (#12521). Index cache metadata has changed to `.info.json` to better align with the [draft CEP](https://github.com/conda-incubator/ceps/pull/48). Improve cache locking when using `jlap`. Improve `jlap` logging. (#12572) ``` conda config --show-sources vi .condarc experimental: - jlap conda search -c conda-forge tensorflow ``` Before ![Before](https://hackmd.io/_uploads/rJJ-SuHSn.png) After ![after](https://hackmd.io/_uploads/Sk9fBdSSn.png) * Format with `black` and replaced `pre-commit`'s `darker` hook with `black`. (#12554) * Format with `isort` and add `pre-commit` `isort` hook. (#12554) * Add functional tests around conda's content trust code. (#11805) - __Automation test case running on GitHub-CI__ * Enable `flake8` checks that are now handled by `black`. (#12620) ![Other Features](https://hackmd.io/_uploads/r1_7SuHHh.png) ### Resolved? Yes ## 23.5.0 (2023-05-17) ### Recommendations [list of stuff that would be good to do] ### Alternatives [list of alternative options, if applicable]