2023-07 Release Issues and Notes for `conda 23.7.x` and `conda-build 3.26.0`

**Authors:** - Bianca Henderson (@bhenderson, Anaconda) - Ken Odegard (@kenodegard, Anaconda) This document discusses the various details related to preparations and problems surrounding the July 2023 releases of [conda 23.7.0](https://github.com/conda/conda/pull/12901) and [conda-build 23.6.0](https://github.com/conda/conda-build/pull/4937). ### **The major theme of this release seems to be: "Whack-a-mole with `argparse`"** The `--json` and `--debug` flags being broken, plus the `--no-plugins` flag and `argparse.REMAINDER` issues were *not* caught by CI, which highlights the need for a discussion on our testing system. Continuing the practice of manual testing during not just during release cycles but also during regular PR reviews is crucial. We also need a list of "standard" commands + flags to run to ensure basic functionality is as expected. * * * ## 🚨 POST-RELEASE ISSUES 🚨 ## `--json` and `--debug` flags are broken Reported by Jaime: https://anaconda.slack.com/archives/C049N05A2R2/p1690392027841329 Issue filed by Jaime: https://github.com/conda/conda/issues/12928 **Related reference info:** - [`ArgumentParser.parse_known_args(args=None, namespace=None)`](https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.parse_known_args) - [Intermixed parsing](https://docs.python.org/3/library/argparse.html#intermixed-parsing) - [So many `argparse` issues!](https://github.com/orgs/python/projects/5) - [Make `argparse.SUPPRESS` work as an argument "dest"](https://bugs.python.org/issue20430) - [`default` (argparse docs)](https://docs.python.org/3/library/argparse.html#default) Ken and Jannis filed a PR to change tests so that CI catches problems like this: - [Fix `conda_cli` to use `main_subshell` instead of mocking it #12929](https://github.com/conda/conda/issues/12928) ### Resolved? Yes 👍 * * * ## `--no-plugins` Flag Shown to Not Be Operational During Manual Testing This issue was first described by Bianca in [this Slack thread](https://anaconda.slack.com/archives/C049N05A2R2/p1689720646403979) on Tuesday, July 18th. A [GitHub issue containing details of the problem](https://github.com/conda/conda/issues/12903) was filed and a [corresponding PR with a fix](https://github.com/conda/conda/pull/12904) was created by Jannis. ### Resolved? Yes 👍 ### Discussions @kenodegard [says](https://anaconda.slack.com/archives/C049N05A2R2/p1689773345199789?thread_ts=1689720646.403979&cid=C049N05A2R2): > Long term I’m wondering if we should consider a more radical 2-phase parser, phase-1 parses “universal” arguments (e.g., `--no-plugins`, `--json`, `--verbose`, perhaps even `--name`/`--prefix`), after phase-1 we disable plugins based on `context.no_plugins` and generate the entire parser (with subcommands) and parse the remaining arguments (phase-2). The “cool” thing with this is we can universally parse standard options even for third party plugins. These universal arguments are already exposed through the `context`. Take `--name`/`--prefix` as an example, in `doctor` we have to include the `--name`/`--prefix` argument in the subparser even though plugin devs should only call `context.target_prefix` in their executable. If this is parsed in phase-1 then subcommand plugins will only need to worry about parsing arguments unique to their subcommand in phase-2. > > (I don’t want to rush the 2-phase idea into this release since there are unknowns and the idea needs more time to develop, e.g., i almost have a working prototype but am still struggling with getting the legacy subcommands to work correctly.) * * * ## [Previous Release Item](https://hackmd.io/hrfvniBGSiKVLtOwh7b-5w#Raising-More-Awareness-About-Upcoming-Releases): Raising More Awareness About Upcoming Releases Bianca attended the [conda-forge core meeting (which takes biweekly) on July 12th](https://hackmd.io/@conda-forge/B1Z-4HLKh) to directly bring more awareness to the upcoming releases. * * * ## `argparse.REMAINDER` Does Not Function as Advertised As reported by @kenodegard: After installing both RCs for [conda](https://anaconda.org/conda-canary/conda) and [conda-build](https://anaconda.org/conda-canary/conda-build/files?channel=rc-conda-build-3.26.x) I get: ``` $ conda build -h usage: conda build [-h] Build conda packages from a conda recipe. options: -h, --help Show this help message and exit. ``` After having reviewed the `argparse` source code and done some more research I’ve found that this bug in `argparse.REMAINDER`’s functionality has been known for quite some time and is why the feature was undocumented back in 2020, see https://github.com/python/cpython/issues/61252. In other words, this is a hard problem and for conda’s purposes we need to go back to the drawing board and come up with some other way to identify arguments that need to be passed along to subcommand plugins. #### Corresponding issues: - [Subcommand arguments are incorrectly parsed #12906](https://github.com/conda/conda/issues/12906) - [Legacy blurb in `conda --help` vanishes when other arguments are present #12907](https://github.com/conda/conda/issues/12907) @jezdez responded: We can salvage this with the workaround mentioned here: > “Note that this means we can fix the bug simply by, whenever the `cmdname` subparser is invoked and the `cmdname` subparser uses `argparse.REMAINDER`, automatically adding an imaginary first positional argument to the subparser and inserting that imaginary first positional argument into the stream before parsing the arguments to `cmdname`. #### Jannis's proposed fix: [Use workaround for use of `argparse.REMAINDER` for subcommands #12908](https://github.com/conda/conda/pull/12908) ❌ @thath says: As I was explaining yesterday to Ken and Bianca, I think that if we always insist on calling `parse_args` for each subcommand, we are going to be doomed. The easiest way is figuring out how to bypass invoking it so that the subcommands handle it themselves. #### Travis's proposed fix (pushing to Jannis's `remainder-fix` PR branch above): [Alternative approach to fixing plugin subcommand line parsing #12909](https://github.com/conda/conda/pull/12909) ❌ #### Ken's fix: [Implement 2-phase parser with greedy subparsers #12910](https://github.com/conda/conda/pull/12910) ✅ ### Resolved? Yes 👍 * * * ## Conda-Build No Longer Works on Sirius's Infrastructure As reported by Justin Wood in the [#build-guild channel](https://anaconda.slack.com/archives/C05B9MPGR4Y): https://anaconda.slack.com/archives/C05B9MPGR4Y/p1690816023772479 Current workaround is to use `conda-build` vs `conda build` command (i.e., the non-subcommand based version of conda-build)