Conda-forge - guidance on releasing on conda-forge first and on contributions from companies?

Note: a different but somewhat related discussion was had in 23 Mar 2022 meeting, see "Helping people help conda-forge" in the minutes.

There's two questions that would be great to get some guidance on from the conda-forge core team, and ideally that guidance gets documented in the form of a process or policy on conda-forge.org:

  1. Is it possible to plan for release of a (new) package on conda-forge on a given timeline, and if so how should this be approached?
  2. How can companies (or other entities) contribute effort in a way that makes them "good citizens" within the conda-forge community?

Context

There's a couple of recent cases that come to mind where there is friction because some contributors, core team members or companies want to get something done within conda-forge, and (for different reasons) that did not go as smoothly as it should have:

Part of the friction here is that conda-forge relies largely on volunteer effort, and one cannot expect from volunteers that they do things on a timeline - or at all for that matter. So when there's funding to get something done, that funding should be done in such a way that it does not negatively impact the experience of and load on volunteers. To quote a principle from NEP 48:

For proposed funded work, include paid time for others to review your work if such review is expected to be significant effort - do not just increase the load on volunteer maintainers. Rationale: we want the effect of spending funds to be positive for everyone, not just for the people getting paid. This is also a matter of fairness.

Independent of the details of what went right and wrong in the above-mentioned three cases, it is worth pointing out that these are not external companies that just want to get something done and not give back - both NVIDIA and Quansight employ conda-forge core team members (John, Jaime, Vini right now) and give them day job time to work on conda-forge, to help sustain/grow conda-forge and "be good citizens". In individual contributions from a funded contributor it's clear that a volunteer may be the knowledge holder and be a key reviewer - this is hard to avoid, but should then be offset by other contributions and reviews (or in other appropriate ways - even a financial or in-kind donation to conda-forge can be considered) so the net result is positive for everyone involved.

Q1: releasing on a given timeline

Say entity X develops a new package. And it decides that it would like to release it on conda-forge, rather than elsewhere (e.g., PyPI, its own conda channel, or a custom download from a website). It may have good reasons for preferring conda-forge, for example it has dependencies that are not available on PyPI. Entity X plans some publicity, and decides the release date will be 1 Dec 2022. Questions:

  • Is such a "conda-forge first" plan desirable from the conda-forge team's perspective? Or would you rather see a release elsewhere first?
  • If this is desirable, what is needed in order to do so? Does it need to end up in a CEP?

For a predictable timeline, a key issue is response times. This is nontrivial to get right. Is there an agreed-upon expected response for a direct question or announced merge plan right now in conda-forge? A few examples that come to mind are:

There's at least 3 possible outcomes I can think of:

  1. Guarantees are impossible and we don't want to really think about it, so please release on your own channel or on PyPI.
  2. We do want to write down a process, but anyone can still be blocking by disagreeing and then be silent for an unspecified amount of time - so it's hard in practice.
  3. Here is a process / steps to follow, and you need to get a core team member to work with you (i.e., employ them or make some other arrangement with them), otherwise planning is not possible.

It may also be useful to consider different types of potential issues. If something is really hairy technically, then it is clearly more difficult to plan for a certain timeline and stick to it. The Qt case was on that side of the spectrum: building Qt is about as hard as it gets, so major changes there are difficult to review and there's not enough reviewer bandwidth to go around. The OmnisciDB to Heavydb rename is on the other end of this spectrum: it was a straight-up rename, the code already worked. Any reviewer comments on technical aspects can be synced back to the original feedstock and integrated there by the feedstock maintainers. So the rename itself should be reviewable easily, and there's little risk of breaking anything or introducing new issues that are hard to deal with later on.

Q2: How can companies (or other entities) contribute effort in a way that makes them "good citizens" within the conda-forge community?

This is more abstract, so a shorter write-up. Let me propose a few basic principles:

  • Companies wanting to actively participate in, and build on top of, conda-forge should aim to be "good citizens" - give more to the community than you take from it.
  • For code contributions that are funded (= part of someone's day job), aim to contribute also 20% of the value of that funded time as a financial or in-kind contribution to conda-forge.
    • Example: if you let a developer do 3 months of full-time work sending PRs to conda-forge, and that had a total cost of $50,000 (salary + overhead), then also contribute $10,000 or the equivalent in Azure cloud credits that conda-forge can use.
  • Companies should aim to employ one or more core team member and give them time to work on topics that conda-forge as a whole needs, in addition to the things a company may need for itself.

To connect this back to Q1: if a company does all these things, it should hopefully be possible to build on top of conda-forge in a sustainable way, and plan for projects or products that are "conda-forge first".

Comment Section

Eric D.

Everyone on the core team has the same access to everything. Anyone on core can merge anything anywhere in all of CF infrastructure. This is a privilege that your peers decided to give you when we all voted you to become part of the core team. To me, the short version is:

  1. use your best judgement and be willing to, in hindsight, have done the wrong thing
  2. if things go sideways, make it a priority to fix the issue, or ask for help fixing the issue.
  3. when we discuss it in a core meeting, show good intent. As long as the intent is good and it was, in hindsight, a bad idea, I'm going to support your choice to take action.
  4. we'll codify the things that went wrong as a "maybe dont do this again in the future"
  5. If someone on core is regularly "abusing" their powers and regularly breaking things in a way that makes CF look bad then we might decide to take action up to and including removal of your core status.

From #1 above, if you're not willing to, in hindsight, have done the wrong thing, then you should start planning months ahead of time to ensure you have plenty of time to carefully walk forward without ever doing "the wrong thing".

Select a repo