Challenges of major version bumps in a monorepo

--- tags: build --- # Challenges of major version bumps in a monorepo Figuring out how to handle major version bumps can be challenging in a monorepo that publishes a main suite package (we'll assume *one* suite for the moment for slightly reduced complexity...) which uses and possibly re-exports other packages. Particularly when partner teams with ## some requirements (notes) - Mix and Match of existing and converged is important, need to use FUR / FURN + converged at the same time # conclusion - Publish single suite package `react-components` will be published this week to Nova (3-4 components) - It will re-export existing converged components packages - Caveat: we will not publicize individual packages for now (except for react-button). Readme: don't use this, use react-components. - version number: - in theory we're okay with either `0.x` or `1.0.0-alpha` - - Suite+seperate packages will lockstep version on Major releases # reasoning - Leaves us open to future decisions in either direction (indvidual packages or suite) - Will revisit once we learn more from integrating this with customers ## Background Historically with `@fluentui/react` . . . .. . . (major release procedure details, problems with major lockstep) - Not all teams will be able to upgrade right away. Especially early in a major version, there will still be a lot of fixes needed (sometimes contributed by product teams) in the previous version. - Due to cross-internal-repo dependencies (and the owning teams' varying upgrade cadence/willingness), it's highly likely that there will be a period of time where a particular app must load two major versions of OUFR/`@fluentui/react` because not all of its deps have upgraded. These are real and significant pain points we've heard about repeatedly from some of our largest partner product teams. ## Concepts ### major-lockstep Most literally this means every published package in the monorepo must be on the same major version. More often, we've used it to mean that when the suite major bumps, every published package must major bump (even if the major versions aren't identical). Advantages: - Very clear to consumers which package versions go together - Simplifies handling of all the other issues discussed here - Disadvantage: ## Existing conclusions - **The suite must major-bump if any package it re-exports major-bumps.** This is a reality of semver. - **Hotfixes to a specific previous version** (`@fluentui/whatever@x.y.z-hotfix.1`): These are very rarely needed, so **don't optimize for them**. It's okay to handle them manually (even if it's painful) if/when the need arises. ## Notes - Stop optimizing for other teams' upgrade problems in our versioning. We probably need to help with this problem, but in a different way. (David, Ken, etc work) - Historically there's been a fear that if we don't make this better, ____ (product team) will revolt. - In theory, if we're providing value, even if people do fork, they'll come back. - But we also need to be customer-focused: listen to partners' pain points and help them out Due to implementation complexity of basically all other options, we'll probably have to continue with major lockstep. Long-term, we probably don't recommend that customers use individual package names (too hard to maintain). You can't use two versions of the same name side by side. Probably need to make a new suite such as `@fluentui/react-components`. Module federation doesn't work well with re-exports. ### Getting rid of suite? #### all packages bump separately Pros: - Theoretically could reduce duplicates in multi-repo gradual upgrade scenario Cons: - Doesn't communicate clearly when people need to upgrade - ... #### all packages bump in major lock-step Pros: - easy to tell (broadly) what's compatible with what Cons: - Very very easy to get duplicate versions of e.g. button due to package manager behavior ### Getting rid of separate packages? (no/not yet) Some of the assumption that we need to split packages was related to lack of tree shaking support. We might want to consider this for the future, but for now we should stay in separate packages because: - It's way harder to split than to combine (if we find out in 6 months we needed split packages) - Everything in one package makes it easier to make spaghetti dependencies within the package - Better for development to keep things separate ## Open questions ... ## Possibly relevant facts ### Semver that matches prerelease or release (this comes up related to deduplication issues with previous-major fixes or some other cases) `^5.79.1-0` will match either `5.79.1` or `5.79.1-hotfixN` but will always pick up the latest version https://semver.npmjs.com/ # Previous discussions ## General assumptions The following snippets from previous discussions tend to have the following assumptions: - Not all teams will be able to upgrade right away. Especially early in a major version, there will still be a lot of fixes needed (sometimes contributed by product teams) in the previous version. - Due to cross-internal-repo dependencies (and the owning teams' varying upgrade cadence/willingness), it's highly likely that there will be a period of time where a particular app must load two major versions of OUFR/`@fluentui/react` because not all of its deps have upgraded. - We've previously been hesitant to assume that our contributor base has strong Git competence. This has historically been an issue with many secondary contributors from product teams. - ...others? ## (Oct) Actual version [What to do about package versioning and naming part 342357320948](/JmrHaZPHRjKTS2w3RqKtTw) where we finalized the actual plan we used for v8 ## (Dec 2) Coordinating library upgrades [Coordinating library upgrades](/FYnz7wfjT6a3DduU6yVEtw) This outlines: - actual issues faced by some of our major consumers with interdependencies between packages from different internal repos - ideas for what the upgrade order might have to be - a proposal (now discarded) of a webpack plugin to do module-level deduping of identical files from different versions - this turned out not to work because (in addition to implementation challenges) we realized *all deps of a module must be identical* for it to be considered a true duplicate, and in practice this isn't common enough for the plugin to be significantly beneficial ## (Sept 24) What would previous-version fixes look like without major lockstep **Note that the actual strategy used for version 8 changed after this was written.** However the outline of previous-major-version fix scenarios is still helpful. [Post beta release, how would v7 fix looks like](/U-FXeVBNQgy_vVprF5b1dw) Most relevant parts, with a few updates: #### Facts - different branches cannot be publishing same package with same major version at the same time. - 2 types of fixes our users need in previous major version - Patch, minor fixes (commonly needed, especially while latest major version is new) - Hotfix (e.g. 7.140.0-hotfix.1; rarely needed and *we should not optimize for this*) #### Proposed solution (if we *had NOT* major bumped all our packages for v8) Before our beta release (the moment we start publising v7 from `7.0` and v8 from `master`), we need to remove all sub-packages which weren't major bumped from `7.0` and use real NPM dependencies in OURF package going forward. This means any fix to v7 which touches any sub-package will need to take following steps starting from beta release: 1. make and merge the fix in sub-package in `master`. 2. wait for sub-package to be released from `master`. 3. bump sub-package dependency version in `7.0`. 4. publish dependency change in `7.0`. ## (Aug 21) v8 versioning/branching proposal WITHOUT major-lockstep (discarded) [v8 release logistics](/PR2qTS_1S3CYU8ungN3Dsw) <details> ### What happens to child packages? - Child packages are **no longer major-lockstep** (they use semver) - This helps resolve the duplicate dependencies issue for partners - If a sub-package is used by (especially re-exported from) the suite, it may **not** major bump except when the suite does - This gets rid of the worst hotfix cases - Breaking changes/experiments within major will be done in /next (then in major release, it will move to default and the old version can be /compat if needed) - Each major release, we'll look at which /next things (or other desired breaking changes) are ready to be promoted - ~~In hotfix branches (`7.0` etc), find a way to ban non-hotfix publishing for child packages (beachball disallowedChangeTypes)~~ *(would be solved by pulling from npm)* </details> ## (Aug 12) v8 versioning/branching WITH major-lockstep (old version) go back in version history on [this note](/PR2qTS_1S3CYU8ungN3Dsw) to "Before major lockstep plan changes" <details> ### What happens to child packages? ("child packages" = published packages that are used and/or exported by the suite) Proposal: - At least for version 8, child packages **continue in major-lockstep**. - The main advantage here is **simplicity**: - We continue to have a simple hotfix story for now (without major-lockstep it becomes *very* complicated and maybe not even feasible; and even if it's feasible there would be a *lot* of extra work involved) - This makes it easy to go ahead with @uifabric to @fluentui scope renames and get it over with (details ~~below~~ *omitted, not relevant to future*) - Child packages which are currently at 0.x will *not* be required to go to 8.x unless they're ready (scope renames make this easier) - The main disadvantage is larger bundle size for people who must use both 7.x and 8.x for awhile. - This is not great, but it's no worse than previous releases > David's notes: > > Recommend we adhere to semver and avoid the major lockstep. This creates yet another major, unnecessary bundle problem for partners. Going from 6 to 7, partners which loaded v6 code AND v7 code in the same scope had duplication of everything. > > We should not optimize for hotfixes. If we need a hotfix of a particular package, they can be rolled individually within the things which need them. </details> ## Previous-major fix story ideas for child packages without major-lockstep If we get rid of major-lockstep, fixes to the previous major version of the suite become complicated if the fix is in a child package. *note that this does NOT reflect the actual versioning strategy we used for version 7 and 8--I updated the scenario to use hypothetical versions to try and make this obvious* <details> Y = current major suite version X = previous major suite version Z = separate "utilities" sub-package version **Scenario:** - Someone needs to fix a bug in `@fluentui/react` major version X, but the actual issue is in `@fluentui/utilities` - `@fluentui/utilities` had no breaking changes between suite major versions X and Y, so: - both master and the X branch still use utilities major version Z - the code for utilities has been removed from the X branch (because you can't publish the same major version out of two branches), so it's pulling utilities version Z from npm - The version of utilities used by the X branch is `Z.1.2`, so FUIR X depends on `utilities@^Z.1.2` - Meanwhile, subsequent versions of utilities have been released out of master: `Z.1.3`, `Z.2.0`, etc  **Possibilities:** - (1) Require fix to be checked in as usual in master (or in a child package hotfix branch if absolutely necessary). Then child branch pulls the latest utilities@Z from npm. - Downsides: - they may not want to pick up all of the changes due to regression risk - may require package manager magic to get new version and prevent duplicates - (2) Use hotfix (technically prerelease) versions - This would require a separate branch for the fix probably - Once the branch exists, it might be possible to fully or partially handle with beachball - problem: **Prerelease versions (probably?) never satisfy a semver range**, meaning we must either: - Publish new versions of all X-series packages with a specific dep on the hotfix version - Which guarantees duplicates if the consumer has multiple FUIR versions, defeating the original point of this mess - The consumer must use their package manager's pinning/resolution options to pin utilities to the new hotfix version (again awkward if the consumer has multiple FUIR versions) - Subsequent fixes and fixes spanning multiple packages get very messy **Even worse future scenario:** - Someone needs a fix in button version 8 - button is now on version 9 or even 10 in master - The suite package is still at version 8 and must continue depending on button 8 until the next major suite version (to avoid breaking changes to a re-exported API) More questions: - If hotfixes must be done in master, who handles the version bump in X branch? - This is probably the reason why most major monorepos bump major versions of all monorepo packages in lock step... - Major-lockstep: jest, babel, ... - may not be entirely comparable due to different consumption patterns/requirements - NOT major-lockstep: ... - [react](https://github.com/facebook/react) seems to have multiple groups of packages which move in major-lockstep, plus experimental packages at a mix of versions (not sure if they ever need hotfixes) - Avoids issue by not using a monorepo: webpack, ant-design, ... </details>