# Package Distributions

## Team Meeting

### Ideas
- registry-side dist/target?
- Blocked by registry/product investment
- [ ] **Action**: "conditonal everything except deps"
- [ ] **Action**: explore "pre-fail optional deps"
- ie. distributions implemented **not** as an override but rather link post/during reify
- [ ] **Action**: @isaac will engage on Yarn's RFC

---

## Goals

- [ ] Allow maintainers to define/distribute different packages under a singular name based on conditional logic for the consuming environment's platform/engines
- [ ] Limit the amount of net-new concepts & scope (ie. try to use existing paradigms & building blocks as much as possible)
- [ ] Make this feature opt-in (at least initially)

### Current Work-Around

The current work-around is to define all of your package distributions/variants as `optionalDeps` & rely on the environment failing to install them - then testing for the dep that got properly reified.

{
  "name": "foo",
  "optionalDependencies": {
    "win-foo": "...",
    "darwin-foo": "..."
  }
}

### Initial Proposal

- Introduce a new field called `distributions` which will be utilized by `npm` to determine how/what `overrides` to create in a consuming project's `package.json`.

### Implementation

- `distributions` be an `Array` of objects defining conditions in which to **override** the root/default package in the consuming project's `package.json`
- Conditions should cascade, so maintaining order is important here (ie. this must be an `Array`)
- `engines` & `platform` fields should respect the existing shape/understanding `npm` is already aware of
- `override` will define a package spec to be used in the `overrides` definitions of the consuming project when the conditions for `engines` & `platform` are met

{ 
    "name": "foo-native",
    "distributions": [
        {
            "engines": {
                "node": "^10",
            },
            "platform": "win32",
            "override": "foo-native-win32-10"
        }, {
            "override": "foo-native-default"
        }
    ],
    "engines": {
        "npm": "^7.12.0"
    }
}

#### New flags/config

- `--with-distributions` Defaults to `false` - when `true`, will respect `distributions` defined in a package's `package.json`
- `--save-overrides` Defaults to `false` - when `true`, will write `overrides` into `package.json`
- will also be useful for the [**Audit Overrides** RFC that has already been accepted](https://github.com/npm/rfcs/blob/latest/accepted/0037-audit-overrides.md)

#### Example `install`

npm install foo-native --with-distributions --save-overrides

{
    "name": "myapp",
    "dependencies": {
        "foo": "^1"
    },
    "overrides": {
        "foo@^1": {
            "foo-native@^1": "foo-native-darwin-10@^1"
        }
    }
}

### References & Links
- Yarn's **Package Variants** Proposal: https://github.com/yarnpkg/berry/issues/2751

### Parking Lot

#### Best practice for maintainers to ensure they're consumers are using `distributions` properly

A best practice we could recommend, to avoid confusion, is to set `engines` value for `npm` & educate maintainers/consumers on `--engines-strict`

{
  ...
  "engines": {
      "npm": "^7.12.0"
  }
  ...
}

npm install --engines-strict

#### Best practice to support widest audience

Utilize the default `optionalDependencies` environment fallback w/ an optional shim while also still defining `distributions`.

{ 
    "name": "foo-native",
    "main": "lib/shim.js",
    "dependencies": {
        "foo-native-win32-10": "*",
        "foo-native-source": "*",
        "@npmcli/optional-shim": "^1",
        "debug": "^4"
    },
    "optionalDependencies": {
        "foo-native-win32-10": "*",
        "foo-native-source": "*"
    },
    "distributions": [
        {
            "engines": {
                "node": "^10",
            },
            "platform": "win32",
            "override": "foo-native-win32-10"
        }, {
            "override": "foo-native-source"
        }
    ],
    "engines": {
        "npm": "^7.12.0"
    }
}

#### Concept: use `dependencyMeta` vs. `distributions`

Zoltan (pnpm) & Mael (Yarn) have started talking more about `dependencyMeta` to store information about your dependencies; This seems to only serve to further nest information that can live in the top level of a `package.json` since this is "metadata" is directly associated with that package. `dependencyMeta` is better served at the consumer level where we already have `overrides` that can solve this problem.

`depdencyMeta` may be useful in the future, but not necessarily for this feature.

## Alternative: "choose and link"

This avoids the "different platforms result in different trees" problem.

{
  "name": "foo",
  "version": "1.2.3",
  "main": "index.js",
  "scripts": {
    "preinstall": "node-gyp rebuild"
  },
  "distributions": [
    {
      "engines": {
        "node": "10"
      },
      "platform": "win32",
      "package": "foo-native-win32-10@1.x"
    },
    {
      "platform": "linux",
      "arch": "x64",
      "package": "foo-native-linux-x64@2.x"
    },
    "..."
  ]
}

### buildIdealTree

* Place _all_ distribution targets as peers of the `foo` package
* Any that cannot be placed are not placed (eg, package not found, cannot be placed for conflicting peer deps, etc.), similar to `optionalDependencies`
* Record status of `idealTree` placed distributions of the main "foo" package (?somehow?), along with their selection criteria.

Thus the idealTree is not platform-specific.

### reify

* Choose one distribution of the main package, and reify that.
* Do not reify the others, or any of their dependencies (unless they are required to meet another dependency in the tree.)
* Reify the main package as a `Link` to the chosen distribution.

### publisher

* Publish the main package with the source, pointing to distribution package specifiers and selection criteria.
* Publish pre-built distributions as needed.

This is amenable to publishing distributions post-hoc as a CI build. If any fail to build and publish, no matter, they will simply fail to install, and fall back to the main package.

Constraints: distributions cannot conflict with one another or other dependencies within the tree. (So, not possible to have one distribution with a peer on `react@15` and another on `react@16`, for example.)

### consumer

* Add `foo` as a dependency.
* Pre-built distributions will be added to lockfile.
* `require('foo')` will return the appropriate context-specific distribution if one was found successfully, or the original package if it was not.

The package-lock.json file will include _all_ the distributions (as it would for `optionalDependencies` today), but only one of these will be reified on disk. No chance of an exponential explosion in lockfile size, as we calculate every possible combination of distribution matrices.

### Polyfill Use Case

While this is most useful for slow and costly binary builds, it is also interesting for providing polyfills for node features!

{
  "name": "fs-readdir",
  "version": "fs.readdir() guaranteed to have withFileTypes:true support",
  "distributions": [
    {
      "engines": { "node": "<v10.11.0"},
      "package": "fs-readdir-polyfill@1"
    },
    {
      "package": "fs-readdir-native@1"
    }
  ]
}

{
  "name": "fs-readdir-polyfill",
  "version": "1.2.3",
  "description": "polyfill the fs.readdir withFileTypes using fs.stat"
}

{
  "name": "fs-readdir-native",
  "version": "1.2.3",
  "description": "just export require('fs').readdir"
}

Of course, we could just as easily have the `fs-readdir-polyfill` define `fs-readdir-native` as a distribution when `"engines": {"node": ">=10.11.0" }`.