HackMDDesign: rules_js linker
The concept of a "linker" is standard across many languages.
It allows the runtime to resolve references from a binary (NodeJS application) to libraries it depends on.
It is similar to:
npm linkpnpm link- build_bazel_rules_nodejs linker
Our task is to make nodejs programs resolve their dependencies when run under Bazel.
At a high-level, our intent is to be "bug-for-bug compatible" with pnpm, modeleled from how Rush uses it.
Tracking issue: https://github.com/aspect-build/rules_js/issues/16
Compatibility vs performance
The linker in build_bazel_rules_nodejs tries to ensure compatibility with all NodeJS programs, by writing a full node_modules tree on every run.
Like pnpm, this linker doesn't aim for 100% compatibility for "bad packages".
From https://rushjs.io/pages/maintainer/package_managers/#considerations-for-pnpm:
Although PNPM's symlinking strategy correctly follows the modern NodeJS module resolution standard, many legacy packages do not, which causes compatibility problems. Teams who migrate existing projects from Yarn/NPM to PNPM often encounter "bad packages" that need workarounds or fixes. The incompatibilities generally reflect real problems with those packages: (1) forgetting to list dependencies in the package.json file, or (2) implementing homebrew module resolution without handling symlinks according to the standard. Most "bad" packages have straightforward fixes, but it may seem daunting for a small team. (The PNPM Discord chat room is a great resource for help, though.)
Static vs dynamic
There are two ways to link: static and dynamic.
Static-linked dependencies are declared along with the program's sources, are included as part of the program and available to every execution of the program.
Under Bazel, static linking can be performed using the Runfiles tree.
Bazel can simply lay out a static dependency under path/to/program.runfiles/node_modules/some-lib.
By doing so, the program can be a Bazel "executable" and therefore appear as the tool in a genrule for example.
Dynamic-linked dependencies are provided by a user who executes the program.
They are not known at the declaration site of the program, and therefore they can't be part of the Bazel "runfiles".
Static linker details
Each npm_package rule links itself, by contributing a root_symlinks entry to its Runfiles object: https://github.com/aspect-build/rules_js/blob/main/js/private/nodejs_package.bzl#L146-L148
It also provides LinkablePackageInfo containing the name of the package.
The runfiles propagate up to the nodejs_binary rule, which merges them:
https://github.com/aspect-build/rules_js/blob/1534570d90fa8b8ecfa48989bd87786a428071a4/js/private/nodejs_binary.bzl#L192
this means that Bazel will lay out a node_modules tree for us.
In addition, we use the LinkablePackageInfo to create a $NODE_PATH environment variable so that NodeJS require function can locate the node_modules folder in the runfiles tree.
We have to set $NODE_PATH because actions run with a working directory in Bazel's execroot, which is not a subfolder of the runfiles, and therefore NodeJS doesn't locate it automatically. Parts of the npm ecosystem are moving away from supporting $NODE_PATH, so we might need to replace this with a --require script which executes before the entry point, and symlinks the runfiles node_modules directory to the execroot. This can have a downside of polluting the execroot however.
