# :herb: Bazel and JavaScript ## Bazel community update ## August 18 2022 --- Bazel: most scalable polyglot Build System.  JavaScript & TypeScript: most popular languages. Note: - Bazel works with all languages, let's make it good with the most popular ones. - image from https://octoverse.github.com/#top-languages-over-the-years --- ## Who is Alex Eagle <img src="https://avatars0.githubusercontent.com/u/47395?s=460&v=4" style="border:2px solid grey; border-radius: 200px;" width=100/> - Worked at Google on DevInfra 2008-2020 - Bazel most of that time: TL for Google's CI, build/test results UI, Angular CLI - Started Rules Authors SIG, maintained rules_{nodejs,python,docker} - twitter.com/jakeherringbone ---- ## What is Aspect I Co-founded Aspect Development to make Bazel the industry-standard full-stack build system - <https://aspect.dev> - Support and consulting to help you adopt Bazel - <https://aspect.build> - Products making Bazel easier to use - <https://github.com/aspect-build> - rules_js is part of our Bazel rules ecosystem ---- ## Build systems: ### Matrix / Hub-and-Spoke :construction: The JS ecosystem took a wrong turn - Grunt and Gulp fell out of favor - Instead, each tool became a Build System - Now each tool needs a plugin for each language ----  ModernWeb Meetup: Layering in JS tooling https://www.aspect.dev/resources :point_right: last one Note: - I drew this live in my talk :laughing: - When each column is a tool like webpack or jest - each row is a language like typescript or sass - MxN support matrix - so many plugin authors to trust - A new framework like Qwik has to support every CSS preprocessor?? ----  Like Gulp or Grunt, but way (way) better. Let's use Bazel! ---- ## What is NodeJS JavaScript engine that runs outside the browser. Typically used for running dev tools to build and test JavaScript programs. Deno is an up-and-coming alternative, see rules_deno Note: - just a runtime, requires a package manager ---- ## What is `rules_nodejs` Bazel rules forked from Google-internal - toolchain to run hermetic NodeJS interpreter - shared Bazel interfaces ("Providers") like TypeScript `DeclarationInfo` ---- ## What is `rules_js` rules_js is a layer on top of `rules_nodejs` About rules_js: My talk from Bazel eXchange 2022 Slides: https://hackmd.io/@aspect/rules_js Video: https://aspect.dev/resources (first video) ---- ## What is `pnpm` - "Fast, disk space efficient package manager": <https://pnpm.io/> - Works with nearly the whole ecosystem - Used by the <https://rushjs.io/> monorepo JS-only build tool - Happens to fit perfectly with Bazel semantics! - Used by `rules_js` Note: - Symlinks to a "virtual store" of fetched packages rather than nested layout - Their lockfile happens to present 100% of the info needed to fetch AND link the node_modules tree ---- ## JS: Client and Server Client runtime: Browser Server runtime: Node.js Both use the same virtual machine (V8), stdlib differs - `rules_nodejs` supports both, though confusingly named - `rules_js` also supports both ---- Modules ```graphviz= digraph G { rankdir="TB" fontname="courier" subgraph cluster_bazelbuild { label = "github.com/bazelbuild" subgraph cluster_0 { style=filled; color=lightgrey; node [style=filled,color=white]; nodejs_binary nodejs_star [label="nodejs_*"] yarn_install label = "build_bazel_rules_nodejs"; } subgraph cluster_1 { toolchain label = "rules_nodejs"; color=blue } } subgraph cluster_npm { label = "npm" subgraph cluster_2 { style=filled; color=lightgrey; node [style=filled,color=white]; ts [label="ts_project"] label = "@bazel/typescript"; } subgraph cluster_6 { style=filled; color=lightgrey; node [style=filled,color=white]; other [label="..."] label = "@bazel/*"; } } subgraph cluster_aspect { label = "github.com/aspect-build" subgraph cluster_3 { js_binary js_star [label="js_*"] npm_star [label="npm_*"] label = "rules_js"; color=blue } subgraph cluster_4 { ts_project label = "rules_ts"; color=blue } subgraph cluster_7 { other_rules [label="..."] label = "rules_*" color=blue } subgraph cluster_8 { copy_to_bin label = "bazel_lib" color=blue } } other_rules -> js_binary ts -> nodejs_binary nodejs_binary -> toolchain js_binary -> toolchain ts_project -> js_binary js_binary -> copy_to_bin } ``` NEWS: Shaded boxes are no longer maintained! --- # Fetch and install npm packages ---- ### How npm/yarn solve it `npm install` Install everything needed for the whole package/workspace Any build/test script can depend on all npm packages :face_with_rolling_eyes: ---- ### How Google solves it Vendor the world: copy npm ecosystem sources into VCS - Never fetches from the internet - Never runs any package installation You *could* do it this way too. :face_with_one_eyebrow_raised: ---- ### How rules_nodejs solved it Just wrap `[npm|yarn] install` - install the world :thumbsdown: Guaranteed slow when repo rule invalidates :thumbsdown: Extra bad when "eager fetching" npm deps Note: - https://blog.aspect.dev/avoid-eager-fetches ---- ### rules_js: ideal solution :point_right: Port pnpm to Starlark :point_left: - re-use pnpm's resolver (via lockfile) - fetch with Bazel's downloader - unpack tarballs with Bazel - re-use `@pnpm/lifecycle` to run hooks - these are actions - can be remote cached - link `node_modules` Note: - More later about linking the node_modules ---- https://blog.aspect.dev/rulesjs-npm-benchmarks Best case: - BUILD file declares fine-grained deps - build only depends on one library - we only fetch/install one library! ----  Note: - This is possible because BUILD files have finer-grained graph than package.json does --- # Resolving npm dependencies at runtime ---- ### How it works in npm NodeJS programs rely on a `node_modules` folder "Was a big mistake" says NodeJS creator, and Deno fixes it (but here we are :face_with_rolling_eyes:) The location of `node_modules` is expected to be relative to the location of the importing script. ---- ### How Google solves it: patch `require` Same strategy as "PnP", e.g. Yarn PnP. :thumbsdown: not compatible. Many npm packages wrote their own `require` implementation. ---- ### How rules_nodejs solves it: runtime "linker" Similar to `npm link`: use symlinks to make monorepo libraries appear in the node_modules tree :thumbsdown: Slow beginning of every NodeJS spawn :thumbsdown: Links appear in source tree w/o sandbox :thumbsdown: Bins don't work with `genrule`/`ctx.actions.run` :thumbsdown: Not compatible with "persistent workers" Note: - rules_nodejs also has the Google "patch require" strategy but is now discouraged. - doesn't make deps under the same tree with sources - 'rootDirs' troubles ---- ### How rules_js solves it :point_right: Linker is now just a standard Bazel target :point_left: Node.js tools assume the working dir is a single tree of src/gen/node_modules: we can do that! - "link" to `bazel-bin/node_modules/...` - copy sources to `bazel-bin` - actions first `cd bazel-out/[arch]/bin` Note: This requires `copy_to_bin` and a bit of care in custom rules. --- # Code Examples https://github.com/aspect-build/bazel-examples/tree/main/vue 1. `WORKSPACE`: read 3p npm package lockfile 2. Link npm packages 3. `BUILD`: declare dependencies on npm packages 4. Run tools from npm 5. Wrap tools in macros 6. Custom rules --- # Roadmap rules_js 1.0.0 is available now Coming soon :tm: - Missing RBE support due to bug with Bazel resolving symlinks - Gazelle extension to generate `BUILD` files from srcs - Bazel 6.0 package manager: bzlmod instead of `WORKSPACE` https://blog.aspect.dev/bzlmod --- ### Thanks! Q&A time Or come to our office hours calendly.com/aspect-build/rules_js-office-hours