<!-- .slide: data-background="https://hackmd.io/_uploads/S10D2xYY5.png" --> --- Slides: https://hackmd.io/@aspect/rules_js Note: This talk has a bunch of links which you can come find in the slides. --- Bazel: most scalable polyglot Build System.  JavaScript & TypeScript: most popular languages. Note: - Bazel works with all languages, let's make it good with the most popular ones. - image from https://octoverse.github.com/#top-languages-over-the-years --- :one: Introductions :two: Fetch and install npm packages :three: Runtime module resolutions :four: How to use rules_js Note: - I'll introduce Bazel and Node.js and explain where they clash, to motivate why we designed rules_js like we did. --- # :one: Introductions - Alex Eagle - Aspect Development - Bazel - NodeJS - pnpm - GH/bazelbuild/**rules_nodejs** ---- ## Who is Alex Eagle <img src="https://avatars0.githubusercontent.com/u/47395?s=460&v=4" style="border:2px solid grey; border-radius: 200px;" width=100/> - Worked at Google on DevInfra 2008-2020 - Bazel most of that time: TL for Google's CI, build/test results UI, Angular CLI - twitter.com/jakeherringbone ---- ## What is Aspect I Co-founded Aspect Development to make Bazel the industry-standard full-stack build system - <https://aspect.dev> - Support and consulting to help you adopt Bazel - <https://aspect.build> - Products making Bazel easier to use - <https://github.com/aspect-build> - rules_js is part of our Bazel rules ecosystem ---- ## What is Bazel - Build system for "every" langugage - Incremental: re-build proportional to what you changed - Cached/parallel: distribute over server farm - Scalable: works for Google's 2 billion line repo - Unix Philosophy: just spawns subprocesses, which can be any tool More: https://www.aspect.dev/resources Note: - Bazel is an "orchestrator" - This talk isn't the "bazel sales pitch" ---- ### When to consider Bazel for frontend - **Large-scale**: 1M SLOC / 100 devs - **Monorepo**: same use cases as Nx/Rush/Lerna - **Polyglot/full-stack**: parachute anywhere - **Integration testing**: fast test against backend - **Have a DevInfra team**: economy of scale More: https://www.aspect.dev/resources Note: - https://www.aspect.dev/blog/cboi-continuous-build-occasional-integration ---- ### None of those apply? Small, disconnected JS apps shouldn't use Bazel. The build system recommended by your framework is well supported for small-to-medium scale. ---- ## What is NodeJS JavaScript engine that runs outside the browser. Typically used for running dev tools to build and test JavaScript programs. Note: - just a runtime, requires a package manager ---- ## What is `pnpm` - "Fast, disk space efficient package manager": <https://pnpm.io/> - Works with nearly the whole ecosystem - Used by the <https://rushjs.io/> monorepo JS-only build tool - Happens to fit perfectly with Bazel semantics! Note: - Symlinks to a "virtual store" of fetched packages rather than nested layout - Their lockfile happens to present 100% of the info needed to fetch AND link the node_modules tree ---- ## What is `rules_nodejs` Bazel rules forked from Google-internal - toolchain to run hermetic NodeJS interpreter - shared Bazel interfaces ("Providers") like TypeScript `DeclarationInfo` rules_js is a layer on `rules_nodejs` `build_bazel_rules_nodejs` is replaced --- ## Build systems: ### Matrix / Hub-and-Spoke :construction: The JS ecosystem took a wrong turn - Grunt and Gulp fell out of favor - Instead, each tool became a Build System - Now each tool needs a plugin for each language ----  ModernWeb Meetup: Layering in JS tooling https://www.aspect.dev/resources :point_right: last one Note: - I drew this live in my talk :laughing: - When each column is a tool like webpack or jest - each row is a language like typescript or sass - MxN support matrix - so many plugin authors to trust - A new framework like Qwik has to support every CSS preprocessor?? ----  Like Gulp or Grunt, but way (way) better. Let's use Bazel! --- ## How Bazel works In five minutes :thinking_face: ---- ### Bazel: Loading Phase Load and evaluate all extensions, `BUILD` files and macros that are needed for the build. ```graphviz digraph { rankdir="LR" "sources" -> "dependency graph" "external repos" -> "dependency graph" } ``` Note: - bazel has to download stuff! ---- `bazel fetch [targets]` [](https://asciinema.org/a/052YRxG2Pix7QsMk9dst1mTw0) ---- https://blog.aspect.dev/configuring-bazels-downloader - Bazel is full-featured for fetching external deps - Can air-gap, security scan, artifactory, etc - Supply-chain secure, Trust-on-first-use model - Cache based on integrity hashes ---- ### Bazel: Dependency Graph ```graphviz digraph { compound=true rankdir="LR" graph [ fontname="Source Sans Pro", fontsize=20 ]; node [ fontname="Source Sans Pro", fontsize=18]; edge [ fontname="Source Sans Pro", fontsize=12 ]; subgraph cluster1 { label="monorepo" my_app -> my_lib my_app -> my_comp } subgraph cluster2 { label="otherrepo" other_team_lib } subgraph cluster3 { label="registry.npmjs.com" request react express } my_lib -> request my_comp -> react my_app -> express my_app -> other_team_lib } ``` Note: - It's pretty much what you'd imagine as the dependency graph - Bazel can show you a graph like this for your workspace, try `bazel query` ---- `bazel query --output=graph [targets]` ```graphviz digraph mygraph { rankdir="LR" node [shape=box]; "//examples/macro:node_modules/mocha/dir" "//examples/macro:node_modules/mocha/dir" -> "//examples/macro:node_modules/mocha" "//examples/macro:node_modules" "//examples/macro:node_modules" -> "//examples/macro:node_modules/mocha" "//examples/macro:node_modules" -> "//examples/macro:node_modules/mocha-junit-reporter" "//examples/macro:node_modules" -> "//examples/macro:node_modules/mocha-multi-reporters" "//examples/macro:node_modules/mocha" "//examples/macro:node_modules/mocha-junit-reporter/dir" "//examples/macro:node_modules/mocha-junit-reporter/dir" -> "//examples/macro:node_modules/mocha-junit-reporter" "//examples/macro:test" "//examples/macro:test" -> "//examples/macro:_test_srcs\n//examples/macro:test__entry_point" "//examples/macro:test" -> "//examples/macro:node_modules/mocha-junit-reporter" "//examples/macro:test" -> "//examples/macro:node_modules/mocha-multi-reporters" "//examples/macro:node_modules/mocha-junit-reporter" "//examples/macro:node_modules/mocha-multi-reporters/dir" "//examples/macro:node_modules/mocha-multi-reporters/dir" -> "//examples/macro:node_modules/mocha-multi-reporters" "//examples/macro:node_modules/mocha-multi-reporters" "//examples/macro:_test_srcs\n//examples/macro:test__entry_point" } ``` ---- ### Bazel: Analysis Phase ```graphviz digraph { rankdir="LR" "dependency graph" -> "action graph" [label = "Bazel rules"] } ``` Action: for a requested output, how to generate it from some inputs and tools e.g. “if you need `hello.js`, run `swc` on `hello.ts`”. Requires predicting the outputs! ---- `bazel aquery [targets]` ```graphviz digraph { rankdir="LR" "index.ts" -> "index.js" [label=swc] "component.ts" -> "component.js" [label=tsc] "index.js" -> "bundle.js" [label=esbuild] "component.js" -> "bundle.js" "styles.scss" -> "styles.css" [label=sass] "styles.css" -> "serve" [label=express] "bundle.js" -> "serve" } ``` ---- ### Bazel: Execution Phase Execute a subset of the action graph by spawning subprocesses (e.g. `node`) ```graphviz digraph { rankdir="LR" "changed index.ts" [shape=note] "changed index.ts" -> "index.js" [label=swc] "component.ts" [shape=none] "component.js" [shape=none] "component.ts" -> "component.js" "index.js" -> "bundle.js" [label=esbuild] "component.js" -> "bundle.js" "styles.scss" [shape=none] "styles.scss" -> "styles.css" "styles.css" [shape=none] "styles.css" -> "serve" [label=express] "bundle.js" -> "serve" } ``` ---- User requested certain targets be built. Bazel is lazy and will only: - fetch precise dependencies needed - run actions required by the transitive dependency closure of those targets - run actions that are a "cache miss" --- # :two: Fetch and install npm packages ---- ### How npm/yarn solve it `npm install` Install everything needed for the whole package/workspace Any build/test script can depend on all npm packages :face_with_rolling_eyes: ---- ### How Google solves it Vendor the world: copy npm ecosystem sources into VCS - Never fetches from the internet - Never runs any package installation You *could* do it this way too. :face_with_one_eyebrow_raised: ---- ### How rules_nodejs solved it Just wrap `[npm|yarn] install` - install the world :thumbsdown: Guaranteed slow when repo rule invalidates :thumbsdown: Extra bad when "eager fetching" npm deps Note: - https://blog.aspect.dev/avoid-eager-fetches ---- ### rules_js: ideal solution :point_right: Port pnpm to Starlark :point_left: - re-use pnpm's resolver (via lockfile) - fetch with Bazel's downloader - unpack tarballs with Bazel - re-use `@pnpm/lifecycle` to run hooks - these are actions - can be remote cached - link `node_modules` Note: - More later about linking the node_modules ---- https://blog.aspect.dev/rulesjs-npm-benchmarks Best case: - BUILD file declares fine-grained deps - build only depends on one library - we only fetch/install one library! ----  Note: - This is possible because BUILD files have finer-grained graph than package.json does ---- ## Workspaces Mix of third-party and first-party deps in a tree of package.json files. Google: single version policy rules_nodejs: independent top-level dep installs rules_js: supports pnpm workspaces! Note: - yarn, npm, pnpm all have this concept - one version policy is too hard to align all apps on the same versions - independent installs is hard to get transitive deps --- # :three: Resolving npm dependencies at runtime ---- ### How it works in npm NodeJS programs rely on a `node_modules` folder "Was a big mistake" says NodeJS creator, and Deno fixes it (but here we are :face_with_rolling_eyes:) The location of `node_modules` is expected to be relative to the location of the importing script. ---- ### How Google solves it: patch `require` Same strategy as "PnP", e.g. Yarn PnP. :thumbsdown: not compatible. Many npm packages wrote their own `require` implementation. ---- ### How rules_nodejs solves it: runtime "linker" Similar to `npm link`: use symlinks to make monorepo libraries appear in the node_modules tree :thumbsdown: Slow beginning of every NodeJS spawn :thumbsdown: Links appear in source tree w/o sandbox :thumbsdown: Bins don't work with `genrule`/`ctx.actions.run` :thumbsdown: Not compatible with "persistent workers" Note: - rules_nodejs also has the Google "patch require" strategy but is now discouraged. - doesn't make deps under the same tree with sources - 'rootDirs' troubles ---- ### How rules_js solves it :point_right: Linker is now just a standard Bazel target :point_left: Node.js tools assume the working dir is a single tree of src/gen/node_modules: we can do that! - "link" to `bazel-bin/node_modules/...` - copy sources to `bazel-bin` - actions first `cd bazel-out/[arch]/bin` Note: This requires `copy_to_bin` and a bit of care in custom rules. --- # :four: How to use rules_js Documentation and migration guide: https://docs.aspect.build/rules_js ---- ### Install Copy the `WORKSPACE` snippet from latest release. https://github.com/aspect-build/rules_js/releases ---- ### Adopt pnpm Just run `pnpm install` and check that your workflows work. > A few npm packages still have "hoisting bugs" where they don't declare correct dependencies and accidentally rely on npm or yarn-specific layout. Note: - there are good workarounds for broken packages ---- ### import `pnpm-lock.yaml` `npm_translate_lock` converts to Bazel's format (Starlark). `WORKSPACE` ```python= load("@aspect_rules_js//npm:npm_import.bzl", "npm_translate_lock") npm_translate_lock( name = "npm", pnpm_lock = "//:pnpm-lock.yaml", ) # Load the starlark version of the lockfile load("@npm//:repositories.bzl", "npm_repositories") npm_repositories() ``` Note: - Check in the result, or compute on-the-fly. - Can also import individual npm packages with no lockfile. - you can look in `$(bazel info output_base)/external/npm` to see what was generated ---- ### Link the npm packages `BUILD` (next to `package.json`) ```python= load("@npm//:defs.bzl", "js_link_all_packages") js_link_all_packages() ``` Result of `bazel build :all` is now ```bash= # the virtual store bazel-bin/node_modules/.aspect_rules_js # symlink into the virtual store bazel-bin/node_modules/some_pkg # If you used pnpm-workspace.yaml: bazel-bin/packages/some_pkg/node_modules/some_dep ``` ---- `bazel build examples/...` [](https://asciinema.org/a/7W5JD1LUuvo84GmXtVVHLiqRi) ---- ### Link first-party packages First declare the package... `my-lib/BUILD` ```python= load("@aspect_rules_js//npm:defs.bzl", "npm_package") npm_package( name = "lib", srcs = [ "index.js", "package.json", ], ) ``` Note: - See rules_js/examples/lib ---- ### Link first-party packages ... then link to `bazel-bin/node_modules` tree... `app/BUILD` ```python= load("@aspect_rules_js//npm:defs.bzl", "npm_link_package") npm_link_package( name = "node_modules/@mycorp/mylib", src = "//examples/lib" ) ``` ---- ...then depend on it just like it came from npm! `app/BUILD` ```python= js_binary( name = "my_app", data = [ "//:node_modules/react-dom", "//:node_modules/@mycorp/mylib", ], entry_point = "index.js", ) ``` Note: - this is a desirable property, so that as you slurp in your manyrepos to the monorepo, the use sites don't change --- ## Running npm tools 1. Just call the `bin` entries from package.json 1. Write a macro wrapping a `bin` entry 1. Write a custom rule 1. Use an existing custom rule (e.g. rules_ts vs `tsc`) There are also more advanced ways, see rules_js/examples Note: These options are easy, hard, harder, easy ---- ### `bin` entries are provided for all packages `BUILD` ```python= load("@npm//typescript:package_json.bzl", typescript_bin = "bin") typescript_bin.tsc( name = "compile", srcs = [ "fs.ts", "tsconfig.json", "//:node_modules/@types/node", ], outs = ["fs.js"], chdir = package_name(), args = ["-p", "tsconfig.json"], ) ``` Note: - this doesn't cause an eager fetch! Bazel doesn't download the typescript package when loading this file ---- Each bin exposes three rules: | Use | With | To | | -------- | -------- | -------- | | `foo` | `bazel build` | produce outputs | | `foo_binary` | `bazel run` | side-effects | | `foo_test` | `bazel test` | assert exit `0` | ---- ### Wrap existing build system Use "component libraries" to get coarse granularity ```graphviz digraph { rankdir="LR" vite [label=vite] vite2 [label=vite] "lib sources" [shape=none] "lib sources" -> "vite" "app sources" [shape=none] "app sources" -> "vite2" vite -> vite2 vite2 -> bundle bundle [shape=none] } ``` ---- Pretty fast developer loop in <https://github.com/aspect-build/bazel-examples/tree/main/vue> ```python= load("@npm//vite:package_json.bzl", vite_bin = "bin") load("@npm//vue-tsc:package_json.bzl", vue_tsc_bin = "bin") vite_bin.vite( name = "dist", args = ["build"], out_dirs = ["dist"], srcs = [...], ) vue_tsc_bin.vue_tsc_test( name = "type-check", args = ["--noEmit"], data = [...], ) vite_bin.vite_binary( name = "vite", data = [...] ) ``` Note: - next we'll run this "vite" target ---- `ibazel run :vite` [](https://asciinema.org/a/501915) ---- ### Write a Macro Bazel macros are like preprocessor definitions. Good way to give "syntax sugar", compose a few rules, set defaults. Indistinguishable from custom rules at use site Example: `mocha` test ---- ```python= def mocha_test(name, srcs, args = [], data = [], env = {}, **kwargs): bin.mocha_test( name = name, args = [ "--reporter", "mocha-multi-reporters", "--reporter-options", "configFile=$(location //examples/macro:mocha_reporters.json)", native.package_name() + "/*test.js", ] + args, data = data + srcs + [ "//examples/macro:mocha_reporters.json", "//examples/macro:node_modules/mocha-multi-reporters", "//examples/macro:node_modules/mocha-junit-reporter", ], env = dict(env, **{ # Add environment variable so that mocha writes its test xml # to the location Bazel expects. "MOCHA_FILE": "$$XML_OUTPUT_FILE", }), ) ``` [https://github.com/aspect-build/rules_js/blob/main/examples/macro/mocha.bzl](examples/macro) Note: - Expand a macro with `bazel query --output=build` ---- ### Write a custom rule Harder and not recommended for most users. Start from https://bazel.build/rules/rules-tutorial and use https://github.com/bazel-contrib/rules-template --- ### Use an existing custom rule From https://github.com/aspect-build: - rules_esbuild - Bazel rules for https://esbuild.github.io/ JS bundler - rules_terser - Bazel rules for https://terser.org/ - a JavaScript minifier - rules_swc - Bazel rules for the swc toolchain https://swc.rs/ - rules_ts - Bazel rules for the tsc compiler from http://typescriptlang.org ---- - rules_webpack - Bazel rules for webpack bundler https://webpack.js.org/ - rules_rollup - Bazel rules for https://rollupjs.org/ - a JavaScript bundler - rules_jest - Bazel rules to run tests using https://jestjs.io - rules_deno - Bazel rules for Deno http://deno.land ---- ... and many more by other vendors <http://docs.aspect.build> Catalog coming soon at https://bazel-contrib.github.io/SIG-rules-authors/ ---- ### Example custom rule: `ts_project` No more `rootDirs` in `tsconfig.json` :grin: --- `BUILD` ```python= load("@bazel_skylib//rules:write_file.bzl", "write_file") # Create a test fixture that is a non-trivial sized TypeScript program write_file( name = "gen_ts", out = "big.ts", content = [ "export const a{0}: number = {0}".format(x) for x in range(100000) ], ) ``` --- `BUILD` ```python= load("@aspect_rules_ts//ts:defs.bzl", "ts_project") ts_project( name = "tsc", srcs = ["big.ts"], declaration = True, source_map = True, ) ``` --- [ts_project with custom transpiler](https://github.com/aspect-build/bazel-examples/tree/main/ts_project_transpiler) `BUILD` ```python= load("@aspect_rules_swc//swc:defs.bzl", "swc_transpiler") ts_project( name = "swc", srcs = ["big.ts"], out_dir = "build-swc", transpiler = partial.make( swc_transpiler, args = ["--env-name=test"], swcrc = ".swcrc", ), ) ``` --- Benchmarks: ts_project w/ SWC https://blog.aspect.dev/rules-ts-benchmarks --- Transpile-only use case on large project `bazel build :devserver`  Note: time for a first, non-incremental build --- ### Putting it all together Sophisticated teams can assemble their own toolchain. Create an entire JS build system just by composing existing tools in a macro!  Note: - "This is our moment": Build system authors can now build on top of Bazel and go back to composing tools --- Example: an entire custom build system called "differential loading": ```python= def differential_loading(name, entry_point, srcs): "Common workflow to serve TypeScript to modern browsers" ts_project( name = name + "_lib", srcs = srcs, ) rollup_bundle( name = name + "_chunks", deps = [name + "_lib"], sourcemap = "inline", config_file = "//:rollup.config.js", entry_points = { entry_point: "index", }, output_dir = True, ) # For older browsers, we'll transform the output chunks to es5 + systemjs loader bin.babel( name = name + "_chunks_es5", srcs = [ name + "_chunks", "es5.babelrc", ":node_modules/@babel/preset-env", ], output_dir = True, args = [ "../../../$(execpath %s_chunks)" % name, "--config-file", "../../../$(execpath es5.babelrc)", "--out-dir", "$(@D)", ], ) terser_minified( name = name + "_chunks_es5.min", src = name + "_chunks_es5", ) ``` --- # Roadmap rules_js 1.0.0 is available now Coming soon :tm: - Gazelle extension to generate `BUILD` files from srcs - Bazel 6.0 package manager: bzlmod instead of `WORKSPACE` https://blog.aspect.dev/bzlmod ---- # Thank you! These slides: https://hackmd.io/@aspect/rules_js Thanks conference organizers and everyone who helped launch rules_js. Come work with us on OSS! http://aspect.dev/careers Paid support and consulting: http://aspect.dev Our projects: github.com/aspect-build