<!-- .slide: data-background="https://hackmd.io/_uploads/S10D2xYY5.png" -->
---
Slides: https://hackmd.io/@aspect/rules_js
Note:
This talk has a bunch of links which you can come find in the slides.
---
Bazel: most scalable polyglot Build System.
![top-languages-over-the-years](https://hackmd.io/_uploads/H1y-RdzK9.png)
JavaScript & TypeScript: most popular languages.
Note:
- Bazel works with all languages, let's make it good with the most popular ones.
- image from https://octoverse.github.com/#top-languages-over-the-years
---
:one: Introductions
:two: Fetch and install npm packages
:three: Runtime module resolutions
:four: How to use rules_js
Note:
- I'll introduce Bazel and Node.js and explain where they clash, to motivate why we designed rules_js like we did.
---
# :one: Introductions
- Alex Eagle
- Aspect Development
- Bazel
- NodeJS
- pnpm
- GH/bazelbuild/**rules_nodejs**
----
## Who is Alex Eagle
<img src="https://avatars0.githubusercontent.com/u/47395?s=460&v=4" style="border:2px solid grey; border-radius: 200px;" width=100/>
- Worked at Google on DevInfra 2008-2020
- Bazel most of that time: TL for Google's CI, build/test results UI, Angular CLI
- twitter.com/jakeherringbone
----
## What is Aspect
I Co-founded Aspect Development to make Bazel the industry-standard full-stack build system
- <https://aspect.dev> - Support and consulting to help you adopt Bazel
- <https://aspect.build> - Products making Bazel easier to use
- <https://github.com/aspect-build> - rules_js is part of our Bazel rules ecosystem
----
## What is Bazel
- Build system for "every" langugage
- Incremental: re-build proportional to what you changed
- Cached/parallel: distribute over server farm
- Scalable: works for Google's 2 billion line repo
- Unix Philosophy: just spawns subprocesses, which can be any tool
More: https://www.aspect.dev/resources
Note:
- Bazel is an "orchestrator"
- This talk isn't the "bazel sales pitch"
----
### When to consider Bazel for frontend
- **Large-scale**: 1M SLOC / 100 devs
- **Monorepo**: same use cases as Nx/Rush/Lerna
- **Polyglot/full-stack**: parachute anywhere
- **Integration testing**: fast test against backend
- **Have a DevInfra team**: economy of scale
More: https://www.aspect.dev/resources
Note:
- https://www.aspect.dev/blog/cboi-continuous-build-occasional-integration
----
### None of those apply?
Small, disconnected JS apps shouldn't use Bazel.
The build system recommended by your framework is well supported for small-to-medium scale.
----
## What is NodeJS
JavaScript engine that runs outside the browser.
Typically used for running dev tools to
build and test JavaScript programs.
Note:
- just a runtime, requires a package manager
----
## What is `pnpm`
- "Fast, disk space efficient package manager": <https://pnpm.io/>
- Works with nearly the whole ecosystem
- Used by the <https://rushjs.io/> monorepo JS-only build tool
- Happens to fit perfectly with Bazel semantics!
Note:
- Symlinks to a "virtual store" of fetched packages rather than nested layout
- Their lockfile happens to present 100% of the info needed to fetch AND link the node_modules tree
----
## What is `rules_nodejs`
Bazel rules forked from Google-internal
- toolchain to run hermetic NodeJS interpreter
- shared Bazel interfaces ("Providers") like TypeScript `DeclarationInfo`
rules_js is a layer on `rules_nodejs`
`build_bazel_rules_nodejs` is replaced
---
## Build systems:
### Matrix / Hub-and-Spoke
:construction: The JS ecosystem took a wrong turn
- Grunt and Gulp fell out of favor
- Instead, each tool became a Build System
- Now each tool needs a plugin for each language
----
![](https://hackmd.io/_uploads/BJWSKWKK5.png)
ModernWeb Meetup: Layering in JS tooling
https://www.aspect.dev/resources :point_right: last one
Note:
- I drew this live in my talk :laughing:
- When each column is a tool like webpack or jest
- each row is a language like typescript or sass
- MxN support matrix - so many plugin authors to trust
- A new framework like Qwik has to support every CSS preprocessor??
----
![](https://hackmd.io/_uploads/SJd_F-ttc.png)
Like Gulp or Grunt, but way (way) better.
Let's use Bazel!
---
## How Bazel works
In five minutes :thinking_face:
----
### Bazel: Loading Phase
Load and evaluate all extensions, `BUILD` files and macros that are needed for the build.
```graphviz
digraph {
rankdir="LR"
"sources" -> "dependency graph"
"external repos" -> "dependency graph"
}
```
Note:
- bazel has to download stuff!
----
`bazel fetch [targets]`
[![asciicast](https://asciinema.org/a/052YRxG2Pix7QsMk9dst1mTw0.svg)](https://asciinema.org/a/052YRxG2Pix7QsMk9dst1mTw0)
----
https://blog.aspect.dev/configuring-bazels-downloader
- Bazel is full-featured for fetching external deps
- Can air-gap, security scan, artifactory, etc
- Supply-chain secure, Trust-on-first-use model
- Cache based on integrity hashes
----
### Bazel: Dependency Graph
```graphviz
digraph {
compound=true
rankdir="LR"
graph [ fontname="Source Sans Pro", fontsize=20 ];
node [ fontname="Source Sans Pro", fontsize=18];
edge [ fontname="Source Sans Pro", fontsize=12 ];
subgraph cluster1 {
label="monorepo"
my_app -> my_lib
my_app -> my_comp
}
subgraph cluster2 {
label="otherrepo"
other_team_lib
}
subgraph cluster3 {
label="registry.npmjs.com"
request
react
express
}
my_lib -> request
my_comp -> react
my_app -> express
my_app -> other_team_lib
}
```
Note:
- It's pretty much what you'd imagine as the dependency graph
- Bazel can show you a graph like this for your workspace, try `bazel query`
----
`bazel query --output=graph [targets]`
```graphviz
digraph mygraph {
rankdir="LR"
node [shape=box];
"//examples/macro:node_modules/mocha/dir"
"//examples/macro:node_modules/mocha/dir" -> "//examples/macro:node_modules/mocha"
"//examples/macro:node_modules"
"//examples/macro:node_modules" -> "//examples/macro:node_modules/mocha"
"//examples/macro:node_modules" -> "//examples/macro:node_modules/mocha-junit-reporter"
"//examples/macro:node_modules" -> "//examples/macro:node_modules/mocha-multi-reporters"
"//examples/macro:node_modules/mocha"
"//examples/macro:node_modules/mocha-junit-reporter/dir"
"//examples/macro:node_modules/mocha-junit-reporter/dir" -> "//examples/macro:node_modules/mocha-junit-reporter"
"//examples/macro:test"
"//examples/macro:test" -> "//examples/macro:_test_srcs\n//examples/macro:test__entry_point"
"//examples/macro:test" -> "//examples/macro:node_modules/mocha-junit-reporter"
"//examples/macro:test" -> "//examples/macro:node_modules/mocha-multi-reporters"
"//examples/macro:node_modules/mocha-junit-reporter"
"//examples/macro:node_modules/mocha-multi-reporters/dir"
"//examples/macro:node_modules/mocha-multi-reporters/dir" -> "//examples/macro:node_modules/mocha-multi-reporters"
"//examples/macro:node_modules/mocha-multi-reporters"
"//examples/macro:_test_srcs\n//examples/macro:test__entry_point"
}
```
----
### Bazel: Analysis Phase
```graphviz
digraph {
rankdir="LR"
"dependency graph" -> "action graph" [label = "Bazel rules"]
}
```
Action: for a requested output, how to generate it from some inputs and tools
e.g. “if you need `hello.js`, run `swc` on `hello.ts`”.
Requires predicting the outputs!
----
`bazel aquery [targets]`
```graphviz
digraph {
rankdir="LR"
"index.ts" -> "index.js" [label=swc]
"component.ts" -> "component.js" [label=tsc]
"index.js" -> "bundle.js" [label=esbuild]
"component.js" -> "bundle.js"
"styles.scss" -> "styles.css" [label=sass]
"styles.css" -> "serve" [label=express]
"bundle.js" -> "serve"
}
```
----
### Bazel: Execution Phase
Execute a subset of the action graph by spawning subprocesses (e.g. `node`)
```graphviz
digraph {
rankdir="LR"
"changed index.ts" [shape=note]
"changed index.ts" -> "index.js" [label=swc]
"component.ts" [shape=none]
"component.js" [shape=none]
"component.ts" -> "component.js"
"index.js" -> "bundle.js" [label=esbuild]
"component.js" -> "bundle.js"
"styles.scss" [shape=none]
"styles.scss" -> "styles.css"
"styles.css" [shape=none]
"styles.css" -> "serve" [label=express]
"bundle.js" -> "serve"
}
```
----
User requested certain targets be built.
Bazel is lazy and will only:
- fetch precise dependencies needed
- run actions required by the transitive dependency closure of those targets
- run actions that are a "cache miss"
---
# :two: Fetch and install npm packages
----
### How npm/yarn solve it
`npm install`
Install everything needed for the whole
package/workspace
Any build/test script can depend on all npm packages :face_with_rolling_eyes:
----
### How Google solves it
Vendor the world: copy npm ecosystem sources into VCS
- Never fetches from the internet
- Never runs any package installation
You *could* do it this way too. :face_with_one_eyebrow_raised:
----
### How rules_nodejs solved it
Just wrap `[npm|yarn] install` - install the world
:thumbsdown: Guaranteed slow when repo rule invalidates
:thumbsdown: Extra bad when "eager fetching" npm deps
Note:
- https://blog.aspect.dev/avoid-eager-fetches
----
### rules_js: ideal solution
:point_right: Port pnpm to Starlark :point_left:
- re-use pnpm's resolver (via lockfile)
- fetch with Bazel's downloader
- unpack tarballs with Bazel
- re-use `@pnpm/lifecycle` to run hooks
- these are actions - can be remote cached
- link `node_modules`
Note:
- More later about linking the node_modules
----
https://blog.aspect.dev/rulesjs-npm-benchmarks
Best case:
- BUILD file declares fine-grained deps
- build only depends on one library
- we only fetch/install one library!
----
![](https://hackmd.io/_uploads/HkEK7nCF5.png)
Note:
- This is possible because BUILD files have finer-grained graph than package.json does
----
## Workspaces
Mix of third-party and first-party deps in a tree of package.json files.
Google: single version policy
rules_nodejs: independent top-level dep installs
rules_js: supports pnpm workspaces!
Note:
- yarn, npm, pnpm all have this concept
- one version policy is too hard to align all apps on the same versions
- independent installs is hard to get transitive deps
---
# :three: Resolving npm dependencies at runtime
----
### How it works in npm
NodeJS programs rely on a `node_modules` folder
"Was a big mistake" says NodeJS creator, and
Deno fixes it (but here we are :face_with_rolling_eyes:)
The location of `node_modules` is expected to be relative to the location of the importing script.
----
### How Google solves it: patch `require`
Same strategy as "PnP", e.g. Yarn PnP.
:thumbsdown: not compatible. Many npm packages wrote their own `require` implementation.
----
### How rules_nodejs solves it: runtime "linker"
Similar to `npm link`: use symlinks to make monorepo libraries appear in the node_modules tree
:thumbsdown: Slow beginning of every NodeJS spawn
:thumbsdown: Links appear in source tree w/o sandbox
:thumbsdown: Bins don't work with `genrule`/`ctx.actions.run`
:thumbsdown: Not compatible with "persistent workers"
Note:
- rules_nodejs also has the Google "patch require" strategy but is now discouraged.
- doesn't make deps under the same tree with sources - 'rootDirs' troubles
----
### How rules_js solves it
:point_right: Linker is now just a standard Bazel target :point_left:
Node.js tools assume the working dir is a single tree of src/gen/node_modules: we can do that!
- "link" to `bazel-bin/node_modules/...`
- copy sources to `bazel-bin`
- actions first `cd bazel-out/[arch]/bin`
Note:
This requires `copy_to_bin` and a bit of care in custom rules.
---
# :four: How to use rules_js
Documentation and migration guide:
https://docs.aspect.build/rules_js
----
### Install
Copy the `WORKSPACE` snippet from latest release.
https://github.com/aspect-build/rules_js/releases
----
### Adopt pnpm
Just run `pnpm install` and check that your workflows work.
> A few npm packages still have "hoisting bugs" where they don't declare correct dependencies and accidentally rely on npm or yarn-specific layout.
Note:
- there are good workarounds for broken packages
----
### import `pnpm-lock.yaml`
`npm_translate_lock` converts to Bazel's format (Starlark).
`WORKSPACE`
```python=
load("@aspect_rules_js//npm:npm_import.bzl",
"npm_translate_lock")
npm_translate_lock(
name = "npm",
pnpm_lock = "//:pnpm-lock.yaml",
)
# Load the starlark version of the lockfile
load("@npm//:repositories.bzl", "npm_repositories")
npm_repositories()
```
Note:
- Check in the result, or compute on-the-fly.
- Can also import individual npm packages with no lockfile.
- you can look in `$(bazel info output_base)/external/npm` to see what was generated
----
### Link the npm packages
`BUILD` (next to `package.json`)
```python=
load("@npm//:defs.bzl", "js_link_all_packages")
js_link_all_packages()
```
Result of `bazel build :all` is now
```bash=
# the virtual store
bazel-bin/node_modules/.aspect_rules_js
# symlink into the virtual store
bazel-bin/node_modules/some_pkg
# If you used pnpm-workspace.yaml:
bazel-bin/packages/some_pkg/node_modules/some_dep
```
----
`bazel build examples/...`
[![asciicast](https://asciinema.org/a/7W5JD1LUuvo84GmXtVVHLiqRi.svg)](https://asciinema.org/a/7W5JD1LUuvo84GmXtVVHLiqRi)
----
### Link first-party packages
First declare the package...
`my-lib/BUILD`
```python=
load("@aspect_rules_js//npm:defs.bzl", "npm_package")
npm_package(
name = "lib",
srcs = [
"index.js",
"package.json",
],
)
```
Note:
- See rules_js/examples/lib
----
### Link first-party packages
... then link to `bazel-bin/node_modules` tree...
`app/BUILD`
```python=
load("@aspect_rules_js//npm:defs.bzl", "npm_link_package")
npm_link_package(
name = "node_modules/@mycorp/mylib",
src = "//examples/lib"
)
```
----
...then depend on it just like it came from npm!
`app/BUILD`
```python=
js_binary(
name = "my_app",
data = [
"//:node_modules/react-dom",
"//:node_modules/@mycorp/mylib",
],
entry_point = "index.js",
)
```
Note:
- this is a desirable property, so that as you slurp in your manyrepos to the monorepo, the use sites don't change
---
## Running npm tools
1. Just call the `bin` entries from package.json
1. Write a macro wrapping a `bin` entry
1. Write a custom rule
1. Use an existing custom rule (e.g. rules_ts vs `tsc`)
There are also more advanced ways, see rules_js/examples
Note:
These options are easy, hard, harder, easy
----
### `bin` entries are provided for all packages
`BUILD`
```python=
load("@npm//typescript:package_json.bzl", typescript_bin = "bin")
typescript_bin.tsc(
name = "compile",
srcs = [
"fs.ts",
"tsconfig.json",
"//:node_modules/@types/node",
],
outs = ["fs.js"],
chdir = package_name(),
args = ["-p", "tsconfig.json"],
)
```
Note:
- this doesn't cause an eager fetch! Bazel doesn't download the typescript package when loading this file
----
Each bin exposes three rules:
| Use | With | To |
| -------- | -------- | -------- |
| `foo` | `bazel build` | produce outputs |
| `foo_binary` | `bazel run` | side-effects |
| `foo_test` | `bazel test` | assert exit `0` |
----
### Wrap existing build system
Use "component libraries" to get coarse granularity
```graphviz
digraph {
rankdir="LR"
vite [label=vite]
vite2 [label=vite]
"lib sources" [shape=none]
"lib sources" -> "vite"
"app sources" [shape=none]
"app sources" -> "vite2"
vite -> vite2
vite2 -> bundle
bundle [shape=none]
}
```
----
Pretty fast developer loop in
<https://github.com/aspect-build/bazel-examples/tree/main/vue>
```python=
load("@npm//vite:package_json.bzl", vite_bin = "bin")
load("@npm//vue-tsc:package_json.bzl", vue_tsc_bin = "bin")
vite_bin.vite(
name = "dist",
args = ["build"],
out_dirs = ["dist"],
srcs = [...],
)
vue_tsc_bin.vue_tsc_test(
name = "type-check",
args = ["--noEmit"],
data = [...],
)
vite_bin.vite_binary(
name = "vite",
data = [...]
)
```
Note:
- next we'll run this "vite" target
----
`ibazel run :vite`
[![asciicast](https://asciinema.org/a/501915.svg)](https://asciinema.org/a/501915)
----
### Write a Macro
Bazel macros are like preprocessor definitions.
Good way to give "syntax sugar",
compose a few rules, set defaults.
Indistinguishable from custom rules at use site
Example: `mocha` test
----
```python=
def mocha_test(name, srcs, args = [], data = [], env = {}, **kwargs):
bin.mocha_test(
name = name,
args = [
"--reporter",
"mocha-multi-reporters",
"--reporter-options",
"configFile=$(location //examples/macro:mocha_reporters.json)",
native.package_name() + "/*test.js",
] + args,
data = data + srcs + [
"//examples/macro:mocha_reporters.json",
"//examples/macro:node_modules/mocha-multi-reporters",
"//examples/macro:node_modules/mocha-junit-reporter",
],
env = dict(env, **{
# Add environment variable so that mocha writes its test xml
# to the location Bazel expects.
"MOCHA_FILE": "$$XML_OUTPUT_FILE",
}),
)
```
[https://github.com/aspect-build/rules_js/blob/main/examples/macro/mocha.bzl](examples/macro)
Note:
- Expand a macro with `bazel query --output=build`
----
### Write a custom rule
Harder and not recommended for most users.
Start from
https://bazel.build/rules/rules-tutorial
and use
https://github.com/bazel-contrib/rules-template
---
### Use an existing custom rule
From https://github.com/aspect-build:
- rules_esbuild - Bazel rules for https://esbuild.github.io/ JS bundler
- rules_terser - Bazel rules for https://terser.org/ - a JavaScript minifier
- rules_swc - Bazel rules for the swc toolchain https://swc.rs/
- rules_ts - Bazel rules for the tsc compiler from http://typescriptlang.org
----
- rules_webpack - Bazel rules for webpack bundler https://webpack.js.org/
- rules_rollup - Bazel rules for https://rollupjs.org/ - a JavaScript bundler
- rules_jest - Bazel rules to run tests using https://jestjs.io
- rules_deno - Bazel rules for Deno http://deno.land
----
... and many more by other vendors
<http://docs.aspect.build>
Catalog coming soon at https://bazel-contrib.github.io/SIG-rules-authors/
----
### Example custom rule: `ts_project`
No more `rootDirs` in `tsconfig.json` :grin:
---
`BUILD`
```python=
load("@bazel_skylib//rules:write_file.bzl", "write_file")
# Create a test fixture that is a non-trivial sized TypeScript program
write_file(
name = "gen_ts",
out = "big.ts",
content = [
"export const a{0}: number = {0}".format(x)
for x in range(100000)
],
)
```
---
`BUILD`
```python=
load("@aspect_rules_ts//ts:defs.bzl", "ts_project")
ts_project(
name = "tsc",
srcs = ["big.ts"],
declaration = True,
source_map = True,
)
```
---
[ts_project with custom transpiler](https://github.com/aspect-build/bazel-examples/tree/main/ts_project_transpiler)
`BUILD`
```python=
load("@aspect_rules_swc//swc:defs.bzl", "swc_transpiler")
ts_project(
name = "swc",
srcs = ["big.ts"],
out_dir = "build-swc",
transpiler = partial.make(
swc_transpiler,
args = ["--env-name=test"],
swcrc = ".swcrc",
),
)
```
---
Benchmarks: ts_project w/ SWC
https://blog.aspect.dev/rules-ts-benchmarks
---
Transpile-only use case on large project
`bazel build :devserver`
![](https://hackmd.io/_uploads/S1Y6E2AK5.png)
Note:
time for a first, non-incremental build
---
### Putting it all together
Sophisticated teams can assemble their own toolchain.
Create an entire JS build system just by composing existing tools in a macro!
![](https://hackmd.io/_uploads/SJd_F-ttc.png)
Note:
- "This is our moment": Build system authors can now build on top of Bazel and go back to composing tools
---
Example: an entire custom build system called
"differential loading":
```python=
def differential_loading(name, entry_point, srcs):
"Common workflow to serve TypeScript to modern browsers"
ts_project(
name = name + "_lib",
srcs = srcs,
)
rollup_bundle(
name = name + "_chunks",
deps = [name + "_lib"],
sourcemap = "inline",
config_file = "//:rollup.config.js",
entry_points = {
entry_point: "index",
},
output_dir = True,
)
# For older browsers, we'll transform the output chunks to es5 + systemjs loader
bin.babel(
name = name + "_chunks_es5",
srcs = [
name + "_chunks",
"es5.babelrc",
":node_modules/@babel/preset-env",
],
output_dir = True,
args = [
"../../../$(execpath %s_chunks)" % name,
"--config-file",
"../../../$(execpath es5.babelrc)",
"--out-dir",
"$(@D)",
],
)
terser_minified(
name = name + "_chunks_es5.min",
src = name + "_chunks_es5",
)
```
---
# Roadmap
rules_js 1.0.0 is available now
Coming soon :tm:
- Gazelle extension to generate `BUILD` files from srcs
- Bazel 6.0 package manager: bzlmod instead of `WORKSPACE`
https://blog.aspect.dev/bzlmod
----
# Thank you!
These slides: https://hackmd.io/@aspect/rules_js
Thanks conference organizers and everyone who helped launch rules_js.
Come work with us on OSS! http://aspect.dev/careers
Paid support and consulting: http://aspect.dev
Our projects: github.com/aspect-build
{"metaMigratedAt":"2023-06-17T01:52:28.552Z","metaMigratedFrom":"YAML","title":"rules_js","breaks":true,"description":"An excellent way to build JavaScript apps with Bazel","slideOptions":"{\"theme\":\"white\"}","contributors":"[{\"id\":\"20586f0c-3c64-4285-895e-d8d3820edbd0\",\"add\":36491,\"del\":17018}]"}