Slides: https://hackmd.io/@aspect/rules_js

Note:
This talk has a bunch of links which you can come find in the slides.

Bazel: most scalable polyglot Build System.

top-languages-over-the-years

JavaScript & TypeScript: most popular languages.

Note:

Bazel works with all languages, let's make it good with the most popular ones.
image from https://octoverse.github.com/#top-languages-over-the-years

Introductions
Fetch and install npm packages
Runtime module resolutions
How to use rules_js

Note:

I'll introduce Bazel and Node.js and explain where they clash, to motivate why we designed rules_js like we did.

Introductions

Alex Eagle
Aspect Development
Bazel
NodeJS
pnpm
GH/bazelbuild/rules_nodejs

Who is Alex Eagle

Worked at Google on DevInfra 2008-2020
Bazel most of that time: TL for Google's CI, build/test results UI, Angular CLI
twitter.com/jakeherringbone

What is Aspect

I Co-founded Aspect Development to make Bazel the industry-standard full-stack build system

https://aspect.dev - Support and consulting to help you adopt Bazel
https://aspect.build - Products making Bazel easier to use
https://github.com/aspect-build - rules_js is part of our Bazel rules ecosystem

What is Bazel

Build system for "every" langugage
Incremental: re-build proportional to what you changed
Cached/parallel: distribute over server farm
Scalable: works for Google's 2 billion line repo
Unix Philosophy: just spawns subprocesses, which can be any tool

More: https://www.aspect.dev/resources

Note:

Bazel is an "orchestrator"
This talk isn't the "bazel sales pitch"

When to consider Bazel for frontend

Large-scale: 1M SLOC / 100 devs
Monorepo: same use cases as Nx/Rush/Lerna
Polyglot/full-stack: parachute anywhere
Integration testing: fast test against backend
Have a DevInfra team: economy of scale

More: https://www.aspect.dev/resources

Note:

https://www.aspect.dev/blog/cboi-continuous-build-occasional-integration

None of those apply?

Small, disconnected JS apps shouldn't use Bazel.

The build system recommended by your framework is well supported for small-to-medium scale.

What is NodeJS

JavaScript engine that runs outside the browser.

Typically used for running dev tools to
build and test JavaScript programs.

Note:

just a runtime, requires a package manager

What is `pnpm`

"Fast, disk space efficient package manager": https://pnpm.io/
Works with nearly the whole ecosystem
Used by the https://rushjs.io/ monorepo JS-only build tool
Happens to fit perfectly with Bazel semantics!

Note:

Symlinks to a "virtual store" of fetched packages rather than nested layout
Their lockfile happens to present 100% of the info needed to fetch AND link the node_modules tree

What is `rules_nodejs`

Bazel rules forked from Google-internal

toolchain to run hermetic NodeJS interpreter
shared Bazel interfaces ("Providers") like TypeScript DeclarationInfo

rules_js is a layer on rules_nodejs
build_bazel_rules_nodejs is replaced

Build systems:

Matrix / Hub-and-Spoke

The JS ecosystem took a wrong turn

Grunt and Gulp fell out of favor
Instead, each tool became a Build System
Now each tool needs a plugin for each language

ModernWeb Meetup: Layering in JS tooling
https://www.aspect.dev/resources last one

Note:

I drew this live in my talk
When each column is a tool like webpack or jest
each row is a language like typescript or sass
MxN support matrix - so many plugin authors to trust
A new framework like Qwik has to support every CSS preprocessor??

Like Gulp or Grunt, but way (way) better.

Let's use Bazel!

How Bazel works

In five minutes

Bazel: Loading Phase

Load and evaluate all extensions, BUILD files and macros that are needed for the build.

digraph {
rankdir="LR"
"sources" -> "dependency graph"
"external repos" -> "dependency graph"
}

Note:

bazel has to download stuff!

bazel fetch [targets]

https://blog.aspect.dev/configuring-bazels-downloader

Bazel is full-featured for fetching external deps
Can air-gap, security scan, artifactory, etc
Supply-chain secure, Trust-on-first-use model
Cache based on integrity hashes

Bazel: Dependency Graph

digraph {
  compound=true
  rankdir="LR"

  graph [ fontname="Source Sans Pro", fontsize=20 ];
  node [ fontname="Source Sans Pro", fontsize=18];
  edge [ fontname="Source Sans Pro", fontsize=12 ];

  subgraph cluster1 {
   label="monorepo"
   
   my_app -> my_lib
   my_app -> my_comp
  }
  subgraph cluster2 {
    label="otherrepo"
    
    other_team_lib
  }
  subgraph cluster3 {
    label="registry.npmjs.com"

    request
    react
    express
  }
  
  my_lib -> request
  my_comp -> react
  my_app -> express
  my_app -> other_team_lib
  
}

Note:

It's pretty much what you'd imagine as the dependency graph
Bazel can show you a graph like this for your workspace, try bazel query

bazel query --output=graph [targets]

digraph mygraph {
  rankdir="LR"
  node [shape=box];
  "//examples/macro:node_modules/mocha/dir"
  "//examples/macro:node_modules/mocha/dir" -> "//examples/macro:node_modules/mocha"
  "//examples/macro:node_modules"
  "//examples/macro:node_modules" -> "//examples/macro:node_modules/mocha"
  "//examples/macro:node_modules" -> "//examples/macro:node_modules/mocha-junit-reporter"
  "//examples/macro:node_modules" -> "//examples/macro:node_modules/mocha-multi-reporters"
  "//examples/macro:node_modules/mocha"
  "//examples/macro:node_modules/mocha-junit-reporter/dir"
  "//examples/macro:node_modules/mocha-junit-reporter/dir" -> "//examples/macro:node_modules/mocha-junit-reporter"
  "//examples/macro:test"
  "//examples/macro:test" -> "//examples/macro:_test_srcs\n//examples/macro:test__entry_point"
  "//examples/macro:test" -> "//examples/macro:node_modules/mocha-junit-reporter"
  "//examples/macro:test" -> "//examples/macro:node_modules/mocha-multi-reporters"
  "//examples/macro:node_modules/mocha-junit-reporter"
  "//examples/macro:node_modules/mocha-multi-reporters/dir"
  "//examples/macro:node_modules/mocha-multi-reporters/dir" -> "//examples/macro:node_modules/mocha-multi-reporters"
  "//examples/macro:node_modules/mocha-multi-reporters"
  "//examples/macro:_test_srcs\n//examples/macro:test__entry_point"
}

Bazel: Analysis Phase

digraph {
rankdir="LR"
"dependency graph" -> "action graph" [label = "Bazel rules"]
}

Action: for a requested output, how to generate it from some inputs and tools

e.g. “if you need hello.js, run swc on hello.ts”.

Requires predicting the outputs!

bazel aquery [targets]


digraph {
rankdir="LR"

"index.ts" -> "index.js" [label=swc]
"component.ts" -> "component.js" [label=tsc]
"index.js" -> "bundle.js" [label=esbuild]
"component.js" -> "bundle.js"
"styles.scss" -> "styles.css" [label=sass]
"styles.css" -> "serve" [label=express]
"bundle.js" -> "serve"
}

Bazel: Execution Phase

Execute a subset of the action graph by spawning subprocesses (e.g. node)


digraph {
rankdir="LR"
"changed index.ts" [shape=note]
"changed index.ts" -> "index.js" [label=swc]
"component.ts" [shape=none]
"component.js" [shape=none]
"component.ts" -> "component.js"
"index.js" -> "bundle.js" [label=esbuild]
"component.js" -> "bundle.js"
"styles.scss" [shape=none]
"styles.scss" -> "styles.css"
"styles.css" [shape=none]
"styles.css" -> "serve" [label=express]
"bundle.js" -> "serve"
}

User requested certain targets be built.

Bazel is lazy and will only:

fetch precise dependencies needed
run actions required by the transitive dependency closure of those targets
run actions that are a "cache miss"

Fetch and install npm packages

How npm/yarn solve it

npm install

Install everything needed for the whole
package/workspace

Any build/test script can depend on all npm packages

How Google solves it

Vendor the world: copy npm ecosystem sources into VCS

Never fetches from the internet
Never runs any package installation

You could do it this way too.

How rules_nodejs solved it

Just wrap [npm|yarn] install - install the world

Guaranteed slow when repo rule invalidates
Extra bad when "eager fetching" npm deps

Note:

https://blog.aspect.dev/avoid-eager-fetches

rules_js: ideal solution

Port pnpm to Starlark

re-use pnpm's resolver (via lockfile)
fetch with Bazel's downloader
unpack tarballs with Bazel
re-use @pnpm/lifecycle to run hooks
- these are actions - can be remote cached
link node_modules

Note:

More later about linking the node_modules

https://blog.aspect.dev/rulesjs-npm-benchmarks

Best case:

BUILD file declares fine-grained deps
build only depends on one library
we only fetch/install one library!

Note:

This is possible because BUILD files have finer-grained graph than package.json does

Workspaces

Mix of third-party and first-party deps in a tree of package.json files.

Google: single version policy
rules_nodejs: independent top-level dep installs
rules_js: supports pnpm workspaces!

Note:

yarn, npm, pnpm all have this concept
one version policy is too hard to align all apps on the same versions
independent installs is hard to get transitive deps

Resolving npm dependencies at runtime

How it works in npm

NodeJS programs rely on a node_modules folder

"Was a big mistake" says NodeJS creator, and
Deno fixes it (but here we are )

The location of node_modules is expected to be relative to the location of the importing script.

How Google solves it: patch `require`

Same strategy as "PnP", e.g. Yarn PnP.

not compatible. Many npm packages wrote their own require implementation.

How rules_nodejs solves it: runtime "linker"

Similar to npm link: use symlinks to make monorepo libraries appear in the node_modules tree

Slow beginning of every NodeJS spawn
Links appear in source tree w/o sandbox
Bins don't work with genrule/ctx.actions.run
Not compatible with "persistent workers"

Note:

rules_nodejs also has the Google "patch require" strategy but is now discouraged.
doesn't make deps under the same tree with sources - 'rootDirs' troubles

How rules_js solves it

Linker is now just a standard Bazel target

Node.js tools assume the working dir is a single tree of src/gen/node_modules: we can do that!

"link" to bazel-bin/node_modules/...
copy sources to bazel-bin
actions first cd bazel-out/[arch]/bin

Note:
This requires copy_to_bin and a bit of care in custom rules.

How to use rules_js

Documentation and migration guide:
https://docs.aspect.build/rules_js

Install

Copy the WORKSPACE snippet from latest release.

https://github.com/aspect-build/rules_js/releases

Adopt pnpm

Just run pnpm install and check that your workflows work.

A few npm packages still have "hoisting bugs" where they don't declare correct dependencies and accidentally rely on npm or yarn-specific layout.

Note:

there are good workarounds for broken packages

import `pnpm-lock.yaml`

npm_translate_lock converts to Bazel's format (Starlark).

WORKSPACE










load("@aspect_rules_js//npm:npm_import.bzl",
     "npm_translate_lock")
npm_translate_lock(
    name = "npm",
    pnpm_lock = "//:pnpm-lock.yaml",
)

# Load the starlark version of the lockfile
load("@npm//:repositories.bzl", "npm_repositories")
npm_repositories()

Note:

Check in the result, or compute on-the-fly.
Can also import individual npm packages with no lockfile.
you can look in $(bazel info output_base)/external/npm to see what was generated

Link the npm packages

BUILD (next to package.json)



load("@npm//:defs.bzl", "js_link_all_packages")

js_link_all_packages()

Result of bazel build :all is now






# the virtual store
bazel-bin/node_modules/.aspect_rules_js 
 # symlink into the virtual store
bazel-bin/node_modules/some_pkg
# If you used pnpm-workspace.yaml:
bazel-bin/packages/some_pkg/node_modules/some_dep

bazel build examples/...

Link first-party packages

First declare the package…

my-lib/BUILD









load("@aspect_rules_js//npm:defs.bzl", "npm_package")

npm_package(
    name = "lib",
    srcs = [
        "index.js",
        "package.json",
    ],
)

Note:

See rules_js/examples/lib

Link first-party packages

… then link to bazel-bin/node_modules tree…

app/BUILD






load("@aspect_rules_js//npm:defs.bzl", "npm_link_package")

npm_link_package(
    name = "node_modules/@mycorp/mylib",
    src = "//examples/lib"
)

…then depend on it just like it came from npm!

app/BUILD








js_binary(
    name = "my_app",
    data = [
        "//:node_modules/react-dom",
        "//:node_modules/@mycorp/mylib",
    ],
    entry_point = "index.js",
)

Note:

this is a desirable property, so that as you slurp in your manyrepos to the monorepo, the use sites don't change

Running npm tools

Just call the bin entries from package.json
Write a macro wrapping a bin entry
Write a custom rule
Use an existing custom rule (e.g. rules_ts vs tsc)

There are also more advanced ways, see rules_js/examples

Note:
These options are easy, hard, harder, easy

`bin` entries are provided for all packages

BUILD













load("@npm//typescript:package_json.bzl", typescript_bin = "bin")

typescript_bin.tsc(
    name = "compile",
    srcs = [
        "fs.ts",
        "tsconfig.json",
        "//:node_modules/@types/node",
    ],
    outs = ["fs.js"],
    chdir = package_name(),
    args = ["-p", "tsconfig.json"],
)

Note:

this doesn't cause an eager fetch! Bazel doesn't download the typescript package when loading this file

Each bin exposes three rules:

Use	With	To
`foo`	`bazel build`	produce outputs
`foo_binary`	`bazel run`	side-effects
`foo_test`	`bazel test`	assert exit `0`

Wrap existing build system

Use "component libraries" to get coarse granularity

digraph {
rankdir="LR"
vite [label=vite]
vite2 [label=vite]
"lib sources" [shape=none]
"lib sources" -> "vite"
"app sources" [shape=none]
"app sources" -> "vite2"
vite -> vite2
vite2 -> bundle
bundle [shape=none]
}

Pretty fast developer loop in
https://github.com/aspect-build/bazel-examples/tree/main/vue




















load("@npm//vite:package_json.bzl", vite_bin = "bin")
load("@npm//vue-tsc:package_json.bzl", vue_tsc_bin = "bin")

vite_bin.vite(
    name = "dist",
    args = ["build"],
    out_dirs = ["dist"],
    srcs = [...],
)

vue_tsc_bin.vue_tsc_test(
    name = "type-check",
    args = ["--noEmit"],
    data = [...],
)

vite_bin.vite_binary(
    name = "vite",
    data = [...]
)

Note:

next we'll run this "vite" target

ibazel run :vite

Write a Macro

Bazel macros are like preprocessor definitions.

Good way to give "syntax sugar",
compose a few rules, set defaults.

Indistinguishable from custom rules at use site

Example: mocha test





















def mocha_test(name, srcs, args = [], data = [], env = {}, **kwargs):
    bin.mocha_test(
        name = name,
        args = [
            "--reporter",
            "mocha-multi-reporters",
            "--reporter-options",
            "configFile=$(location //examples/macro:mocha_reporters.json)",
            native.package_name() + "/*test.js",
        ] + args,
        data = data + srcs + [
            "//examples/macro:mocha_reporters.json",
            "//examples/macro:node_modules/mocha-multi-reporters",
            "//examples/macro:node_modules/mocha-junit-reporter",
        ],
        env = dict(env, **{
            # Add environment variable so that mocha writes its test xml
            # to the location Bazel expects.
            "MOCHA_FILE": "$$XML_OUTPUT_FILE",
        }),
    )

https://github.com/aspect-build/rules_js/blob/main/examples/macro/mocha.bzl

Note:

Expand a macro with bazel query --output=build

Write a custom rule

Harder and not recommended for most users.

Start from
https://bazel.build/rules/rules-tutorial
and use
https://github.com/bazel-contrib/rules-template

Use an existing custom rule

From https://github.com/aspect-build:

rules_esbuild - Bazel rules for https://esbuild.github.io/ JS bundler
rules_terser - Bazel rules for https://terser.org/ - a JavaScript minifier
rules_swc - Bazel rules for the swc toolchain https://swc.rs/
rules_ts - Bazel rules for the tsc compiler from http://typescriptlang.org

rules_webpack - Bazel rules for webpack bundler https://webpack.js.org/
rules_rollup - Bazel rules for https://rollupjs.org/ - a JavaScript bundler
rules_jest - Bazel rules to run tests using https://jestjs.io
rules_deno - Bazel rules for Deno http://deno.land

… and many more by other vendors

http://docs.aspect.build

Catalog coming soon at https://bazel-contrib.github.io/SIG-rules-authors/

Example custom rule: `ts_project`

No more rootDirs in tsconfig.json

BUILD











load("@bazel_skylib//rules:write_file.bzl", "write_file")

# Create a test fixture that is a non-trivial sized TypeScript program
write_file(
    name = "gen_ts",
    out = "big.ts",
    content = [
        "export const a{0}: number = {0}".format(x)
        for x in range(100000)
    ],
)

BUILD








load("@aspect_rules_ts//ts:defs.bzl", "ts_project")

ts_project(
    name = "tsc",
    srcs = ["big.ts"],
    declaration = True,
    source_map = True,
)

ts_project with custom transpiler

BUILD












load("@aspect_rules_swc//swc:defs.bzl", "swc_transpiler")

ts_project(
    name = "swc",
    srcs = ["big.ts"],
    out_dir = "build-swc",
    transpiler = partial.make(
        swc_transpiler,
        args = ["--env-name=test"],
        swcrc = ".swcrc",
    ),
)

Benchmarks: ts_project w/ SWC

https://blog.aspect.dev/rules-ts-benchmarks

Transpile-only use case on large project

bazel build :devserver

Note:
time for a first, non-incremental build

Putting it all together

Sophisticated teams can assemble their own toolchain.

Create an entire JS build system just by composing existing tools in a macro!

Note:

"This is our moment": Build system authors can now build on top of Bazel and go back to composing tools

Example: an entire custom build system called
"differential loading":









































def differential_loading(name, entry_point, srcs):
    "Common workflow to serve TypeScript to modern browsers"

    ts_project(
        name = name + "_lib",
        srcs = srcs,
    )

    rollup_bundle(
        name = name + "_chunks",
        deps = [name + "_lib"],
        sourcemap = "inline",
        config_file = "//:rollup.config.js",
        entry_points = {
            entry_point: "index",
        },
        output_dir = True,
    )

    # For older browsers, we'll transform the output chunks to es5 + systemjs loader
    bin.babel(
        name = name + "_chunks_es5",
        srcs = [
            name + "_chunks",
            "es5.babelrc",
            ":node_modules/@babel/preset-env",
        ],
        output_dir = True,
        args = [
            "../../../$(execpath %s_chunks)" % name,
            "--config-file",
            "../../../$(execpath es5.babelrc)",
            "--out-dir",
            "$(@D)",
        ],
    )

    terser_minified(
        name = name + "_chunks_es5.min",
        src = name + "_chunks_es5",
    )

Roadmap

rules_js 1.0.0 is available now

Coming soon

Gazelle extension to generate BUILD files from srcs
Bazel 6.0 package manager: bzlmod instead of WORKSPACE
https://blog.aspect.dev/bzlmod

Thank you!

These slides: https://hackmd.io/@aspect/rules_js

Thanks conference organizers and everyone who helped launch rules_js.

Come work with us on OSS! http://aspect.dev/careers
Paid support and consulting: http://aspect.dev
Our projects: github.com/aspect-build

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

Introductions

Who is Alex Eagle

What is Aspect

What is Bazel

When to consider Bazel for frontend

None of those apply?

What is NodeJS

What is pnpm

What is rules_nodejs

Build systems:

Matrix / Hub-and-Spoke

How Bazel works

Bazel: Loading Phase

Bazel: Dependency Graph

Bazel: Analysis Phase

Bazel: Execution Phase

Fetch and install npm packages

How npm/yarn solve it

How Google solves it

How rules_nodejs solved it

rules_js: ideal solution

Workspaces

Resolving npm dependencies at runtime

How it works in npm

How Google solves it: patch require

How rules_nodejs solves it: runtime "linker"

How rules_js solves it

How to use rules_js

Install

Adopt pnpm

import pnpm-lock.yaml

Link the npm packages

Link first-party packages

Link first-party packages

Running npm tools

bin entries are provided for all packages

Wrap existing build system

Write a Macro

Write a custom rule

Use an existing custom rule

Example custom rule: ts_project

Putting it all together

Roadmap

Thank you!

What is `pnpm`

What is `rules_nodejs`

How Google solves it: patch `require`

import `pnpm-lock.yaml`

`bin` entries are provided for all packages

Example custom rule: `ts_project`