Slides: https://hackmd.io/@aspect/rules_js

Note:
This talk has a bunch of links which you can come find in the slides.

Bazel: most scalable polyglot Build System.

JavaScript & TypeScript: most popular languages.

Note:

• Bazel works with all languages, let's make it good with the most popular ones.
• image from https://octoverse.github.com/#top-languages-over-the-years

Note:

• I'll introduce Bazel and Node.js and explain where they clash, to motivate why we designed rules_js like we did.

Image Not Showing Possible Reasons

• The image file may be corrupted
• The server hosting the image is unavailable
• The image path is incorrect
• The image format is not supported

Learn More →

Introductions

• Alex Eagle
• Aspect Development
• Bazel
• NodeJS
• pnpm
• GH/bazelbuild/rules_nodejs

Who is Alex Eagle

• Worked at Google on DevInfra 2008-2020
• Bazel most of that time: TL for Google's CI, build/test results UI, Angular CLI
• twitter.com/jakeherringbone

What is Aspect

I Co-founded Aspect Development to make Bazel the industry-standard full-stack build system

• https://aspect.dev - Support and consulting to help you adopt Bazel
• https://aspect.build - Products making Bazel easier to use
• https://github.com/aspect-build - rules_js is part of our Bazel rules ecosystem

What is Bazel

• Build system for "every" langugage
• Incremental: re-build proportional to what you changed
• Cached/parallel: distribute over server farm
• Scalable: works for Google's 2 billion line repo
• Unix Philosophy: just spawns subprocesses, which can be any tool

More: https://www.aspect.dev/resources

Note:

• Bazel is an "orchestrator"
• This talk isn't the "bazel sales pitch"

When to consider Bazel for frontend

• Large-scale: 1M SLOC / 100 devs
• Monorepo: same use cases as Nx/Rush/Lerna
• Polyglot/full-stack: parachute anywhere
• Integration testing: fast test against backend
• Have a DevInfra team: economy of scale

More: https://www.aspect.dev/resources

Note:

• https://www.aspect.dev/blog/cboi-continuous-build-occasional-integration

None of those apply?

Small, disconnected JS apps shouldn't use Bazel.

The build system recommended by your framework is well supported for small-to-medium scale.

What is NodeJS

JavaScript engine that runs outside the browser.

Typically used for running dev tools to
build and test JavaScript programs.

Note:

• just a runtime, requires a package manager

What is pnpm

• "Fast, disk space efficient package manager": https://pnpm.io/
• Works with nearly the whole ecosystem
• Used by the https://rushjs.io/ monorepo JS-only build tool
• Happens to fit perfectly with Bazel semantics!

Note:

• Symlinks to a "virtual store" of fetched packages rather than nested layout
• Their lockfile happens to present 100% of the info needed to fetch AND link the node_modules tree

What is rules_nodejs

Bazel rules forked from Google-internal

• toolchain to run hermetic NodeJS interpreter
• shared Bazel interfaces ("Providers") like TypeScript DeclarationInfo

rules_js is a layer on rules_nodejs
build_bazel_rules_nodejs is replaced

Build systems:

Matrix / Hub-and-Spoke

• Grunt and Gulp fell out of favor
• Instead, each tool became a Build System
• Now each tool needs a plugin for each language

ModernWeb Meetup: Layering in JS tooling
https://www.aspect.dev/resources

Note:

• I drew this live in my talk
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
• When each column is a tool like webpack or jest
• each row is a language like typescript or sass
• MxN support matrix - so many plugin authors to trust
• A new framework like Qwik has to support every CSS preprocessor??

Like Gulp or Grunt, but way (way) better.

Let's use Bazel!

How Bazel works

In five minutes

Bazel: Loading Phase

Load and evaluate all extensions, BUILD files and macros that are needed for the build.

digraph {
rankdir="LR"
"sources" -> "dependency graph"
"external repos" -> "dependency graph"
}

Note:

• bazel has to download stuff!

bazel fetch [targets]

https://blog.aspect.dev/configuring-bazels-downloader

• Bazel is full-featured for fetching external deps
• Can air-gap, security scan, artifactory, etc
• Supply-chain secure, Trust-on-first-use model
• Cache based on integrity hashes

Bazel: Dependency Graph

digraph {
  compound=true
  rankdir="LR"

  graph [ fontname="Source Sans Pro", fontsize=20 ];
  node [ fontname="Source Sans Pro", fontsize=18];
  edge [ fontname="Source Sans Pro", fontsize=12 ];

  subgraph cluster1 {
   label="monorepo"
   
   my_app -> my_lib
   my_app -> my_comp
  }
  subgraph cluster2 {
    label="otherrepo"
    
    other_team_lib
  }
  subgraph cluster3 {
    label="registry.npmjs.com"

    request
    react
    express
  }
  
  my_lib -> request
  my_comp -> react
  my_app -> express
  my_app -> other_team_lib
  
}

Note:

• It's pretty much what you'd imagine as the dependency graph
• Bazel can show you a graph like this for your workspace, try bazel query

bazel query --output=graph [targets]

digraph mygraph {
  rankdir="LR"
  node [shape=box];
  "//examples/macro:node_modules/mocha/dir"
  "//examples/macro:node_modules/mocha/dir" -> "//examples/macro:node_modules/mocha"
  "//examples/macro:node_modules"
  "//examples/macro:node_modules" -> "//examples/macro:node_modules/mocha"
  "//examples/macro:node_modules" -> "//examples/macro:node_modules/mocha-junit-reporter"
  "//examples/macro:node_modules" -> "//examples/macro:node_modules/mocha-multi-reporters"
  "//examples/macro:node_modules/mocha"
  "//examples/macro:node_modules/mocha-junit-reporter/dir"
  "//examples/macro:node_modules/mocha-junit-reporter/dir" -> "//examples/macro:node_modules/mocha-junit-reporter"
  "//examples/macro:test"
  "//examples/macro:test" -> "//examples/macro:_test_srcs\n//examples/macro:test__entry_point"
  "//examples/macro:test" -> "//examples/macro:node_modules/mocha-junit-reporter"
  "//examples/macro:test" -> "//examples/macro:node_modules/mocha-multi-reporters"
  "//examples/macro:node_modules/mocha-junit-reporter"
  "//examples/macro:node_modules/mocha-multi-reporters/dir"
  "//examples/macro:node_modules/mocha-multi-reporters/dir" -> "//examples/macro:node_modules/mocha-multi-reporters"
  "//examples/macro:node_modules/mocha-multi-reporters"
  "//examples/macro:_test_srcs\n//examples/macro:test__entry_point"
}

Bazel: Analysis Phase

digraph {
rankdir="LR"
"dependency graph" -> "action graph" [label = "Bazel rules"]
}

Action: for a requested output, how to generate it from some inputs and tools

e.g. “if you need hello.js, run swc on hello.ts”.

Requires predicting the outputs!

bazel aquery [targets]

digraph {
rankdir="LR"

"index.ts" -> "index.js" [label=swc]
"component.ts" -> "component.js" [label=tsc]
"index.js" -> "bundle.js" [label=esbuild]
"component.js" -> "bundle.js"
"styles.scss" -> "styles.css" [label=sass]
"styles.css" -> "serve" [label=express]
"bundle.js" -> "serve"
}

Bazel: Execution Phase

Execute a subset of the action graph by spawning subprocesses (e.g. node)

digraph {
rankdir="LR"
"changed index.ts" [shape=note]
"changed index.ts" -> "index.js" [label=swc]
"component.ts" [shape=none]
"component.js" [shape=none]
"component.ts" -> "component.js"
"index.js" -> "bundle.js" [label=esbuild]
"component.js" -> "bundle.js"
"styles.scss" [shape=none]
"styles.scss" -> "styles.css"
"styles.css" [shape=none]
"styles.css" -> "serve" [label=express]
"bundle.js" -> "serve"
}

User requested certain targets be built.

Bazel is lazy and will only:

• fetch precise dependencies needed
• run actions required by the transitive dependency closure of those targets
• run actions that are a "cache miss"

Image Not Showing Possible Reasons

• The image file may be corrupted
• The server hosting the image is unavailable
• The image path is incorrect
• The image format is not supported

Learn More →

Fetch and install npm packages

How npm/yarn solve it

npm install

Install everything needed for the whole
package/workspace

Any build/test script can depend on all npm packages

How Google solves it

Vendor the world: copy npm ecosystem sources into VCS

• Never fetches from the internet
• Never runs any package installation

You could do it this way too.

How rules_nodejs solved it

Just wrap [npm|yarn] install - install the world

Note:

• https://blog.aspect.dev/avoid-eager-fetches

rules_js: ideal solution

• re-use pnpm's resolver (via lockfile)
• fetch with Bazel's downloader
• unpack tarballs with Bazel
• re-use @pnpm/lifecycle to run hooks
  • these are actions - can be remote cached
• link node_modules

Note:

• More later about linking the node_modules

https://blog.aspect.dev/rulesjs-npm-benchmarks

Best case:

• BUILD file declares fine-grained deps
• build only depends on one library
• we only fetch/install one library!

Note:

• This is possible because BUILD files have finer-grained graph than package.json does

Workspaces

Mix of third-party and first-party deps in a tree of package.json files.

Google: single version policy
rules_nodejs: independent top-level dep installs
rules_js: supports pnpm workspaces!

Note:

• yarn, npm, pnpm all have this concept
• one version policy is too hard to align all apps on the same versions
• independent installs is hard to get transitive deps

Image Not Showing Possible Reasons

• The image file may be corrupted
• The server hosting the image is unavailable
• The image path is incorrect
• The image format is not supported

Learn More →

Resolving npm dependencies at runtime

How it works in npm

NodeJS programs rely on a node_modules folder

"Was a big mistake" says NodeJS creator, and
Deno fixes it (but here we are

The location of node_modules is expected to be relative to the location of the importing script.

How Google solves it: patch require

Same strategy as "PnP", e.g. Yarn PnP.

How rules_nodejs solves it: runtime "linker"

Similar to npm link: use symlinks to make monorepo libraries appear in the node_modules tree

Note:

• rules_nodejs also has the Google "patch require" strategy but is now discouraged.
• doesn't make deps under the same tree with sources - 'rootDirs' troubles

How rules_js solves it

Node.js tools assume the working dir is a single tree of src/gen/node_modules: we can do that!

• "link" to bazel-bin/node_modules/...
• copy sources to bazel-bin
• actions first cd bazel-out/[arch]/bin

Note:
This requires copy_to_bin and a bit of care in custom rules.

Image Not Showing Possible Reasons

• The image file may be corrupted
• The server hosting the image is unavailable
• The image path is incorrect
• The image format is not supported

Learn More →

How to use rules_js

Documentation and migration guide:
https://docs.aspect.build/rules_js

Install

Copy the WORKSPACE snippet from latest release.

https://github.com/aspect-build/rules_js/releases

Adopt pnpm

Just run pnpm install and check that your workflows work.

A few npm packages still have "hoisting bugs" where they don't declare correct dependencies and accidentally rely on npm or yarn-specific layout.

Note:

• there are good workarounds for broken packages

import pnpm-lock.yaml

npm_translate_lock converts to Bazel's format (Starlark).

WORKSPACE

load("@aspect_rules_js//npm:npm_import.bzl",
     "npm_translate_lock")
npm_translate_lock(
    name = "npm",
    pnpm_lock = "//:pnpm-lock.yaml",
)

# Load the starlark version of the lockfile
load("@npm//:repositories.bzl", "npm_repositories")
npm_repositories()

Note:

• Check in the result, or compute on-the-fly.
• Can also import individual npm packages with no lockfile.
• you can look in $(bazel info output_base)/external/npm to see what was generated

Link the npm packages

BUILD (next to package.json)

load("@npm//:defs.bzl", "js_link_all_packages")

js_link_all_packages()

Result of bazel build :all is now

# the virtual store
bazel-bin/node_modules/.aspect_rules_js 
 # symlink into the virtual store
bazel-bin/node_modules/some_pkg
# If you used pnpm-workspace.yaml:
bazel-bin/packages/some_pkg/node_modules/some_dep

bazel build examples/...

Link first-party packages

First declare the package…

my-lib/BUILD

load("@aspect_rules_js//npm:defs.bzl", "npm_package")

npm_package(
    name = "lib",
    srcs = [
        "index.js",
        "package.json",
    ],
)

Note:

• See rules_js/examples/lib

Link first-party packages

… then link to bazel-bin/node_modules tree…

app/BUILD

load("@aspect_rules_js//npm:defs.bzl", "npm_link_package")

npm_link_package(
    name = "node_modules/@mycorp/mylib",
    src = "//examples/lib"
)

…then depend on it just like it came from npm!

app/BUILD

js_binary(
    name = "my_app",
    data = [
        "//:node_modules/react-dom",
        "//:node_modules/@mycorp/mylib",
    ],
    entry_point = "index.js",
)

Note:

• this is a desirable property, so that as you slurp in your manyrepos to the monorepo, the use sites don't change

Running npm tools

1. Just call the bin entries from package.json
2. Write a macro wrapping a bin entry
3. Write a custom rule
4. Use an existing custom rule (e.g. rules_ts vs tsc)

There are also more advanced ways, see rules_js/examples

Note:
These options are easy, hard, harder, easy

bin entries are provided for all packages

BUILD

load("@npm//typescript:package_json.bzl", typescript_bin = "bin")

typescript_bin.tsc(
    name = "compile",
    srcs = [
        "fs.ts",
        "tsconfig.json",
        "//:node_modules/@types/node",
    ],
    outs = ["fs.js"],
    chdir = package_name(),
    args = ["-p", "tsconfig.json"],
)

Note:

• this doesn't cause an eager fetch! Bazel doesn't download the typescript package when loading this file

Each bin exposes three rules:

Wrap existing build system

Use "component libraries" to get coarse granularity

digraph {
rankdir="LR"
vite [label=vite]
vite2 [label=vite]
"lib sources" [shape=none]
"lib sources" -> "vite"
"app sources" [shape=none]
"app sources" -> "vite2"
vite -> vite2
vite2 -> bundle
bundle [shape=none]
}

Pretty fast developer loop in
https://github.com/aspect-build/bazel-examples/tree/main/vue

load("@npm//vite:package_json.bzl", vite_bin = "bin")
load("@npm//vue-tsc:package_json.bzl", vue_tsc_bin = "bin")

vite_bin.vite(
    name = "dist",
    args = ["build"],
    out_dirs = ["dist"],
    srcs = [...],
)

vue_tsc_bin.vue_tsc_test(
    name = "type-check",
    args = ["--noEmit"],
    data = [...],
)

vite_bin.vite_binary(
    name = "vite",
    data = [...]
)

Note:

• next we'll run this "vite" target

ibazel run :vite

Write a Macro

Bazel macros are like preprocessor definitions.

Good way to give "syntax sugar",
compose a few rules, set defaults.

Indistinguishable from custom rules at use site

Example: mocha test

def mocha_test(name, srcs, args = [], data = [], env = {}, **kwargs):
    bin.mocha_test(
        name = name,
        args = [
            "--reporter",
            "mocha-multi-reporters",
            "--reporter-options",
            "configFile=$(location //examples/macro:mocha_reporters.json)",
            native.package_name() + "/*test.js",
        ] + args,
        data = data + srcs + [
            "//examples/macro:mocha_reporters.json",
            "//examples/macro:node_modules/mocha-multi-reporters",
            "//examples/macro:node_modules/mocha-junit-reporter",
        ],
        env = dict(env, **{
            # Add environment variable so that mocha writes its test xml
            # to the location Bazel expects.
            "MOCHA_FILE": "$$XML_OUTPUT_FILE",
        }),
    )

https://github.com/aspect-build/rules_js/blob/main/examples/macro/mocha.bzl

Note:

• Expand a macro with bazel query --output=build

Write a custom rule

Harder and not recommended for most users.

Start from
https://bazel.build/rules/rules-tutorial
and use
https://github.com/bazel-contrib/rules-template

Use an existing custom rule

From https://github.com/aspect-build:

• rules_esbuild - Bazel rules for https://esbuild.github.io/ JS bundler
• rules_terser - Bazel rules for https://terser.org/ - a JavaScript minifier
• rules_swc - Bazel rules for the swc toolchain https://swc.rs/
• rules_ts - Bazel rules for the tsc compiler from http://typescriptlang.org

• rules_webpack - Bazel rules for webpack bundler https://webpack.js.org/
• rules_rollup - Bazel rules for https://rollupjs.org/ - a JavaScript bundler
• rules_jest - Bazel rules to run tests using https://jestjs.io
• rules_deno - Bazel rules for Deno http://deno.land

… and many more by other vendors

http://docs.aspect.build

Catalog coming soon at https://bazel-contrib.github.io/SIG-rules-authors/

Example custom rule: ts_project

No more rootDirs in tsconfig.json

BUILD

load("@bazel_skylib//rules:write_file.bzl", "write_file")

# Create a test fixture that is a non-trivial sized TypeScript program
write_file(
    name = "gen_ts",
    out = "big.ts",
    content = [
        "export const a{0}: number = {0}".format(x)
        for x in range(100000)
    ],
)

BUILD

load("@aspect_rules_ts//ts:defs.bzl", "ts_project")

ts_project(
    name = "tsc",
    srcs = ["big.ts"],
    declaration = True,
    source_map = True,
)

ts_project with custom transpiler

BUILD

load("@aspect_rules_swc//swc:defs.bzl", "swc_transpiler")

ts_project(
    name = "swc",
    srcs = ["big.ts"],
    out_dir = "build-swc",
    transpiler = partial.make(
        swc_transpiler,
        args = ["--env-name=test"],
        swcrc = ".swcrc",
    ),
)

Benchmarks: ts_project w/ SWC

https://blog.aspect.dev/rules-ts-benchmarks

Transpile-only use case on large project

bazel build :devserver

Note:
time for a first, non-incremental build

Putting it all together

Sophisticated teams can assemble their own toolchain.

Create an entire JS build system just by composing existing tools in a macro!

Note:

• "This is our moment": Build system authors can now build on top of Bazel and go back to composing tools

Example: an entire custom build system called
"differential loading":

def differential_loading(name, entry_point, srcs):
    "Common workflow to serve TypeScript to modern browsers"

    ts_project(
        name = name + "_lib",
        srcs = srcs,
    )

    rollup_bundle(
        name = name + "_chunks",
        deps = [name + "_lib"],
        sourcemap = "inline",
        config_file = "//:rollup.config.js",
        entry_points = {
            entry_point: "index",
        },
        output_dir = True,
    )

    # For older browsers, we'll transform the output chunks to es5 + systemjs loader
    bin.babel(
        name = name + "_chunks_es5",
        srcs = [
            name + "_chunks",
            "es5.babelrc",
            ":node_modules/@babel/preset-env",
        ],
        output_dir = True,
        args = [
            "../../../$(execpath %s_chunks)" % name,
            "--config-file",
            "../../../$(execpath es5.babelrc)",
            "--out-dir",
            "$(@D)",
        ],
    )

    terser_minified(
        name = name + "_chunks_es5.min",
        src = name + "_chunks_es5",
    )

Roadmap

rules_js 1.0.0 is available now

Coming soon

• Gazelle extension to generate BUILD files from srcs
• Bazel 6.0 package manager: bzlmod instead of WORKSPACE
  https://blog.aspect.dev/bzlmod

Thank you!

These slides: https://hackmd.io/@aspect/rules_js

Thanks conference organizers and everyone who helped launch rules_js.

Come work with us on OSS! http://aspect.dev/careers
Paid support and consulting: http://aspect.dev
Our projects: github.com/aspect-build