rules_ts benchmarks

tags: blog

–DRAFT–

Aspect's rules_ts is a port of rules_nodejs's @bazel/typescript package that provides a ts_project rule layered on top of rules_js, Aspect's new high-performance Bazel rule set for Javascript, designed from the group up with performance in mind.

The ts_project rule from rules_ts has the same API as its predecessor from @bazel/typescript, but it leaves the gate with performance improvements that were not possible under rules_nodejs.

To learn about how rules_js also makes npm dependencies fast with Bazel check out our rules_js npm benchmarks

In this post, we'll compare build times for ts_project from rules_ts against its predecessor, ts_project from @bazel/typescript, as well as against ts_library from @bazel/concatjs (the original TypeScript rule from Google), and against the vanilla TypeScript compiler, tsc.

Methology

These benchmarks wre run against a generated TypeScript code base of 5 features, 10 modules per feature, 10 components per module and 1001 lines of code per component. This makes for a total of 555 TypeScript files containing 500995 lines of TypeScript code in aggregate.

For the Bazel build, each module maps to one Bazel target for a total of 55 ts_project/ts_library targets.

Configuration

These benchmarks were run on a,

MacBook Pro (16-inch 2019)
2.4 GHz 8-Core Intel Core i9
64 GB 2667 MHz DDR4
Running macOS Monterey 12.3.1

Versions of typescript and rule sets used were,

• TypeScript 4.6.3
• build_bazel_rules_nodejs 5.5.0
• @bazel/typescript 5.5.0
• @bazel/concatjs 5.5.0
• aspect_rules_ts 0.7.0
• aspect_rules_swc PR#57 (a soon to be landed performance enhancement which uses a new pure rust CLI for swc)

Full builds vs. "devserver" builds

In these benchmarks we measure two different scenarios:

1. A full clean build (bazel build ...) followed by an incremental bazel build ... after making a change to a leaf TypeScript file.
2. A clean "devserver" build (bazel build :devserver), which emulates a typical developer workflow of building while running a tool such as a devserver, followed by an incremental bazel build :devserver after making a change to a leaf TypeScript file.

The "devserver" scenario is an important measure that emulates the typical local development workflow of coding while running tools such as a devserver or a test runner such as jest. These tools are often run in watch mode while making changes to source code. The faster build times are on changes the shorter the round-trip-time is to get feedback on those changes.

Ideal build times to maximize developer productivity are less than 1 second on changes to leaf nodes and less than 10 seconds on changes that affect large parts of the graph. Ideally these ideal times are sustained even on large projects.

ts_project vs. ts_library

ts_project was originally developed in rules_nodejs as an alternative to ts_library to provide a cleaner API better suited for the many ways TypeScript is used outside of Google. While the API was better suited for the wild, it could not compete with ts_library, a heavily optimized and deeply integrated wrapper around the TypeScript compiler, on performance.

The new ts_project from rules_ts has significantly reduced the performance gap with ts_library by adding first-class support for Bazel workers.

rules_js, which rules_ts is layered on, has made first-class worker support in rules_ts possible by doing away with the dynamic runtime node_modules linking that rules_nodejs uses.

While ts_library can still slightly outpace ts_project with worker mode in full clean build times, in the "devserver" scenario ts_project has an significant advantage over ts_library by allowing you to configure a separate tool for transpiliation.

In these benchmarks, we'll measure ts_project configured with swc as the transpiler. swc is an order of magnitude faster that TypeScript for pure transpilation but it does not type check, so TypeScript is still used for type checking in this split configuration.

The split configuration also removes type checking from the build graph for devserver and test targets, so only transpiliation is needed to build them, reducing the round-trip-time on changes when running such targets by an order of magnitude. Type-checking is handled in separate targets that can be run explitly or with the catch-all bazel build ....

Results

Without further ado, here are the results of the benchmarks.

Full clean builds

Save as SVGSave as PNGView SourceView Compiled VegaOpen in Vega Editor

ts_library leads the pack for full (transpilation & type checking) clean build times. It has been heavily optimized inside Google and is integrated deeply with TypeScript compiler internals. The ts_project API, however, is not well suited for the many ways that TypeScript projects are configured outside of Google and it does not integrate well with many other rules and tools in the frontend ecosystem.

rules_ts's ts_project is a competetive runner up. It makes significant performance gains over its predecessor from @bazel/typescript by adding first-class support for worker mode.

This is only our initial pass at worker mode for ts_project and we believe we can optimize it further in the future by taking advantage of Bazel features such as multiplexed workers. Stay tuned for future performance improvements in this rule.

Incremental full builds

Save as SVGSave as PNGView SourceView Compiled VegaOpen in Vega Editor

All the Bazel rules measured are relatively close in incremental full build times with ts_library taking the lead and ts_project from rules_ts with worker-mode and swc for transpilation runner up. In this benchmark, vanilla tsc is slowest but in smaller projects it can be quite fast.

Clean "devserver" builds

Save as SVGSave as PNGView SourceView Compiled VegaOpen in Vega Editor

Clean "devserver" builds is where ts_project configured with swc as the transpiler really stands out and is an order of magnitude faster than the rest. The 500,000+ lines TypeScript code in this benchmark take even the heavily optimized ts_library more than 40 seconds to build while swc can transpile the same in 3 seconds flat.

swc is fast enough to spawn to be configured as one target per TypeScript file, which means that with remote execution the 555 .ts file targets in this benchmark could be distributed across 555 remote executors and transpile near instantaneously. ts_library, on the other hand, does not split into one target per file so it could parallelize into only 55 targets with remote execution in this benchmark.

Incremental "devserver" builds

Save as SVGSave as PNGView SourceView Compiled VegaOpen in Vega Editor

On incremental "devserver" builds, where ts_library and even ts_project without swc are fairly fast, ts_project configured with swc for transpilation is still an order of magnitude faster.

The bottom line

With rules_ts, front-end developers can finally get the near instant round-trip-times they are used to with optimized front-end build systems such as Vite with Bazel.

We feel this improvement, along with fast npm dependency management will get more web developers on board with building with Bazel.

Bazel & JavaScript is about to get a whole lot better!