# Systems Patterns from a Compiler Driver

## Highway to Compatibility (AC/DC rustc edition)

Context for this one: stable-mir-json depends on rustc internals that change on roughly a weekly cadence, and before this PR those internal API calls were scattered across the entire codebase. [PR-123](https://github.com/runtimeverification/stable-mir-json/pull/123) introduces a `src/compat/` module that isolates all rustc internal API usage behind a single boundary, makes `rust-toolchain.toml` the single source of truth for the rustc commit hash, and adds git worktree support for the enormous rust checkout used by UI tests. The patterns here are about building an adapter for an API you don't own that changes on a weekly cadence, validating abstraction boundaries empirically rather than hoping, and managing large dependency checkouts without losing your build caches.

## 1. Build an Adapter for an API That Changes Often

### The situation

stable-mir-json depends on rustc internals (`rustc_middle`, `rustc_smir`, `rustc_span`). These are explicitly unstable; the Rust team changes them frequently and without notice. Before this PR, calls into these crates were scattered across `printer.rs`, all 5 `mk_graph/` files, and `driver.rs`. When rustc changed an API (which happens roughly every nightly), fixing it meant hunting through the entire codebase for affected call sites. The blast radius of any single API break was "somewhere in these 2000+ lines of code across 7 files, good luck."

### The approach

Create a `src/compat/` module organized by concern:

| Submodule | What it wraps |
|-----------|---------------|
| `bridge.rs` | Stable-to-internal conversions (`Instance`, `InstanceKind`, unevaluated consts) |
| `mono_collect.rs` | `tcx.collect_and_partition_mono_items()` and symbol name resolution |
| `output.rs` | Output filename resolution from the compiler session |
| `spans.rs` | `span_to_location_info()` and `FileNameDisplayPreference` |
| `types.rs` | Type queries: generics, signatures, discriminants, attributes |
| `mod.rs` | Crate re-exports and common type aliases (`TyCtxt`, `DefId`) |

The boundary contract is explicit: rustc implementation details go through compat; the stable MIR public contract flows through directly. When `stable_mir` changes its types (e.g., `Rvalue::AddressOf` gaining a new variant), any consumer has to adapt; that's expected. But when `collect_and_partition_mono_items` changes its return type from a tuple to a `MonoItemPartitions` struct, only `compat/mono_collect.rs` needs to change.

A concrete example: the `stable_mir` crate was later renamed to `rustc_public`. Without the compat layer, that's a change to every file with `extern crate stable_mir`. With the compat layer, it's one line in `compat/mod.rs`:

pub extern crate rustc_public as stable_mir;

Every consumer imports via `use crate::compat::stable_mir`, so nothing else changes.

Credit to jberthold, code-owner and reviewer, who caught that some compat functions were thin wrappers that just forwarded to `crate::compat::...::blah(args..)`. These were inlined at the call site; the compat layer wraps unstable *boundaries*, not every function call. The distinction matters: if a wrapper doesn't absorb any instability (the function signature is the same on both sides), it's not doing compat work; it's just indirection. Indirection without purpose is negative value; it hides call sites without reducing maintenance.

### The principle

**Adapter layer for unstable dependencies.** When you depend on an API that changes frequently and you can't control, concentrate all usage behind an adapter. The goal is not abstraction for its own sake; it's blast-radius containment. A single module that changes with every nightly bump is much cheaper to maintain than N scattered call sites that each need independent attention.

(The more precise term for what `compat/` does is an *anti-corruption layer*, from Eric Evans' *Domain-Driven Design* (Chapter 14, "Maintaining Model Integrity" - See [Wiki entry](https://en.wikipedia.org/wiki/Domain-driven_design#Context_Mapping_patterns)). An adapter translates one interface shape to another; an anti-corruption layer goes further by actively defending your domain model against ongoing instability in a foreign system. The distinction matters here because `compat/` isn't a one-time translation; it's a boundary we maintain forever, and the maintenance *is the point*. But "adapter" communicates the core idea to a wider audience, so I'll use that.)

The adapter also serves as documentation: reading `compat/` tells you exactly which rustc internals the project uses, which is invaluable when evaluating a toolchain bump. We can diff a candidate nightly's API against our compat surface and know immediately what breaks, without grepping the whole codebase.

But be disciplined about what goes in the adapter. If a wrapper doesn't reduce change propagation (because the wrapped function's signature is stable, or because the wrapper just forwards arguments), it belongs at the call site. The compat layer's value is proportional to how often it absorbs a change that would otherwise propagate; thin wrappers that never change dilute that signal.

## 2. Validate an Abstraction Boundary with Stress Tests

### The situation

It's easy to claim "the compat layer absorbs all rustc API changes." But is that actually true? Maybe we missed a call site. Maybe some future API change crosses the boundary in a way the layer can't handle. We could stare at the code and convince ourselves it's right, but the only way to *know* is to subject it to the exact kind of stress it's designed to handle: a real toolchain jump with real API breaks.

### The approach

Two spike branches tested multi-month toolchain jumps on top of the compat layer.

**6-month jump** (nightly-2024-11-29 to nightly-2025-06-01, rustc 1.85 to 1.89):

| Change | Where absorbed |
|--------|---------------|
| `collect_and_partition_mono_items` tuple to `MonoItemPartitions` struct | `compat/mono_collect.rs` |
| `RunCompiler::new().run()` to `run_compiler()` | `driver.rs` |

**13-month jump** (nightly-2024-11-29 to nightly-2026-01-15, rustc 1.85 to 1.94):

| Change | Where absorbed |
|--------|---------------|
| `stable_mir` renamed to `rustc_public` | `compat/mod.rs` (one-line alias) |
| `rustc_smir` renamed to `rustc_public_bridge` | `compat/mod.rs`, `driver.rs` |
| `FileNameDisplayPreference::Remapped` removed | `compat/spans.rs` |

All rustc internal API changes were fully contained in `compat/` and `driver.rs`. Changes that affected `printer/` and `mk_graph/` were all stable MIR public API evolution: `Rvalue::AddressOf` changing from `Mutability` to `RawPtrKind`, `StatementKind::Deinit` being removed, `AggregateKind::CoroutineClosure` being added, and so on. These are the kind of changes any consumer of stable MIR would need to handle, regardless of whether they have a compat layer. The boundary held exactly where it was supposed to.

The 13-month jump also exposed the cost of the original `mk_graph/` gap. Before this PR, `mk_graph/` files used `extern crate stable_mir` directly (bypassing the compat layer). When `stable_mir` was renamed to `rustc_public`, all 5 mk_graph files needed `extern crate rustc_public as stable_mir`. After the PR, a future rename would be one line in `compat/mod.rs`. The spike demonstrated the cost of the gap; the PR closes it.

### The principle

**Empirical boundary validation: prove your abstractions under stress.** Don't just claim your abstraction works; verify it by subjecting it to the exact stress it's designed to handle. A 6-month toolchain jump exercises the adapter with real API changes, not hypothetical ones. If the boundary holds, you know the design is sound. If it leaks, you know where to strengthen it.

The invariant being tested here is simple: are all rustc internal API usages confined to compat/? To check that, we construct a scenario where violations become visible, run it, and observe the result.

Note that the spike branches are disposable. They're not meant to land; they're experiments that test a structural hypothesis. The cost is a few hours of fixing build errors on a throwaway branch. The payoff is high-confidence evidence about whether the abstraction boundary actually works, which is worth considerably more than a code-review assertion that it "should" work. There's a second benefit that justified the hours of build-error fixes: the spikes exposed blind spots I hadn't considered. I'd been thinking about code paths absorbing API changes, but I'd completely ignored the test and CI flows. Those broke too, and they broke *outside* `compat/`, which meant the boundary was narrower than I thought. The spikes told me to widen it!

## 3. Consolidate Scattered Constants

### The situation

The pinned nightly's rustc commit hash was needed in three places: `rust-toolchain.toml` (which pins the nightly but doesn't record the commit), the CI workflow (hardcoded), and the UI test scripts (looked up manually and pasted). Three copies of the same derived value, each maintained independently. When the nightly bumped, someone had to update all three; if they forgot one, things broke in ways that were hard to diagnose ("the UI tests are checking out the wrong commit" doesn't obviously trace back to "someone updated CI but forgot the test script").

### The approach

Add a `[metadata]` section to `rust-toolchain.toml` with the commit hash, and make everything else read from it:

[metadata]
rustc-commit = "a2545fd6fc66b4323f555223a860c451885d1d2b"

This is still a manually-maintained value, which creates a sync obligation. But it's a *single* value in a *single* file, and it changes exactly when the nightly changes (so you're already editing `rust-toolchain.toml`). The alternative was two sync obligations in two different files. One is strictly better than two.

A natural question: why not derive the commit hash from `rustc -vV` instead of declaring it at all? For the local scripts, you can (they have the right toolchain installed via `rust-toolchain.toml`). But CI needs the commit hash *before* it has the right toolchain installed; the commit is used to check out the rust source for UI tests, which happens before the toolchain is fully set up. You can't run `rustc -vV` if you haven't installed the right `rustc` yet.

(A later PR, covered in this series, took the derivation further: eliminating the `[metadata]` section entirely and computing the commit hash from the running compiler at every point of use. That's the strictly better solution for contexts where the toolchain is available. But at the time of this PR, the incremental improvement from "three hardcoded copies" to "one declared source" was the right step.)

`yq` (the Go version, mikefarah/yq) reads the value:

RUSTC_COMMIT=$(yq -r '.metadata.rustc-commit' rust-toolchain.toml)
if [ -z "$RUSTC_COMMIT" ]; then
    echo "Error: metadata.rustc-commit not found in rust-toolchain.toml"
    exit 1
fi

The explicit error check matters. Without it, a missing `[metadata]` section gives you an empty `RUSTC_COMMIT` variable, which propagates silently and eventually produces an inscrutable `git checkout` failure three steps later. Failing loudly at the point where the value is expected converts a mystery into a diagnosis.

### The principle

**Declaration consolidation: reduce sync obligations incrementally.** When a value is duplicated across N files, the first step is consolidation: reduce N copies to one. The second step (if your context supports it) is derivation: compute the value instead of declaring it. Both steps reduce sync obligations; derivation eliminates them entirely.

| Strategy | Sync obligations | Failure mode |
|----------|-----------------|--------------|
| N hardcoded copies in N files | N | Stale value, detected by whoever notices |
| Single declared value read by all consumers | 1 (one file) | Stale value, detected by CI |
| Derive at point of use (`rustc -vV`) | 0 | Requires toolchain installed |

This PR moved from the top row to the middle. A later PR (Part 01 of this series) moved from the middle to the bottom. Each step was an incremental improvement that was correct for its context. The lesson isn't "always derive"; it's "reduce sync obligations as far as your execution context allows, and recognize that 'as far as possible' may change as your infrastructure matures."

## 4. Git Worktrees: Isolate Branches to Preserve Build Caches

Keep each compiler commit in its own workspace with [git-worktrees](https://git-scm.com/docs/git-worktree) so switching versions does not invalidate build artifacts or trigger expensive recompilation.

### The situation

UI tests require the Rust compiler source to be checked out at a specific commit. The rust repository is enormous (4+ GB), and the cost of switching commits becomes very real when bumping nightlies.

In a normal clone, `git checkout <new-commit>` rewrites hundreds of thousands of files. That invalidates build caches and forces incremental compilation to start from scratch if you previously built at the old commit. For `rustc`, where a full build takes 15+ minutes, this becomes a meaningful cost every time you switch between nightlies. When testing compatibility across multiple versions, that can happen several times a day.

Long rebuild times don't just slow the machine; they slow the developer. Each forced pause increases the likelihood of context switching, making the workflow more expensive than the raw build time suggests. The real cost isn't only CPU cycles; it's interrupted thought.

### The approach

Use a bare repo with git worktrees. Each worktree shares the same object store but keeps its own working tree and build artifacts:

# One-time setup: convert to bare
cd .. && mv rust rust-tmp
git clone --bare [e-mail]:rust-lang/rust.git rust

# Create a worktree for a specific commit
cd rust
git worktree add ./a2545fd a2545fd6fc66 --detach

The `--detach` flag creates a worktree with a detached HEAD at the specified commit, which is exactly what we want: we're not developing on these commits, just building and testing against them.

The test scripts auto-detect bare vs. regular repos and create (or reuse) worktrees:

if [ -f "$RUST_DIR_ROOT/HEAD" ]; then
    # Bare repo: create detached worktree
    WORKTREE_DIR="$RUST_DIR_ROOT/${RUSTC_COMMIT:0:12}"
    if [ ! -d "$WORKTREE_DIR" ]; then
        git -C "$RUST_DIR_ROOT" worktree add "$WORKTREE_DIR" "$RUSTC_COMMIT" --detach
    fi
    RUST_DIR="$WORKTREE_DIR"
else
    # Regular clone: just checkout
    git -C "$RUST_DIR_ROOT" checkout "$RUSTC_COMMIT"
    RUST_DIR="$RUST_DIR_ROOT"
fi

The bare-repo detection (`[ -f "$RUST_DIR_ROOT/HEAD" ]`) checks for a bare repo's `HEAD` file directly in the root directory. A regular clone has `HEAD` inside `.git/`; a bare repo has it at the top level. This lets the same script work for users who have a regular clone (fallback to `git checkout`) and those who've set up the bare-repo-plus-worktrees pattern.

Switching between commits is `cd`; no files get rewritten, no caches get invalidated. When the nightly bumps, a new worktree gets created; the old one sticks around and you can prune it whenever with `git worktree prune`.

### The principle

**Worktree isolation for multi-version dependencies.** When you need the same repository at multiple commits (for testing, bisecting, or just comparing), git worktrees give you isolated working copies that share a single object store. The tradeoff is disk space (each worktree has its own checked-out files and build artifacts) vs. time (no cache invalidation, no file rewrites, instant switching). For a project like rustc where a full build takes 30+ minutes, the disk space is cheap relative to the rebuild time.

The bare-repo pattern is particularly nice: the "main copy" has no working tree at all, and every useful state is an explicit worktree with a meaningful name (we use the commit hash prefix, e.g., `a2545fd6fc66`). There's no ambiguity about which commit a worktree represents; it's right there in the directory name. And because the object store is shared, creating a new worktree is fast (just checking out files from the local store; no network fetch required, assuming you've already fetched the commit).

The dual-mode detection in the script (bare vs. regular) is important for adoption. You don't want to force all contributors to restructure their rust checkout; the bare-repo pattern is a power-user optimization. The script should work with the simple setup and be faster with the optimized one. Making the optimization transparent means contributors discover it when they hit the pain (slow nightly switches), not when they read the setup docs.