build-std as a rustup feature

If there was a rustup toolchain customize flag which..

..took the current toolchain (or one provided)
..copied it to a new toolchain w/out the pre-built standard library
..downloaded rust-src component for it (or used the already
downloaded component)
..invoked Cargo to build the standard library with the users'
specified options, Cargo features, or crates (e.g. core and not
alloc)
- Using RUSTC_BOOTSTRAP or any other preferred mechanism^[1]
..copied the built standard library into the new toolchain

..then this would be sufficient to allow users to build std for
targets where it is not distributed by the Rust project and to
re-build std with compiler options which need to be used in
every crate (e.g. sanitizers, branch protection, etc).

This approach avoids friction by embracing std as special and not
just a regular dependency - a natural consequence of this is that
rustup is the appropriate place to implement build-std.

$ # Customize a toolchain with flags that shouldn't
$ # be mixed between crates, e.g. `-Zbranch-protection`
$ # (assuming aarch64-unknown-linux-gnu host)

$ rustup toolchain customize \
    branch-protected \
    --base-toolchain=stable -- \
    -Zbranch-protection=pac-ret
customizing toolchain 'stable'
building libstd for aarch64-unknown-linux-gnu..
built libstd
created toolchain 'branch-protected'

$ cargo +branch-protected build

$ # Customize a toolchain with different Cargo features
$ # (assuming aarch64-unknown-linux-gnu host)

$ rustup toolchain customize \
    small-std \
    --base-toolchain="1.81.0" \
    --features=optimize_for_size,backtrace
customizing toolchain '1.81.0'
building libstd for aarch64-unknown-linux-gnu..
built libstd
created toolchain 'small-std'

$ cargo +small-std build

$ # Customize a toolchain for a niche target
$ # (assuming aarch64-unknown-linux-gnu host)

$ rustup default nightly
$ rustup target add aarch64-unknown-freebsd
$ rustup toolchain customize freebsd-std
customizing toolchain 'nightly'
building libstd for aarch64-unknown-linux-gnu..
built libstd
building libstd for aarch64-unknown-freebsd..
built libstd
created toolchain 'freebsd-std'

$ cargo +freebsd-std build --target aarch64-unknown-freebsd

$ # Customize a toolchain for a niche target with options
$ # which aren't supported on the host
$ # (assuming a `x86_64-unknown-linux-gnu` host)

$ rustup default nightly
$ rustup target add aarch64-unknown-linux-gnu
$ rustup toolchain customize \
    branch-protected 
    --skip=x86_64-unknown-linux-gnu \
    -Zbranch-protection=pac-ret
customizing toolchain 'nightly'
skipping libstd for x86-64-unknown-linux-gnu..
building libstd for aarch64-unknown-linux-gnu..
built libstd
created toolchain 'branch-protected'

$ cargo +branch-protected build
error[E0463]: can't find crate for `std`
  |
  = note: the `x86_64-unknown-linux-gnu` target may not be installed
  = help: consider downloading the target with `rustup target add x86_64-unknown-linux-gnu`
  = help: consider building the standard library from source with `rustup target customize`

error: aborting due to 1 previous error

$ cargo +branch-protected build --target aarch64-unknown-linux-gnu

Crucially, this approach means that Cargo and rustc don't need to
know about build-std, both are given a valid sysroot which can be
used in the same way as the toolchains distributed by the Rust
project can. Cargo's existing implementation of build-std could
be entirely removed.

Implementing build-std within rustup instead of Cargo is much
simpler - the implementation can be built largely using existing
functionality in rustup, the only new part is invoking Cargo to
build the rust-src component^[2]. A very simple prototype has been
created to demonstrate that this is possible.

In contrast, Cargo needs to integrate a newly-built std into its
unit graph and consider various trade-offs involved in the user
experience for build-std in Cargo (do users add -Zbuild-std to
every command? do users add a build-std subcommand? does Cargo
try to guess when to rebuild std?). These challenges are a
consequence of building std being stateful in a way that Cargo's
other functionalities are not^[3].

rustup's existing support for rust-toolchain.toml could be
extended for when projects require customized toolchains.

This approach will need to consider compatibility with other
features in rustup that operate on targets, such as..

Adding a component (e.g. rust-analyzer)
- Download the components from the base toolchain.
Updating a toolchain (1.82.0 to 1.83.0)
- Keep track of which toolchain was customized and the
  provided options so that the libstd of the updated
  base toolchain can be re-customised.
Adding a target
- Same as above but with the existing sources and the
  new target

There are two non-technical perspectives that this approach
could challenge:

std is special and not just-another-dependency
- std being just-another-dependency is often cited as a
  motivation for build-std, but this constraint
  requires build-std be a Cargo feature
rustup only downloads and moves toolchains around
- this is a increase in the scope of rustup - while rustup's
  purpose is to manage Rust toolchains and sysroots, and that
  is what build-std requires, invoking Cargo to build std is
  different-in-kind than existing rustup functionality.

From a user experience perspective, this approach is likely very
suitable for users who need to customize a toolchain infrequently
and then use it regularly. It's less ideal for experimentation and
a hypothetical ideal of having std automatically built whenever the
set of options/features that a project requires doesn't match the
current std.

A promising approach for enabling core/alloc/std to be built
without RUSTC_BOOTSTRAP is allowing crates in the sysroot
to use nightly features. This is compatible with this
approach (if rust-src includes vendored dependencies).
Alternatives like having a crate name whitelist are difficult
because regular crates that are dependencies of std use
nightly features when they are a std dependency. ↩︎
LLVM's compiler-rt would need to be included in rust-src
for profiler-builtins to be built this way. ↩︎
Introducing a subcommand to Cargo that built std for a
project for all subsequent commands would feel out-of-place
as Cargo doesn't have any commands that behave like this
currently. Any "state" like this is normally encoded in
Cargo.toml or .cargo/config, but whether std should be
re-built isn't appropriate for a for-all-users Cargo.toml
and might work for .cargo/config. Providing -Zbuild-std
to every command or trying to intelligently determine when
std should be built are both alternatives to maintaining
a is-build-std-being-used state. ↩︎