Field projections design meeting

Before We Start…

Purpose of this Meeting

The design space is very big, lots of mutually exclusive changes as well as hard to balance interests between expressiveness and ergonomics. For this reason, this meeting shouldn't try to produce a working solution. The purpose is to perform vibe checks and get everyone on the same page as to what problems should be addressed by a field projection language feature. Additionally, the meeting should discern which design axioms should guide the development of the feature. We should also evaluate the current approach and green-light it for a lang experiment. Finally, we can try to tackle some of the open design dilemmas that the current approach has surfaced.

To summarize, this meeting should:

• Determine the use-cases a field projection language feature must be able to solve.
• Select design axioms for creating the field projection language feature.
• Decide if the current approach is ready for a lang experiment.
• Provide guidance on the open design dilemmas.

Project Goal

This design meeting is the first for the field projection project goal 2025H2. Note that this document goes into more details than the goal, so only the table at the bottom is important to read.

Motivation

Field projections are available on references: given x: &Struct, we can create a reference to a field (&Field) with &x.field. A field projection language feature makes this kind of operation available to all types. The syntax of a projection is similar to reborrowing: @expr->field and @mut expr->field.

Custom Reference Types

Many of the types that benefit from field projections fall into the category "custom reference types". These are types that behave similarly to &/&mut, for example:

• &mut MaybeUninit<T> the referent may be uninitialized,
• cell::Ref[Mut]<'a, T> carries runtime borrow info to access the RefCell<T>,

&mut MaybeUninit<T>

While &mut MaybeUninit<T> already is a reference, it doesn't provide field projections to &mut MaybeUninit<U>. There is an unstable function transpose(MaybeUninit<[T; N]>) -> [MaybeUninit<T>; N] that does this projection for arrays, so it's only natural to also want to have it for structs.

Projections for &mut MaybeUninit<T> would allow initializing a struct step-by step more safely:

struct Point {
    x: i32,
    y: i32,
}

struct Player {
    hp: usize,
    pos: Point,
}

impl Player {
    fn init(this: &mut MaybeUninit<Self>) -> &mut Self {
        let hp: &mut MaybeUninit<usize> = @mut this->hp;
        (@mut this->point->x).write(0);
    //   ^^^^ this is inferred like `&mut` would be, you can also write:
        this->point->y.write(0);
        unsafe { this.assume_init_mut() } // the only unsafe block needed
    }
}

In this small example just assigning Foo { ... } directly would of course also work, but there are several reasons when that's not possible:

• different functions initialize different fields,
• only some parts of the struct should be initialized,
• the entire struct would be too big for the stack, but every field fits.

cell::Ref[Mut]<'a T>

Both cell::Ref[Mut]<'a, T> already provide a map function that is specifically designed to give users access to field projections. However that is not enough: if one wants to project two fields simultaneously, map cannot be used, as it consumes the Ref[Mut] by-value. For this reason the map_split function exists; but it too only permits to project two fields simultaneously.

Field projections allow any number of different fields to be projected simultaneously, allow interleaving with control flow and even allow the use of the projected value afterwards if all field borrows end before then:

#[derive(Debug)]
struct Data {
    x: usize,
    y: i64,
    z: bool,
    w: usize,
}

fn handle(mut data: RefMut<'_, Data>, mut out: &mut usize) {
    let mut x: RefMut<'_, usize> = @mut data->x;
    if *x == 0 {
        *out = 0;
        out = &mut *x;
    }
    compute(@mut data->y);
    if *@mut data->z {
    // ^^^^^ we cannot use `Deref` here, as that would invalidate the projection
    // to `x` that we want to use via `out` below.
        let w = @mut data->w;
        mem::swap(out, &mut *w);
    }
    // All projections of fields end before here, so we can use the entire struct again.
    info!("{data:?}");
}

NonNull<T> and *{const,mut} T

While these types aren't references, for field projections they behave exactly the same way as &/&mut. So given a valid *const Struct, one is able to project to *const Field with unsafe means today. However, doing so is very verbose.

struct Point {
    x: i32,
    y: i32,
}

unsafe fn project_x(point: NonNull<Point>) -> NonNull<i32> {
    unsafe { NonNull::new_unchecked(&raw const (*point.as_ptr()).x) }
}
// with field projections:
unsafe fn project_x2(point: NonNull<Point>) -> NonNull<i32> {
    unsafe { @point->x }
}

These projections need to be unsafe, because NonNull could wrap around the address space. For raw pointers, see this comment.

Other Reference Types

There are lots of other types that could benefit from field projections in the same way as the previous examples of this section. A non-exhaustive list:

• pyo3::pycell::PyRef[Mut]<'_, T>: there is an open issue to add a map function.
• Cpp[Mut]Ref<T>: a pointer to a C++ value.
• &[mut] Cell<T>: to write only part of a bigger struct, making it available for bigger, possibly non-Copy types.
• &Atomic<T>: if T can be a struct made up of primitives or a tuple of primitives.
• ArcRef<T>: an Arc<T> with separate pointers to the refcount and data, permitting projecting to fields.
• VolatilePtr<'_, T>: a pointer that always uses volatile operations (wanted by the general community & RfL to remove the dma_read macro (also dma_write)).

Rust for Linux also has several custom reference types that would benefit from field projections:

• &[mut] Untrusted<T> used to mark data as untrusted (it needs to be validated before it can be read, but writing it is fine).
• SeqLockRef<'_, T>: similar to ArcRef, have separate pointers for the sequence count and the data, allowing to lock & only care about a field of the entire stored data.
• UserPtr<T> a pointer into userspace that must not be conflated with a normal pointer.
• Ptr<'_, T> essentially just an &Opaque<T>, so a valid pointer whose pointee might be uninitialized & concurrently modified.

Pin ergonomics

While pin ergonomics are most likely going to be hard-coded by the compiler, field projections should be able to simulate pin ergonomics. Users might want to have custom pinned references and equip that type with field projections. For example, the Mutex<T> type in RfL is structurally pinning its contents and thus MutexGuard<'_, T> must behave like Pin<&mut T> for accessing fields. We would like to provide field projections on that type to allow convenient access.

Both Unpin-guided pin ergonomics as implemented in the current lang experiment and marked-fields pin ergonomics will should be supported by field projection. An example for the marked field approach:

struct FairRaceFuture<F1, F2> {
    fair: bool,
    #[pin]
    fut1: F1,
    #[pin]
    fut2: F2,
}

With structurally pinned fields fut1, fut2 we get the following projections when given f: &pin mut FairRaceFuture<F1, F2>:

• @mut f->fair has type &mut bool,
• @mut f->fut1 has type &pin mut fut1,
• @mut f->fut2 has type &pin mut fut2,

There also are projections like @f->fair to &bool.

RCU (Read-Copy-Update)

Rust for Linux has a very exotic use-case for field projections: creating a safe abstraction for RCU. This use-case is very important to RfL, because RCU is used fairly often in the kernel and without field projections there most likely is no way to write a safe abstraction for it. It also is included in C++26 so it could see more uses in other projects the future.

RCU is an acronym for "read copy update" and it is an efficient locking mechanism for rarely written data. It synchronizes between readers and writers; however, it cannot synchronize between multiple writers. For this reason it must combined with an additional locking mechanism that only the writers use. This external lock can be a mutex, a spinlock or another type of lock.

Now the problem for a Rust abstraction is that in RfL we also adopted the design that locks contain the data they protect. But in the RCU case, only the writer is supposed to take the external lock, the reader is only takes the RCU lock, circumventing the external lock. This is where field projections come in. They enable us to project through the mutex in the reader case. The writer just normally locks the mutex.

// The data stored by the driver. Usually as `Arc<rfl::Mutex<Data>>`
struct Data {
    // RCU-protected data must be stored in a pointer type wrapped by RCU.
    // internally, this is just an `AtomicPtr<Config>`
    #[pin]
    cfg: Rcu<Box<Config>>,
    // Data that isn't protected by RCU is just stored normally.
    other: i32,
}

struct Config {
    size: usize,
    name: &'static CStr,
}

// Reader case:
fn size(data: &rfl::Mutex<Data>) -> usize {
    // `&rfl::Mutex<T>` allows projecting to fields of type `Rcu<U>`,
    // but not to other fields (that would be unsound).
    let cfg: &Rcu<Box<Config>> = @data->cfg;
    // now we begin the critical read section of RCU.
    let rcu = rcu::read_lock();
    let cfg: &Config = cfg.get(&rcu);
    cf.size
}

// Writer case:
fn set_config(data: &rfl::Mutex<Data>, config: Config) {
    // We normally lock the mutex.
    let mut data: rfl::MutexGuard<'_, Data> = data.lock();

    // Normal data can just be handled as usual using field projections.
    // (remember: the RfL mutex pins its data, so we need projections here)
    data->other = 42;

    // Maybe somewhat surprisingly, we can obtain a `Pin<&mut Rcu<...>>`:
    let cfg: Pin<&mut Rcu<Box<Config>>> = @mut data->cfg;
    // This is fine, because it will use `UnsafePinned` to remove `noalias`,
    // and it's API cannot be abused by having both `&pin mut` and `&`
    // concurrently existing. The pinning is needed to prevent `mem::swap`
    // being used to change the value without using atomics.

    // `Rcu::set` has `Pin<&mut Rcu>` as the receiver, so it can only be called,
    // if the external lock is taken.
    let _old = cfg.set(Box::new(config));

    // When `_old` is dropped, `synchronize_rcu` is executed, waiting for a
    // grace period to end (ie all currently active critical read sections
    // must end). This guarantees that any readers still holding onto a
    // pointer to the contents of `_old` have a valid pointer.
    drop(_old);
}

Field Reflection Use-Cases

The current approach uses at its core a kind of field reflection to inform generic code about which field is being projected. Outside of field projection, this kind of field reflection also is useful. In Rust for Linux, there are types that currently use a const ID: usize generic to allow having multiple differentiable fields with the same type (well except for the ID of course). With field reflection, we could use the field type/identifier itself as the ID and remove the generic improving the ergonomics of the API.

Design Axioms

The project goal lists these two design axioms:

• Simple & easy-to-remember syntax. Using field projections in a non-generic context should look very similar to normal field accesses.
• Broadly applicable solution. Field projections should be very general and solve complex projection problems such as pin-projections and Rcu<T>.

To better explain the intentions behind these axioms here is some more prose text: the ergonomics of using field projections really is the most important piece. People already write map functions today to perform projections when references are backing the actual data. And as later mentioned it is possible to implement unergonomic projections with macros. But adding them with a native operator is the biggest benefit that comes from a language feature. The second axiom ensures that the feature is beneficial to as many applications of field projections as possible. If we're introducing a language feature for field projections, it better serve all cases of the concept.

What we're willing to compromise on – and the current approach heavily compromises this aspect – is the ergonomics of adding projection support to a type (so implementing the traits for the projection operator). Since we're trying to fit so many use-cases into a single feature & want to have nice ergonomics for the usage site, this part necessarily has to suffer. This aspect reminds me of the current API design of the Try & Residual traits, they also are very ergonomic to use and serve many different use-cases; but they also pay the same price of complexity at the implementation site.

Aside from these two axioms I don't think there are others that we should adhere to, since most other decisions are going to be a balancing act and there will not really be an easy answer. But if you have any suggestions feel free to post them.

The Current Approach

A discussion at RustWeek resulted in the following overarching design idea:

• Make the behavior of &/&mut w.r.t. field projections available for all types.

This means that projections follow the same borrow checker behavior and can reuse the existing compiler code. They also are very familiar to Rust developers and thus relatively easy to teach. In this section, we go over a high level overview of the current design.

Library Implementation

Note that a library implementation of this approach exists. It has bad ergonomics due to several reasons, but it showcases the various traits for the operator. Additionally many of the motivational examples are implemented there, so it gives a good impression of what it would be like to have it as a language feature.

The ergonomic shortcomings of the library solution are:

• structs must be annotated with #[derive(HasFields)] to be available for projecting
• field types aren't defined locally next to the struct, but by the field-projection crate, thus one can't implement traits on them, preventing extensions to the field types
• can only project through a single field
• can only project variables and not arbitrary expressions
• need to wrap the projection expression with the p! macro
• in order to be able to have projections on var, one needs to write start_proj!(var);
• cannot omit @[mut] when calling a function on a projected value
• lifetime problems when returning a projected value, for this reason there is a special move projection that needs to be used in such a case (note the unfortunate naming, as this is something different from "Moving projections" that I have mentioned sometimes)

An example showing the last five issues:

fn buffer_config<'a>(&'a self, rcu_guard: &'a RcuGuard) -> &'a BufferConfig {
    let buf: &'a RcuMutex<Buffer> = &self.buf;
    start_proj!(move buf);
    p!(@move buf->cfg).read(rcu_guard)
}
// with a language feature it could look like this:
fn buffer_config<'a>(&'a self, rcu_guard: &'a RcuGuard) -> &'a BufferConfig {
    (&self.buf)->cfg.read(rcu_guard)
}

Field Reflection (Field trait & generated types)

To make field information available in generic code, every field of every struct is associated with a field representing type. It can be named via the field_of!(Struct, field) macro that uses the same syntax as offset_of!. A field representing type implements the Field marker trait:

pub unsafe trait Field: UnalignedField {}

// fields of `repr(packed)` structs only implement this trait
pub unsafe trait UnalignedField {
    /// The type that this field is a part of.
    type Base: ?Sized;

    /// The type of this field.
    type Type: ?Sized;

    /// The offset of this field in bytes.
    const OFFSET: usize;
}

The associated types and the constant must be correct (that's why the trait is unsafe). So one can rely on them and write code like:

fn project_ref<F: Field>(r: &F::Base) -> &F::Type
where
    // needed for the `.cast` call below
    F::Type: Sized,
{
    unsafe { &*ptr::from_ref(r).add_bytes(F::OFFSET).cast::<F::Type>() }
}

Field representing types are also generated for nested fields, so field_of!(Struct, bar.baz) will return a field representing type that has Struct as the base, the offset will be that of bar plus that of baz inside of bar and the type will be that of bar.

Syntax & Operator Traits

We add a new operator: @[mut] $base:expr$(->$field:ident)+ which is governed by three traits:

• Projectable: this trait is a supertrait of the next two and is not generic over the projected field. It is similar to Receiver, since it stores of which struct type we project the fields of.
• Project<F>: this trait is used for @base->field desugaring,
• ProjectMut<F>: this trait is used for @mut base->field desugaring,

pub trait Projectable: Sized {
    type Inner: ?Sized;
}

Projectable is used to figure out which field type to use in the expression @[mut] $base:expr$(->$field:ident)+:

• the expression $base must be of a type that implements Projectable,
• the compiler then can look up the Inner type and then use field_of!(Inner, $($field).+) as the field type
• then Project[Mut]<F> is used for the desugaring where F is the field type from above.

pub trait Project<F>: Projectable
where
    F: UnalignedField<Base = Self::Inner>,
{
    type Output<'a>
    where
        Self: 'a;

    unsafe fn project<'a>(this: *const Self) -> Self::Output<'a>
    where
        Self: 'a;
}

ProjectMut looks identical except the Output and project functions have a Mut/_mut suffix.

Finally there is one last marker trait:

pub unsafe trait SafeProject: Projectable {}

That makes all projection operations on that type not require an unsafe block.

Desugaring

Before @[mut] base->field is desugared, the compiler performs a borrow-check:

• @base->field is treated exactly as &base'.field where base' is a new variable of type &<typeof(base) as Projectable>::Inner. And the borrow lasts for exactly the same length as is required by the projection (@base->field is of type Project<...>::Output<'a>).
• @mut base->field is treated exactly as &mut base'.field
• accessing base in any other way counts as using the whole of base', so field projections and any other access cannot be combined (only if the field projection ends before the other operation).

After this borrow check passes, the following desugaring is used for @[mut]base->field:

Project[Mut]::<
    field_of!(
        <typeof(base) as Projectable>::Inner,
        field
    ),
>::project[_mut](&raw {const,mut} base)

When the projection goes through multiple fields, so @base->field1->field2, then those fields are just appended in the field_of! invocation:

Project::<
    field_of!(
        <typeof(base) as Projectable>::Inner,
        field1.field2
    ),
>::project(&raw const base)

Solving the Motivation

We're only going to inspect a few example implementations for time reasons.

See the library implementation for the details, as it implements a lot of the motivational examples (look in the examples/ directory and the src/projections/ directory).

&mut MaybeUninit<T>

Using the current approach to implement field projections for MaybeUninit would look like this:

impl<T> Projectable for &mut MaybeUninit<T> {
    type Inner = T;
}

unsafe impl<T> SafeProject for &mut MaybeUninit<T> {}

impl<'a, T, F> ProjectMut<F> for &'a mut MaybeUninit<T>
where
    F: Field<Base = T>,
    F::Type: Sized + 'a,
{
    type OutputMut<'b>
        = &'b mut MaybeUninit<F::Type>
    where
        Self: 'b;

    unsafe fn project_mut<'b>(this: *mut Self) -> Self::OutputMut<'b>
    where
        Self: 'b,
    {
        let ptr: *mut MaybeUninit<T> = unsafe { this.read() };
        unsafe { &mut *ptr.byte_add(F::OFFSET).cast() }
    }
}

(The library implementation also implements Project, but it looks almost identical to ProjectMut)

Pin<&mut T>

With Marking Structurally Pinned Fields

We need a new field subtrait:

pub unsafe trait PinnableField: UnalignedField {
    /// Either `Pin<&'a mut Self::Type>` or `&'a mut Self::Type`.
    type Projected<'a>
    where
        Self::Type: 'a;

    unsafe fn from_pinned_ref(r: &mut Self::Type) -> Self::Projected<'_>;
}

It is implemented via a derive macro in the crate, but with an edition change it could become the default. Then this is how the traits implemented on Pin<&mut T> look like:

impl<T> Projectable for Pin<&mut T> {
    type Inner = T;
}

unsafe impl<T> SafeProject for Pin<&mut T> {}

impl<'a, T, F> ProjectMut<F> for Pin<&'a mut T>
where
    F: PinnableField<Base = T> + Field<Base = T>,
    F::Type: Sized + 'a,
{
    type OutputMut<'b>
        = F::Projected<'b>
    where
        Self: 'b;

    unsafe fn project_mut<'b>(this: *mut Self) -> Self::OutputMut<'b>
    where
        Self: 'b,
    {
        let r = unsafe { Pin::into_inner_unchecked(this.read()) };
        let ptr: *mut T = r;
        let ptr = unsafe { ptr.byte_add(F::OFFSET).cast() };
        unsafe { F::from_pinned_ref(&mut *ptr) }
    }
}

Current pin_ergonomics lang experiment version

Simulating pin projections using the Unpin rules that the current pin_ergonomics lang experiment define would look like this:

impl<T> Projectable for &pin mut T {
    type Inner = T;
}

unsafe impl<T> SafeProject for &pin mut T {}

impl<'a, T, F> ProjectMut<F> for &'a pin mut T
where
    T: !Unpin,
    F: Field<Base = T>,
    F::Type: Sized + 'a,
{
    type Output<'b>
        = &'b pin mut F::Type
    where
        Self: 'b;

    unsafe fn project_mut<'b>(this: *mut Self) -> Self::OutputMut<'b>
    where
        Self: 'b,
    { /* ... */ }
}

// sadly this impl is overlapping...
impl<'a, T, F> ProjectMut<F> for &'a pin mut T
where
    F: Field<Base = T>,
    F::Type: Sized + Unpin + 'a,
{
    type Output<'b>
        = &'b pin mut F::Type
    where
        Self: 'b;

    unsafe fn project_mut<'b>(this: *mut Self) -> Self::OutputMut<'b>
    where
        Self: 'b,
    { /* ... */ }
}

This doesn't completely work due to the overlapping traits, it could be solved by having a marker trait that is allowed to overlap (the marker trait is implemented for F: Field where F::Base: !Unpin or F::Type: Unpin). "Alternative B" proposed in the tracking issue would work out of the box, since there projection is always allowed. The ProjectMut impl would look like this:

impl<'a, T, F> ProjectMut<F> for &'a pin mut T
where
    T: IsMarkedPinV2,
    F: Field<Base = T>,
    F::Type: Sized + 'a,
{
    type Output<'b>
        = &'b pin mut F::Type
    where
        Self: 'b;

    unsafe fn project_mut<'b>(this: *mut Self) -> Self::OutputMut<'b>
    where
        Self: 'b,
    { /* ... */ }
}

Key Features

This approach informs us about several key features that we want for field projections given that we want to solve the examples from the motivation. Those key features are:

• Support projections on custom wrapper/pointer types (not just &T and &mut T),
  • this one is hopefully obvious: RfL, pyo3, c++ interop & many other projects will want access to projections
• support selective projections in generic wrapper/pointer types via where clauses on the struct and field types,
  • RfL's Rcu<U> use-case needs this as well as pin projections of any flavor
• support distinguishing between shared and exclusive projection at the usage site,
  • needed by mutable pointer types that also have a shared variant, so [Py]Ref[Mut] or Cpp[Mut]Ref
• support distinguishing between user-defined and reference projection at the usage site
  • there already exist Deref operations on several types that would like to have field projections, but those projections don't return references. Thus rendering Deref and the projections incompatible & adding the need to distinguish between them
• support unsafe projections
  • sadly raw pointers and NonNull won't have projections otherwise. we could bite the bullet on this one and just not have their projections, instead we add a new pointer type AllocatedAndAligned<T> that always points at a valid allocation, it could have safe projections
• support projections to different types based on which field is being projected
  • this is required by pin projections when using the field-marking approach

In addition, as a "sniff test", the design should be able to generalize over all types of projection currently supported; i.e., reference and pin projection can be implemented in terms of this (whether or not they actually are), as well as all the use cases we can think of for RFL and interop.

Design Guidance

There are several mutually exclusive interests at play as with any design. This section contains the most pressing issues that need to be resolved in order to move forward. While reaching a consensus on every single one is unlikely (& we want to also get to any questions/discussions posted by participants of the meeting), it would still be great to have some guidance in coming to a decision on these for the lang experiment.

Projecting References of Custom Types

Decision: should &T and &mut T implement Projectable & Project<F> for any field F?

Conflicting Interests:

• If they do implement it, it allows using them in APIs asking for impl Project<F>.
• If they don't, it allows users to have custom projections on &Custom<T> (e.g. &mut MaybeUninit<T>).

(we need to have a decision here, as &T and &mut T are fundamental and thus either have to implement the trait from the get-go, or never implement the trait in the future)

Verdict: (preliminary) having projections for references to custom types is essential for the RCU use-case. It also is very useful in practice and as such we should allow them. This sadly means that &[mut] T cannot be used in generic projection contexts. But since we won't add those in the beginning and we might never add them, we can't let them get in the way of the best design now.

unsafe Projections

Decision: should it be possible to have an unsafe projection operation?

Conflicting Interests:

• NonNull<T> and raw pointers need their projections to be unsafe.
• But this would be the first unsafe operator of Rust that can be overloaded. This might make the unsafety requirements not as obvious as with functions. How would users properly document the safety requirements?

Verdict: (preliminary) we should have unsafe projections, as without them, we lose the ergonomic gain for unsafe Rust.

Optimizing the Last Projection

Decision: should the last projection be handled specially?

Conflicting Interests:

• if yes, it would allow for example ArcRef<T> to prevent an additional increment and decrement of the refcount if the original value is not used afterwards:
  ​​fn coords(point: ArcRef<Point>) -> (ArcRef<i32>, ArcRef<i32>) {
  ​​    let x = @point->x; // increments the refcount
  ​​    let y = @point->y; // increments the refcount
  ​​    (x, y)
  ​​    // now `point` is dropped, decrementing the refcount, but it could have
  ​​    // just been consumed by the `->y` projection instead of incrementing
  ​​    // there & decrementing here.
  ​​}
  

• if no, it would improve the simplicity of the API, as every projection is the same.

Verdict: (preliminary) no, we need to keep the API as simple as possible, as it already is pretty complex.

Syntax

Decision: what syntax should the field projection operator use? The concrete symbols can be decided later when bikeshedding, it's more about where do we place the expression, identifier and extra symbols.

The syntax used in the current approach tries to avoid parenthesis when combining it with other Rust syntax & ensures that it's not ambiguous with other syntax. It also copies the syntax of references:

• it supports projecting through multiple fields: @foo->bar->baz
• the reason for using -> instead of . is to be able to omit the leading @ & not use any parenthesis when
  • calling a function after projecting: foo->bar.baz(),
  • writing to a field after projecting: foo->bar.baz = Baz::new(),

While writing this section, I'm no longer convinced that this is the best syntax, since it might run into ambiguity in the function call & field write cases. So it might be that one of these alternatives fits better:

• use postfix syntax to disambiguate exclusive projections:
  • [e-mail] and [e-mail] (also showing different sigils for the infix operator)
• have different operators for shared/exculsive projections base.@field and base.mut@field
  • this is adding a lot of characters to type in the mut case: [e-mail]@baz (especially when projecting multiple fields), which I don't like, but using something else would probably be confusing

Discussion

The first four discussion topics are the main output of this meeting. We should

• Decide on the use-cases that are supported by the field projection language feature.
• Decide on the design axioms used for the development.
• Green-light the current approach for a lang experiment or suggest changes.
• Express an opinion on the design questions (or decide which direction to go).

Use-Cases

Which of the use-cases explained above should field projections support?

• &mut MaybeUninit<T> (also covers &[mut] Cell<T>, &Atomic<T>, &[mut] Untrusted<T>, Ptr<'_, T>)
• cell::Ref[Mut]<'_, T> (also covers ArcRef<T>, pyo3::cell::PyRef[Mut]<'_, T>, SeqLockRef<'_, T>)
• NonNull<T> (also covers *{const,mut} T, VolatilePtr<'_, T>, UserPtr<T>)
• Pin<&mut T> (also covers rfl::MutexGuard)
• Rcu<T>

Design Axioms

Which of the following design axioms should be used to guide the lang experiment?

• Simple & easy-to-remember syntax.
• Broadly applicable solution.
• Should any other axioms be added?

Lang Experiment Based on the Current Approach

Which changes should be made to the current approach before starting a lang experiment?

Design Guidance

For which of the design questions have we discussed?

• Projecting References of Custom Types
• unsafe Projections
• Syntax
• Optimizing the Last Projection

---

Discussion

Attendance

• People: Josh, Tyler, TC, Benno Lossin, Gary Guo, Miguel, Tomas, Zachary Sample, Rusty Yato, Xiang, Eric Holk, Yosh, Boqun Feng

Meeting roles

• Driver:
• Minutes: Tomas

Benno: I have the purpose of this meeting section at the top of the doc. That's exactly what I'd like to achieve here. The last section (Discussion) to check whether everyone aligns with what the document has to say. Decide on the use-cases, design axioms, if the current approach is a good start for a lang experiment (or have changes suggested) adn get more opinions on the design guideance section.

Vibe Checks

tmandry

I'm excited to have a design that's vetted through so many use cases. This seems like a useful feature in many contexts, and we haven't even started talking about how it could be used for Option and Result. I also see generalizing what &T can do as an important step for having an extensible and general-purpose systems language.

I'm in favor of an experiment, and I agree with the preliminary decisions. I think we'll have to do some exploration of the interaction with related features, like autoref, autoreborrow, and coercions to see whether all of the key features are necessary or whether we can handle the use cases with those other features. As far as syntax goes, I'm leaning in the direction of something like foo.@bar. I also don't personally see anything wrong with foo.@mut bar. (edit: I'd like to extend this to foo.&bar as well.)

Josh

Josh: Enthusiastically in support of having field projections, and all the things that'll enable (e.g. Arc projections).

Very much appreciate that the question about &T and &mut T needs to get settled up front. Would like to explore that further and see if it's possible to have a design that makes &T and &mut T work with projection, so that projections of types that contain intermediate references work well.

Would like to see an ACP to T-libs-api early for ArcRef (and RcRef presumably), so we can get that concept reviewed and approved early to go into nightly.

Have thoughts about syntax but broadly speaking the "shape" seems approximately correct. This may interact with the discussion about applying projections serially rather than all-at-once. Like Tyler, I do like the concept of the .@ approach as well.

TC

TC: Great work. Let's experiment here. I particularly like language features that reduce the need for library surface area, and this is one of those.

We'll need to keep comparing notes as it relates to pin ergonomics, and the discussions we'd had so far have been insightful and have raised interesting and useful points, e.g. about the !Unpin bound approach. I have an open item to talk that one over with Niko.

As mentioned below, I intuitively suspect that building pin into the language, as it relates to projections, will imply that we want @pin mut x->f projections for the same sort of reasons that we want @mut x->f projections, but need to look into this further.

It's worth noting too that this is a long document and a lot of design questions to consider here, and I certainly have only skimmed these and will need to have a more careful look later.

Yosh (not T-Lang)

Yosh: I'm excited to see the use cases listed. I was worried that this would be primarily useful for pin projections, but the proposal feels like it will apply to a broad range of use cases.

Intuitively I feel like syntax might end up being the biggest sticking point here; more so than in many other proposals. While not something that we need to get into in this meeting, that's probably not something that should be left as the last item to get to either.

Eric (not on T-Lang)

eholk: Definitely happy to see experimentation here. It seems like it has to potential to open up a lot of ergonomic opportunities for libraries and such going forward. I'm not sure the syntax is exactly right yet, but that's probably more on me needing to get familiar with it.

Also, it's very cool to see how RfL is driving a lot of language design. Languages are at their best when they have a kill app in mind.

Scott

Short because I'm late getting here, but I've often pondered what Rust would have been like if . was always projected this way (instead of to places), so I think having it around as a primitive thing makes a ton of sense to me. I'm particularly thinking that having a projection as a thing we can talk about would be great for MaybeUninit usage and usage of a hypothetical Packed<_> type, as well as replacing a bunch of offset_of manual addressing, since staying in value-land (avoiding references entirely) has a whole bunch of good underlying advantages in other ways.

Use cases

Benno: Should all the use cases in the document be something we are interested in supporting?

TC: Are you asknig whether this is an exclusive set of use cases? Or whether each of these is valid?

Benno: The latter. Let's agree on a minimum set of cases we want to support.

Tyler: I think all of these seem quite reasonable. None stand out as something we don't want to support. The more use cases we can support, the better. Might benefit having an RFC earlier to help collect the use cases early.

TC: One thing we talked about is RFC-ing a set of requirements / use cases separately.

Tyler: I had trouble with that in the past because I think it can quickly become vapid. But in this case we do have a very concrete solution and problem statement in mind.

TC: To make it not vapid, we could RFC specific requirements that a later RFC would be evaluated against. IETF did that.

Tyler: I think that could be useful here.

TC: Let's talk about the use cases:

• Mutable reference to MaybeUninit<T>
• Raw pointers that and NotNull
• cell::Ref[Mut]<'_, T> (also covers ArcRef<T>, pyo3::cell::PyRef[Mut]<'_, T>, SeqLockRef<'_, T>)
• Pin (depending on the property of the field you may have to change the output type)
• RCU (this restricts which fields you're allowed to do based on type)

Josh: Those last two cases (Pin and RCU) sound closely related.

TC: The only one I have a question about is the fourth one (the pin one). This gets into the question of treatment of pin in the language. We treat Pin as a library type rather than something like &mut. So this leads us to treating it like a library type (like a RefCell). The concept of Pin ergonomics is making it a real builtin type rather than just a library type. I'll focus on that one and we should talk about it.

Benno: In Rust for Linux, the MutexGuard has the inside pinned. So it is a pinned mutable reference. We'd like to have the same projection here for pin. So here, the question is about: do you think the usecase should be same as what field projection would solve.

Josh:

for these use cases. I very much want to see the RCU usecase covered. Would also like to discuss the Option<T> use case.

Josh: There's a case and language feature worth bringing up because it interacts with this: Projecting through Option. If you have Option<&T> you can project through that to a field of T. But if you have an option of a struct (not a reference). You can't materialize that Option through anywhere – it doesn't exist in memory. But we talked about having a special kind of reference that could let us say that "this logically exists even though I can't create an actual reference here". That would let us project through Option in general.

TC: I don't recall the discussion but what you said makes sense and is worth pursuing. The other context is match ergonomics. The natural ways of composing things lead to types that haven't materialized so that may be another place where this may be applicable.

Scott: I don't remember this a discussion for a type. But I do remember discussions about fields.

Josh: Those two things go hand in hand. If you have it as a field type, the compiler can optimize it because it doesn't exist in memory. If you have it as a special kind of reference, you can take that reference and pass it around.

Scott: But only if they're encoded in the same way.

Josh: Only if they're encoded in a fashion similar to an impl subject to monomorphisation.

Scott: Ah so the reference would have to carry metadata like vtable.

Josh: It would have to be metadata in the style of dyn or impl.

If you have Option<Struct>, you can project to &magic Option<Field>.

You don't have the &Option<Field> materialized anywhere. But in reality you can treat it that way.

Josh: The reason I'm bringing this up now is that having this "magic" reference type could interect with how we implement the concept of projection. It generalizes what you project through. This feels similar to what Benno talked about earlier about being to bring up a wrapper. There may be a design opportunity there if we have something like these move-only references where you could make the projections more orthogonal.

tmandry: Fwiw, I think I would spell it Option<&move Field>.

Josh: You can get Option<&Field> already. But you can't get &Option<Field>, because there's no Option<Field> in memory to reference. That's the motivation for having &magic (whatever we call it), so you can get &magic Option<Field>.

TC: I propose we say the lang team reviewed the usecases and didn't immediately see any problems with these usecase being motivating. We'll look later, but we're excited about the experiment and addressing about each of these usecases.

(general agreement)

Benno: Sounds like a lang experiment has been approved?

TC: We're absolutely doing a lang experiment. What you need to do next is create a tracking issue for the lang experiemnt. Link the tracking issue to the project goal too. Ask Tomas if you have any questions. We're championing this. Tyler this goal, I the ping ergonomics. We'll keep talking.

Benno: I think the conflict for &T/&mut T projections and custom projections is good.

Benno: If we want to have custom projections on reference types, we're not allowed ot implement @mut for the reference. The trait implementations are overlapping so we can only have one of the two. If we go the route of allowing it for references, we get the generit benefit but we lose custom projections – for MaybeUninit, Cell types, RCU.

Either: projections on references (&T) or allow implementing projections for &Custom<T>.

impl<T> Projectable for &T {}

impl<F, T> Project<F> for &T 
where
    F: Field<Base = T>
{}

or

impl<T> Projectable for &Custom<T> {}

impl<F, T> Project<F> for &Custom<T>
where
    F: Field<Base = T>
{}

Tyler: Another way of framing the choice is that either we can have @foo.bar mean the exact same thing as &foo.bar when foo: &Struct, OR we can support projections like &mut MaybeUninit<Struct> to &mut MaybeUninit<Field>. I find the second very motivating.

TC: Could we have both with specialization?

Benno: No we cannot, we also have to add the Projectable impls (added above in the code)

Josh: Don't want to open the whole rabbit hole again here (we can handle it async), but if we can project using Custom<T> (without having to use &Custom<T>), then we could independently project through the & and the Custom.

TC: Clearly we shouldn't depend on specialization. I'm pursuing this line of inquiry to understand better how fundamental this dilemma is. If it's not fundamental, sometimes we can come up with an argument that we can convince the compiler of about why to accept this, and then maybe we can make a general language feature out of it.

Benno: The current design couldn't work even with specialization.

Gary: I don't think it's fundamentally unworkable in the compiler, because I draw parallelism to Deref. Compiler can resolve whether you're referencing on a field on a struct or you're referencing a field of the Deref'd target, so it shouldn't fundamentally not work. It might be hard to make it work with trait resolution, but technically if you add an extra generic parameter if can solve conflicting impl issue and you can have compiler magically insert the special generic argument for you, it can work.

Xiang: First a question, does some type F such that F: Field<Base = T> for any type T, or a small but diverging set of type Ts, even exist?

(The meeting ended here.)

---

Understanding the conflict between &T/&mut T projections and custom projections

(suggestion: handle several shorter items first)

Josh: I'd like to better understand the conflict described between supporting projections on references and supporting custom projections as needed for RCU. The nature of the conflict is not obvious to me, and I'm not seeing where in the document that's discussed (other than saying they conflict). Some further elaboration on that would help.

Benno: The problem is overlapping implementations: if we add impl<F, T> Project<F> for &T where F: Field<Base = T> , then it would overlap with the RCU projections impl<F, T, U> Project<F> for &Mutex<T> where F: Field<Base = T, Type = Rcu<U>>. For mutable references it would also overlap with projections on &mut MaybeUninit.

Josh: So, when you mention custom projections on references, you're talking about distinguishing projections on the basis of the type of the field? This seems like something we could potentially overcome by using different types or transparent wrappers; for instance, could we have rfl's Mutex type have a .rcu() method that returns a transparent wrapper, and then write a different projection on that transparent wrapper? I absolutely want to support the RCU use case; I'd also like the orthogonality of having projections on references Just Work.

Benno: That'd be annoying, since it also affects &mut MaybeUninit<T> or &UnsafeCell<T> etc.

Josh: Can you elaborate on the MaybeUninit case, please? (I don't want to break the ergonomics of common cases, but I also think reference projections are important.)

Benno: Sure, if we had a ProjectMut<F> impl for &mut T, then we can't have one for &mut MaybeUninit<T>. Since again they would overlap

Josh: It feels like there's a problem of levels of indirection here, where there should be a projection through &mut and separately a projection through MaybeUninit and they should be possible to combine.

Benno: In an earlier RFC, I had these kinds of projections for structs where you could essentially declare "hey I have this field". So essentially we could say that we are allowed to write field_of!(MaybeUninit<Struct>, field) and it would resolve to impl Field<Base = MaybeUninit<Struct>, Type = MaybeUninit<Field>>. But this will mean that you can only ever project &Custom<T> to &U and this U must be contained within Custom<T> at offset_of!(T, u), Which is pretty restrictive IMO.

Gary: Is this possible to use associated types and have trait resolver understand that F: Field<Base = A> and F: Field<Base = B> cannot possibly overlap if A != B? In the case above, the impl for &_ would have F: Field<Base = Mutex<T>> on &Mutex<T>, while for the impl for &Mutex<_> it would have F: Field<Base = T>.

Benno: in that case we sadly would have matching bases (mutex doesn't have any accessible fields to the outside world & we want projections on the fields of T)

Josh: It feels like there's some overlap here with the conversations we've had about match ergonomics, where you're kinda "matching under a reference", and that leads to the resulting match having an implicit ref. To elaborate on why reference projection seems important, it seems important to be able to project through, for instance, Arc<Something<&T>> and have the & work just as easily as the Arc and the Something.

Benno:

• I haven't added this to the doc, but I have some ideas for a @match operator that would essentially just project ever field that you mention in there (it also allows to support enums).
• How would you imagine the Arc<Something<&T>> projection to look like? Ie what's the output type and which impl would be doing the projection.
• I would be very careful with projections of the kind impl<P: Project<F>> Project<F> for MyType<P>, since that involves a lot of unsafe stuff.

Josh: So, part of the problem here is that I'm wondering if the projection-of-a-projection case actually belongs in the type/trait system, or if it belongs in the compiler that can apply one projection after another without necessarily having to have a transitive impl. (I do appreciate that transitive trait impls can create a lot of problems.)

You need to know how to project from T to U, and from U to V, and I'm wondering if the compiler can treat that as two steps with two separate trait impls, rather than looking for a single impl that says how to project from T to V.

Pin Ergonomics

eholk: I'm interested in the pin ergonomics aspects of this feature. In generaly, I'd say one feature that serves two use cases perfectly is better than two features that each serve one use case perfectly, so if field projections can subsume pin ergonomics, that seems like a win.

Benno: I agree, pin ergonomics probably still want to add &pin references though, so additional syntactic sugar on top of the ones added through field projections.

(Resolved) Is it possible to project from one type to another?

Josh: For instance, can we project directly from Arc to ArcRef, without having to convert the top-level Arc to an ArcRef first?

Benno: Yes that's possible.

@pin mut x->f

TC: Need to analyze this further, but it occurs to me that we might want @pin mut x->f for the same reasons that we want @mut x->f and not just @x->f. The underlying truth behind the pin ergonomics work is that pin is essentially similar to mut in being a separate non-fixed property of a place tracked by the borrow checker. So everywhere that we put mut with respect to references, it ends up making sense to put pin as well. I'd suspect that'd end up being true here too, but as said, it needs specific analysis.

Benno: I don't really follow. If you want to pin your type T, just wrap it in Pin<T> and provide pinning projections on Pin<T> & normal ones on T?

TC: The analysis here involves if and where this analogy breaks down: "if you want to mutate (or force exclusive access to) your type T, just wrap it in &mut T and provide exclusive projections on &mut T and normal ones on T."

tmandry: One relevant question is whether we want Pin<MyPtr<T>> to be meaningful, and whether we want to support projections through that.

Benno: I think that should be the decision of the author of MyPtr<T>.

(Resolved, can summarize) "Optimizing the Last Projection"

Josh: Is it possible to explicitly do this, by writing it out explicitly, even if the compiler isn't helping you by doing that automatically?

Benno: the current API of the Project trait only has one project function that assumes that the value is still accessible (through other projections for example) after it ran. We'd probably have to add a new final_project function that can assume that it's the last projection.

Xiang: maybe we can have a mir-opt to identify project locations where the base is last known to be live, if we have the library support like final_project as mentioned.

Benno: My problem here was the addition of such a final_project method, it makes the API pretty complicated as you now have to implement the two projections at the same time.

tmandry: I think final_project could be added later, as a provided method that defaults to calling project.

tmandry: I think to answer Josh's question though, maybe you can do the "inner" projections as in this RFC and then use an explicit .map_final(|x| x.foo) call or something? It wouldn't be very ergonomic.

Benno:

on adding it later, we could have it be a separate trait that would permit the optimization, then only types that need it can implement it.

Josh: As long as it's possible to add it later,

for not adding it in the initial version.

Borrow checking of field projection and soundness

Before @[mut] base->field is desugared, the compiler performs a borrow-check

Gary: Do we have any other precedent of desugaring after borrow checking (I guess only async functions)? How can we ensure soundness of this?

Xiang: handwavingly I have a hunch that we can track it like "place"s like any other move paths or move out index.

Benno: It doesn't have to happen before desugaring, if we can track the stuff through the desugaring that would work too. I just phrased it that way, because it's much clearer that way.

Xiang: So encoding this information in some opaque type? nvm I have to think through.

Multiple-level of projection

When the projection goes through multiple fields, so @base->field1->field2, then those fields are just appended in the field_of! invocation:

Boqun: I obviously don't have enough knowledge to know the cost of this desugaring, but this means for a:

struct Foo {
    bar: Bar { 
        baz: i32
    }
}

it's going to generate two Field types.

Benno: field types are only generated if you use field_of!, so you only pay for what you use

Boqun: Ok, but would desugaring to two project() calls also work?

Benno: you have to add parenthesis, @(@foo->bar)->baz does two projections and @foo->bar->baz only does one.

Boqun: Oh, I see. I asked this question mostly because I'm curious about the compile time impact. But as I said, I don't have the enough knowledge to make further comment here, but your "pay as you go" argument sounds reasonable to me at the moment.

Unsafe NonNull<T> projections

These projections need to be unsafe, because NonNull could wrap around the address space.

TC: Talk with me about this analysis. I'd interested to see a worked example of the safety comment someone would need to write here to satisfy the proof obligation, and I'm curious if there aren't some constraints we could add somewhere to make a NonNull<T> -> NonNull<Field> projection safe for the many cases where this kind of proof will be trivially satisfiable.

tmandry: I think it would be the same as for calling pointer::offset.

scottmcm: yes, it's always unsafe for the same reason that &raw const (*p).foo is unsafe. Plus it's unsafe for the same reason that NonNull::wrapping_add needs to be unsafe.

scottmcm: The usual way you can prove this is the same way that miri can: the allocation exists, so any movement you do inside that allocation (including the past-the-end) isn't going to wrap the address space. (The allocator, be that for local variables or impl Allocator is thus the root of the "can't wrap the address space" proof.)

New kind of reference

Josh: This is something that has come up in various contexts for years. We've talked about the idea of a type being able to offer a pseudo-reference to a field but not actually "have" a materialized instance of that field. If you actually want an instance you'd have to copy/move/etc, but you can in some ways act as if you have one. For instance, as the simplest example, if you have an Option<Struct>, you can have a &magic Option<Field> (for some bikeshedded magic). You don't actually have an Option<Field> in memory anywhere, but you logically have one.

scottmcm: I don't know how to do this as a type, since different places might want to represent it in different ways. If it's generic or carrying metadata, then it's basically just passing impl GetSetField or dyn GetSetField?

Option projections

Josh: This is potentially complicated due to the issue of materializing the existence of an Option, but it'd be nice to be able to project an Option<&Struct> to an Option<&Field>, as one part of other projections. (e.g. projecting through an Arc with an Option in it).

tmandry: +1, I would hope and expect that this design can handle that.

scottmcm: reminds me of field pointers in C++, as having something we can pass into the option to use to do the projection inside it.