---
title: "Design meeting 2024-02-28: Arbitrary Self Types v2"
tags: ["T-lang", "design-meeting", "minutes"]
date: 2024-02-28
---

# 2024-02-28: Arbitrary Self Types v2

## The decision to be made

Options:

1. Do not progress arbitrary self types;
2. Do a simple version, with support for new smart pointer types, solving most use-cases;
3. Do a more complex version which can additionally support raw pointers, `NonNull` and `Weak` receivers, solving some additional use-cases.

Compromise option: we agree to do the simple version, so long as we see a path to later support raw pointers, `Weak` and `NonNull`.

## What's "arbitrary self types", anyway?

This is the pattern we want to support:

struct MyRc<T>(T);

struct Dino;

impl Dino {
    fn roar(self: MyRc<Self>) { }   // note receiver type
}

fn main() {
    let dino = MyRc(Dino);
    dino.roar();
}

Stable Rust already supports method calls like this:

impl Dino {
    fn roar(self: std::rc::Rc<Self>) { }
}

fn main() {
    let dino = std::rc::Rc::new(Dino);
    dino.roar();
}

but this is possible only because `Rc` is special (along with `Box`, `Arc` and `Pin`).

We can't currently call methods on some custom smart pointer type. In today's [stable Rust](https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=ae1dfe5e3875f4736d98600b016a3bbf), this breaks, saying:

error[E0307]: invalid `self` parameter type: MyRc<Dino>
 --> src/main.rs:6:19
  |
6 |     fn roar(self: MyRc<Dino>) { }
  |                   ^^^^^^^^^^
  |
  = note: type of `self` must be `Self` or a type that dereferences to it
  = help: consider changing to `self`, `&self`, `&mut self`, `self: Box<Self>`, `self: Rc<Self>`, `self: Arc<Self>`, or `self: Pin<P>` (where P is one of the previous types except `Self`)

There's already a form of arbitrary self type support in nightly, tied to use of `Deref`:

#![feature(arbitrary_self_types)]

struct MyRc<T>(T);

impl<T> std::ops::Deref for MyRc<T> {
    type Target = T;
    fn deref(&self) -> &Self::Target {
        &self.0
    }
}

struct Dino;

impl Dino {
    fn roar(self: MyRc<Self>) { }
}

fn main() {
    let dino = MyRc(Dino);
    dino.roar();
}

The use of `Deref` has some disadvantages:

* some smart pointer types contain a pointer to a `T` which can't safely be turned into a reference that's compliant with Rust's aliasing rules (there are sometimes workarounds here using ZSTs, or`MaybeUninit` and `UnsafeCell` to wrap the `T`)
* some smart pointer types `P<T>` don't always have a `T`, e.g. imagine wrapping a pointer which might be null.
* some smart pointer types want to prevent the vending of normal references, and only vend special references which might (for instance) have `Drop` semantics

In these cases, implementing `Deref::deref` is impossible, so in v2 we propose using a `Receiver` trait instead:

impl<T> std::ops::Receiver for MyRc<T> {
    type Target = T;
}

// Everything else identical to previous code example

We propose a blanket implementation of `Receiver` for all `T: Deref`. This is unusual, but this is believed the best choice because:

* We would otherwise be searching along different paths for method candidates, vs. the types to which the actual receiver can be converted as a `self` type in those candidates. This sounds like a recipe for great user confusion.
* Method resolution already searches along the `Deref` chain, even in stable Rust, so we'd have to preseve behavior by searching along _both_ the `Deref` and `Receiver` chains in some way. Extra confusion.
* Existing smart pointer types which implement `Deref` can immediately be used as `self` types.

The net effect of this blanket implementation is that a type has a chain of `Receiver`s to follow, the first few steps of which may also be found by following `Deref` instead:

graph LR
  A --Receiver--> B
  A --Deref--> B
  B --Receiver--> C
  B --Deref--> C
  C --Receiver--> D
  D --Receiver-->E

# Rationale for arbitrary self types

* [Rust for Linux](https://github.com/rust-lang/rfcs/pull/3519#discussion_r1492385549) would like their kernel-side equivalents of `Arc`, `Box` etc. to have the same capabilities as `std`'s user-side equivalents. @Darksonn says:
> The kernel needs a custom Arc for various reasons, but the most important reason is that we need to use the kernel's refcounting logic, instead of the logic used by `alloc::sync::Arc`. This is because the standard library `Arc` will abort the program on overflow, which is entirely unacceptable in the kernel. ... Usually, making `Arc` into a receiver comes up because we have many methods where we want to call `self.clone()` inside a method. ... There are also several other types that we would like to become receivers
* Interop with other languages. In the case of JavaScript, Python and [C++](https://medium.com/@adetaylor/are-we-reference-yet-c-references-in-rust-72c1c6c7015a), we want to represent foreign language references/pointers using a Rust type, and to call methods on such references. (In many cases these foreign language pointers/references can't obey Rust aliasing rules, so Rust references are no good.) Example:

  // Call into C++ to get a C++ reference/pointer...
  let cpp_obj_reference: CppRef<ConcreteCppType> = get_cpp_reference(); 
  // cpp_obj_reference does not obey Rust reference semantics. Other
  // "references" to the same data may exist in the Rust or C++ domain.
  // But it can effectively be used as an opaque token to pass safely
  // through Rust back into C++
  let some_value: u32 = cpp_obj_reference.some_cpp_method();

* Use-cases where smart references have semantics of their own, for instance custom reference counting types or cases where a UI should be re-laid-out after the last reference disappears.
* We currently can't add methods to `Rc`, `Box` etc. because they might shadow methods on contained types. These smart pointers would be more intuitive if they had methods.
* Accepting raw pointers, `NonNull` and so-on as method receivers in order to allow (for example) field projection methods without any dereferencing or `unsafe`. @Manishearth says:
> raw pointer receivers are quite important for the future of safe Rust, because stacked borrows makes it illegal to materialize references in many positions, and there are a lot of operations (like going from a raw pointer to a raw pointer to a field) where you don't need to or want to do that.
>
and @nikomatsakis says:
> it enables `*const self` methods -- this is a big win for unsafe code, in my opinion, for all the same reasons. Right now code that wants to take a raw pointer and doesn't want to guarantee reference validity is pretty stuck.

## Adding new methods to smart pointers

We currently cannot add new methods to `Rc`, `Arc`, `Box`, or `Pin`. That's because they may be used as `self` types, and so any method calls must pass through to the contained type. This results in an awkward API for these types, and it would be great to relax this. (We might, for example, want to add new methods for provenance tracking in future).

We'd also very much like to be able to support `NonNull`, `Weak` and raw pointers as `self` types. Such method calls would look like this:

use std::ptr::NonNull;
use std::rc::{Rc, Weak};

struct Dino;

impl Dino {
    fn roar(self: NonNull<Self>) { }
    fn eat_toilet(self: *mut Self) { } // *const also OK
    fn trash_visitor_center(self: Weak<Self>) { }
}

fn main() {
    let mut dino = Dino;
    let dino_ptr = &dino as *mut Dino;
    dino.eat_toilet();
    let dino_nonnull = NonNull::new(dino_ptr).unwrap();
    dino_nonnull.roar();
    let dino2 = Rc::new(Dino);
    let dino2_weak = Rc::downgrade(&dino2);
    dino2_weak.trash_visitor_center();
}

Supporting these types of calls is seen as highly desirable by many in the Rust lang community, but without special handling this would prevent us adding more methods to `NonNull`, `Weak` or raw pointers, because any such methods would shadow method calls on the inner type.

In this case, imagine we add `NonNull::roar()`. It's ambiguous whether `dino_nonnull.roar()` should call `NonNull::roar()` or `Dino::roar()`.

## How we solve this

We propose a new set of rules for disambiguating method calls. The goal is: if we add a new method to `Rc`, `Arc`, `Box`, `NonNull`, `Weak`, raw pointers or any other type implementing `Receiver`, we'd like to continue calling pre-existing inner type methods.

The proposal is to disambiguate to the inner method (for instance `Dino::roar()`), but to show a warning such that the user takes action to disambiguate the call.

Specifically:
* In method probing, we identify if there are multiple method candidates with the same `self` type, but different distances through the `Receiver` chain.
* If so, we choose the candidate which is furthest along the `Receiver` chain and show a warning.

There's more detail on the implementation of these rules below.

### Why some say this feels counterintuitive

Normally in the Rust world, we probe for methods from the outer to the inner along the `Deref` chain.

In this case, we need to disambiguate by choosing the _inner_ method (`Dino::roar` not the outer `NonNull::roar`), because we can be confident that it was added _first_.

An example using a new custom smart pointer type:

// in crate myrc

struct MyRc<T>(T);

impl<T> std::ops::Receiver for MyRc<T> {
    type Target = T;
}

// in crate jurassic

struct Dino;

impl Dino {
    fn roar(self: MyRc<Self>) { }
}

// somewhere else

fn main() {
    let dino = MyRc(Dino);
    dino.roar();
}

If we *later* add `MyRc::roar`, it's important that `main` continues calling `Dino::roar`.

### Why I (Nadrieril) think this is the correct resolution order regardless

*(section written by Nadrieril)*

Compare:

impl<T> CustomRc<T> {
    fn frob(&self) { ... }
    // sugar for:
    fn frob(self: &CustomRc<T>) { ... }
}

impl MyType {
    fn frob(self: &CustomRc<MyType>) { ... }
}

This looks like specialization to me. In other words, the impl on `MyType` is more specific than the generic impl, so it makes sense to select it instead. If we ever get specialization on traits, this will feel consistent with that.

This is different from how `Deref` works. If we had `fn frob(self: &MyType)`, this is then less specific than the generic `CustomRc<T>` impl. So it makes sense to select the generic method first.

To sum up, I argue that method resolution should be "most specific first", which means the following order of priority:

1. Inherent method `fn frob(self: &CustomRc<MyType>)` on `MyType`;
1. Inherent method `fn frob(self: &Self)` on `CustomRc<T>`;
1. Trait method `fn frob(self: &Self)` if implemented by `CustomRc<T>`;
1. Inherent method `fn frob(self: &MyType)` on `MyType` (via `Deref`);
1. Trait method `fn frob(self: &Self)` if implemented by `MyType` (via `Deref`).

When mixing several layers of `Receiver` and `Deref`, "most specific first" gives:

1. `fn frob(self: &Arc<Box<Self>>)` on `ConcreteType`;
1. `fn frob(self: &Arc<Self>)` on `Box<T>`;
1. `fn frob(self: &Self)` on `Arc<T>`;
1. `fn frob(self: &Box<Self>)` on `ConcreteType` (via `Deref`);
1. `fn frob(self: &Self)` on `Box<T>` (via `Deref`);
1. `fn frob(self: &Self)` on `ConcreteType` (via two layers of `Deref`).

which is already what was implemented in `arbitrary_self_types` [last I checked](https://gist.github.com/Nadrieril/10d909b02e07ae493db2fe98ce48c715).

### What about if the outer type adds a method first?

Imagine we have:

impl<T> MyRc<T> {
    fn eat(self) {}
}

If the author of `Dino` later adds `Dino::eat(self: MyRc<Self>)` then this will shadow `MyRc::eat`. But! The self-type is `MyRc` and they should already be aware of the methods on `MyRc`. So we don't need to guard against this eventuality.

The same applies if we're talking about adding a trait method `Carnivore::eat(self: MyRc<Self>)` where `Dinosaur: Carnivore`. The creator of `Carnivore` should be aware of the methods on `MyRc`, so there doesn't appear to be any way a compatibility break can occur here except in race conditions. (Thought on this point is appreciated.)

### Why this is OK

Whenever we disambiguate to an inner method like this, we show a warning, which can be easily resolved by fully qualifying the function call. **All warning-free Rust code resolves methods from the outside in, in the normal intuitive fashion.** In my opinion this means that this rule, even if some see it as counterintuitive, can cause no significant confusion in the Rust ecosystem.

## How these warnings would be experienced by users

Imagine:

use std::rc::{Rc, Weak};

struct Orbit;

impl Orbit {
    fn retrograde(self: Weak<Self>) { }
}

fn main() {
    let orbit = Rc::new(Orbit);
    let orbit_weak = Rc::downgrade(&dino2);
    orbit_weak.retrograde();
}

then, we add `Weak::retrograde(&self)` to the standard library.

The above code would receive a **warning** something like:

warning[W0666]: ambiguous function call
 --> src/main.rs:13:4
   |
13 |     orbit_weak.retrograde();
   |                ^^^^^^^^^^^^
   |
  = note: you may have intended a call to `Orbit::retrograde` or
    to `Weak::retrograde`
  = note: this method won't be called
     --> src/rc/rc.rs:136:21
    |
136 |    fn retrograde(&self) {
    |       ^^^^^^^^^^^^^^^^^
    |
  = note: because we'll call this method instead
     --> src/space/near_earth.rs:357:68
    |
357 |    fn retrograde(self: Weak<Self>) {
    |       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
  = help: call as a function not a method:
  ~       Orbit::retrograde(orbit_weak)
  = help: call as a function not a method:
  ~       Weak::retrograde(orbit_weak)

The warning can readily be resolved by adopting either of the suggestions.

(This lint is not yet fully implemented; [work in progress here](https://github.com/rust-lang/rust/compare/master...adetaylor:rust:receiver_trait_with_target#diff-090eb0e88a58a0f9ee4138978029c8d73a05e77306555ab98f076bf24a6f35b9R1289) so some of the details of the lint may not be exactly right. Suggestions much appreciated.)

## More examples of when this lint might appear

* If a user implements a method with a raw pointer receiver, say, `impl WindowPane { fn clear(self: *mut Self) {} }` then we in the standard library implement a new `clear` method on `*mut T`, a warning will be generator for callers of `WindowPane::clear`.
* If a user in crate A implements a smart pointer type, say, `CppRef<T>`. User in crate B implements a method `impl CppParty { fn dance(self: CppRef<Self>) {} }` and then the creators of crate A implement `CppRef::dance()`, such a warning will be produced for anyone calling `CppParty::dance`.
* We have `Box<Door>` and `impl Door { fn open(self: Box<Self>) {} }`. We then add `Box::open(self)` to the standard library. Callers of `Door::open` will start to experience a warning.

## Details of the deshadowing rules

The deshadowing rules apply in two circumstances, which aim to have the same effect:

### Case A: when there are multiple candidates at the same step

For each step of the `Deref` chain, and each type of receiver (by-value, by-reference, by-mut-reference, by-const-pointer) the function `consider_candidates` is called to decide between possible method candidates.

Prior to this RFC, if it found multiple possible candidates, it would show *an error*.

We downgrade this error to a warning if we believe that the "outer"
candidates may be newly added and may be shadowing a pre-existing
"inner" candidate.

Specifically the rule is:
* If of all the candidates, exactly one has the greatest "depth" - that is, there's one which is deeper through the `Receiver` chain than all
the others, then
* We emit a warning
* We discard all the other candidates.

Under all other circumstances, we continue to emit an error as we do now.

This rule applies in cases like this:

impl<T> SmartPtr<T> {
    fn m(&self) {}
}

impl Foo {
    fn m(self: &SmartPtr<Self>) {}
}

fn main() {
    let f = SmartPtr(Foo);
    f.m();
}

### Case B: when we've found picks of different kinds

The picking code currently picks:
* The best possible candidate where `self` is received by value; and if none, it picks:
* The best possible candidate where `self` is received by reference; and if none, it picks:
* The best possible candidate where `self` is received by mutable reference; and if none, it picks:
* The best possible candidate where `self` is received by const ptr.

This serial approach doesn't work in cases like this:

impl<T> SmartPtr<T> {
    fn m(self) {}   // note by value
}

impl Foo {
    fn m(self: &SmartPtr<Self>) {}  // note by reference
}

fn main() {
    let f = SmartPtr(Foo);
    f.m();
}

Assume that `SmartPtr::m` was added later; it may shadow `Foo::m` which may have been added earlier.

So, in `pick_all_method`, we no longer do these steps in series. We instead work out the best candidate by value, by reference, and by mutable reference.

We consider `&mut T` to be the weakest, `&` to be intermediate, and `T` to be the strongest kind of method calls. If a stronger method call might shadow a weaker method call, _and_ the weaker method call is further along the `Receiver` chain, then we instead pick that method and show the same warning.

### Possible enhancements to these rules

As an implementation detail: possibly, the current method resolution logic (`pick_all_method`, `pick_by_value_method`, `pick_autorefd_method`, `consider_candidates`) should be flattened to consider a single list of candidates such that these deshadowing rules can be applied in one place not two.

## Other benefits of these rules

If we employ these rules, they allow us to add methods to `Box`, `Rc` etc. safe in the knowledge that we won't shadow methods in the contained types.

Also, all current workarounds for the lack of "arbitrary self types" pose similar shadowing hazards. For instance:

// crate: myrc
pub mod myrc {
    pub struct MyRc<T>(pub T);
    impl<T> MyRc<T> {
        // Consider what happens to downstream if we later add:
        // pub fn eat(&self) {}
    }
}

// crate: dino
pub mod dino {
    use super::myrc::MyRc;

    pub mod prelude {
        pub use super::MyRcExt;
    }

    pub struct Dino;

    pub trait MyRcExt {
        fn eat(&self) {}
    }

    impl MyRcExt for MyRc<Dino> {
        fn eat(&self) {
            println!("dino eat");
        }
    }
}

// crate: downstream
use dino::{prelude::*, Dino};
use myrc::MyRc;

fn main() {
    MyRc(Dino).eat();
}

The addition of these rules would remove these shadowing hazards.

## Other options

* Decide to allow method calls on `NonNull`, `Weak` and raw pointers, by simply deciding never to add more methods to them again.
* Decide to support arbitrary self types without adding support for `NonNull`, `Weak` and raw pointers. Do not add this rule. Disallow, or at least advise against, implementing `Recever` for types with methods.
* Radically change a future edition of Rust to resolve all methods from the inside out!

## Non-goals of this proposal

The current proposal covers only method dispatch with a custom `self` type. It does not cover:

* Dynamic upcasting
* Dynamic dispatch. Some say this is more important: for example @nbdd0121 [says](https://hackmd.io/z4n40072Tqy8MhQPZn0N-g):
> [dynamic dispatch] provides real power to library authors. Currently, smart pointers that allow dynamic dispatch is a privilege that only standard library smart pointers enjoy. It's extremely difficult if not impossible to replicate this feature in 3rd party libraries, and even it's possible, it won't be as ergnomical.

It's certainly important that we don't constrain our ability to broaden dynamic dispatch in the future.

## Decisions

@Darksonn says:

> One idea that has come up again and again in the Rust for Linux discussions is that, if there is some feature where we aren't able to stabilize it because some details are uncertain, then we can still stabilize the part of the feature that Rust for Linux needs.
>
> This is very relevant for this RFC: The kernel does not really need raw pointers and `Weak` to become receivers. If this RFC ends up being blocked just because we can't find a solution to those things, then that would be really unfortunate. There is a clear path forward on the easy part of the `Arc` problem. Let's solve that so we can move on to the hard part.

## Background reading & references

* The [proposed RFC](https://github.com/rust-lang/rfcs/pull/3519).
* The consensus that we _really_ want to support `NonNull`, `Weak` etc. even if we have to do something a bit weird:
* [Summary of views](https://github.com/rust-lang/rfcs/pull/3519#issuecomment-1824448078)
* Results of [a prior T-lang meeting discussing this](https://github.com/rust-lang/rfcs/pull/3519#issuecomment-1853465851)
* The [comment explaining the rules needed to avoid compatibility problems with `NonNull` etc.](https://github.com/rust-lang/rfcs/pull/3519#issuecomment-1858223400).
* [Zulip thread](https://rust-lang.zulipchat.com/#narrow/stream/213817-t-lang/topic/Arbitrary.20self.20types.20v2.20RFC)