I explain
This is a living document, which I am refining as my understanding improves.
Inspired by Gankra's post and elaborating on the tracking issue, we would like
rustcinttoptr for some niche use cases, like pointer compression or "pointer in u32"We believe that pointer-to-int transmutes (not casts but transmutes) are incredibly rare. I agree that they can't reasonably be supported. Consider instead migrating to MaybeUninit<integer type>.
I think the following approach gets us everything or 99% of what we want.
We keep usize as ptr permissive as is, and we discourage its use. We don't change its meaning compared to what it meant before — it is definitely able to produce valid pointers in the right circumstances. For an idea of what it could mean formally, see Formal memory models below.
We deprecate the usize as ptr syntax, to be replaced with something like std::ptr::from_exposed that can be documented better.
We keep ptr as usize permissive as is, but we discourage their use.
We deprecate the ptr as usize syntax, to be replaced with something like ptr.expose_addr() that can be documented better.
We make any pointer-to-int transmute undefined behavior.
MVP for CHERI is to only support strict-provenance operations.
For CHERI, allow the user to opt in to a polyfill implementation of std::ptr::from_exposed and ptr.expose_addr(). Otherwise, those functions do not exist. The polyfills are only functional for pointers into allocations by Rust.
Polyfill proposal #1 is the more secure one — it lets the hardware keep every allocation secure that hasn't been explicitly exposed.
Polyfill proposal #2 has a lower overhead, but it has no real checks on std::ptr::from_exposed. Compared to traditional machines, the machine still traps on pointer reads from non-pointer memory, which is a major safety improvement on x64.
For CHERI, we use the *.CAP instructions, not *.DDC.
There is ptr.addr() which is fast on CHERI – but ptr.addr() as ptr is principally UB.
With the new #[feature(strict_provenance)] functions in std::ptr, we can convert ptr.addr()-based addresses back into real pointers, even on CHERI.
We keep std::ptr::invalid as in nightly. This corresponds to "strict" inttoptr. It is UB to dereference these.
We introduce a performance lint against usize as ptr and ptr as usize and their synonyms — because compared to their strict analogs, they have a performance cost, and (when compiling to CHERI) remove some safety
All other ways to create a pointer (in particular various transmutes) are UB.
For our memory model, we keep aiming for something like Stacked Borrows — which allows for permissive and strict provenance in the same program, but is difficult to understand.
We aim to define a simpler submodel of our real memory model, which (1) only permits strict provenance ptr2int2ptr, and (2) is efficiently machine-checkable by Miri, and (3) can more easily be learned and understood by developers.
Then language-wise,
We don't need no compiler flag, no opt-in, no opt-out
We break pointer-to-int transmute
We break pretty much nothing else (presumably, to be investigated)
This doesn't create new questions around FFI
We can have much better stdlib documentation on all involved operations
We plan to rewrite crates so usize as ptr and ptr as usize (and their legacy as spellings) are no longer used. This is only a performance improvement — the original still works.
CHERI works well for strict-provenance programs
CHERI works for permissive-provenance programs. There is a performance/memory cost compared to strict-provenance-only programs, but this is mostly isolated to call sites of ptr.expose_addr() and std::ptr::from_exposed.
CHERI is fast after the ecosystem transitions over
Compressed pointers keep working as they do now — and pretty fast on x64.
Our relation with LLVM stays principally the same.
But we have a "solid ground" for working on rustc — we know how it should behave.
Transmuting / memory semantics needs to be refined further.
Then developers who want to write Unsafe Rust
Note: here strict provenance is a tool to write good Unsafe Rust. It is not necessary that every crate becomes strict-only-provenance. Once you have good Unsafe Rust, you can combine it with other crates and you don't have to worry how they have been written.
(Although strict provenance may still be slightly faster. And on CHERI, also safer.)
There are three memory models relevant for discussion. We list them in order of least UB to most UB.
This is pretty good already. It is a reasonable formal model, and we can check it with Miri.
In this model, there is no real distinction between ptr and int. (You can still only cast from references to ptr, not from references to int.)
But the compiler is limited from doing some optimizations. Take this example.
let mut storage : i8 = 0;
let x_ptr: *mut i8 = &mut storage as *mut i8;
let y: &mut i8 = &mut storage;
{
//! ------------------------------------
//!
//! Pretend that the optimizer can only look
//! inside this block. It is aware of only two
//! variables:
//!
//! - x_ptr : *mut i8
//! - y : &mut i8
let y_ptr = y as *mut i8
*x_ptr = 12; // (1)
*y_ptr = 34; // (2)
}
storage
Under plain SB, this code is valid and does not contain UB. But the compiler is not allowed to swap the assignments (1) and (2). This can realistically prevent other optimizations from kicking in.
In this model and below, ptr is actually something different from usize. Raw pointers have provenance, integers don't.
This lets the optimizer be better: now it is allowed to swap (1) and (2).
The example program becomes UB at point (1). You are not allowed to use a raw pointer when a newer &mut is active for the same location. (Here, that's y.)
This is a viable candidate for Rust's official memory model.
The advantage of this model is that ptr2int2ptr is still allowed as it is today. (Only through cast, not through transmute.) Developers can pretty much treat ptr and usize as interchangable.
The drawbacks of this model are:
This model would be a "useful simplification" of Rust.
In this model, (1) is still UB. Additionally you're not able to cast integers to pointers any more.
The advantages are
-Zmiri-strict-provenance.There is no UB in this example under permissive provenance.
let mut storage : i8 = 0;
let x_int: usize = &mut storage as *mut i8 as usize;
let y: &mut i8 = &mut storage;
{
//! ------------------------------------
//!
//! - x_int : usize
//! - y : &mut i8
let y_ptr = y as *mut i8;
let _ = y_ptr as usize;
// The raw pointer receives provenance from the
// conversion "y_ptr as usize" above! 🤯
*(x_int as *mut i8) = 12;
*y_ptr = 34;
}
storage
This polyfill lets the hardware keep every allocation secure that hasn't been explicitly exposed. You can use std::ptr::from_exposed, but only if the address points into an allocation that is still live that has been exposed.
The runtime puts 1 byte of spacing between every allocation. This is needed to map pointers unambiguously to allocations.
The runtime maintains capabilities to all allocatable address ranges, hidden from the program. (It probably needs this anyway for malloc.)
There is a global collection of live "exposed" allocations, each associated to a capability that's valid for the whole allocation.
ptr.expose_addr() looks in the state of the allocator to find the full address range of the allocation. Then it uses (2) to create a r/w capability for the whole allocation, and inserts into collection (3).
std::ptr::from_exposed looks in the collection (3) for the r/w capability for the whole allocation. If the collection does not contain such an allocation, a panic is raised.
free looks up the allocation in the collection (3), and removes it if it was there.
the ptr2int2ptr integer type ("usize") might be pointer-sized. Then it could contain the base+top+permissions; step (5) would apply them again.
the ptr2int2ptr integer type ("usize") might be address-sized. But then the program can observe that the base+top+permissions changed.
There are open questions around the fact that you can observe the base and top of a pointer. So the program can observe that ptr2int2ptr(p) != p.
With this polyfill, int2ptr is not checked at runtime, and the overhead of ptr2int2ptr is lower. However there is still a big advantage over non-CHERI machines: the machine prevents bad transmutes at runtime. These transmutes are UB anyway, but instead of leading to security vulnerabilities, a panic is raised. The program may even recover at runtime from bad transmutes.
The runtime puts 1 byte of spacing between every allocation. This is needed to map pointers unambiguously to allocations.
The runtime maintains capabilities to all allocatable address ranges, hidden from the program. (It probably needs this anyway for malloc.)
ptr.expose_addr() extracts the address portion of the capability.
std::ptr::from_exposed uses (2) to create a the r/w capability for the whole allocation.
the ptr2int2ptr integer type ("usize") might be pointer-sized. Then it could contain the base+top+permissions; step (4) would apply them again.
the ptr2int2ptr integer type ("usize") might be address-sized. But then the program can observe that the base+top+permissions changed.
There are open questions around the fact that you can observe the base and top of a pointer. So the program can observe that ptr2int2ptr(p) != p.
In my understanding, strict-provenance-only doesn't gain us anything with regards to optimizations.
Tackling the issue of integer sizes is desirable, but out of scope of this position document.
-
Any changes
Be notified of any changes
-
Mention me
Be notified of mention me
-
Unsubscribe
Subscribe