Design Meeting 2020-01-06

This week's topic: Weak refcounted pointers, dangling, alignment, and long-term implications for custom DSTs.

Lang: nikomatsakis, josh, pnkfelix, scottmcm, cramertj
Guests: Christopher Durham (presenter), Mark, Amanieu

Background

• Proposal was to make into_raw and from_raw available for unsized types https://github.com/rust-lang/rust/pull/80407
• This would allow Weak::into_raw and Weak::from_raw to produce dangling Weak<T> pointers due to unsizing coercions:

let ptr: Weak<[u8]> = Weak::<[u8; 32]>::new();
Weak::from_raw(Weak::into_raw(ptr));

• Issue:
  • it has to get the alignment of the unsized type without having a valid value of that type
  • it does have valid metadata
• Valid metadata suffices for existing types, such as [u8] and dyn Trait, but does it suffice for custom DST?
• What we would be committed to:
  • we can get alignment from the raw pointer to a previously (but not currently) valid unsized type
• Options
  • require that it is possible to get alignment without a valid data pointer (just from type + metadata, or just type for a thin DST)
  • change impl of Weak::from_raw so that it does not care about the alignment
  • introduce Weak::from_raw_unsized that takes the alignment as an extra parameter
  • don't stabilize Weak::from_raw with unsized
    • there exist some use cases
• Note that inherent to the nature of Weak is that the data it references may not be valid (may have been dropped)
  • you need to get the layout (size + alignment) in order to deallocate the underlying memory
  • the custom allocator work thus far has presumed that we provide the Layout
  • in principle we could modify the design of Weak to store the layout, but that would mean extra fields in every Rc, which seems like a non-starter
• Custom DST implications
  • Existing RFC
  • Committed to: supporting deallocation of dropped unsized values contained in a Weak in some form
  • If we stabilize unsized Weak::from_raw: need to be able to get alignment without a valid data pointer
    • this seems less problematic
• But requiring ability to get size from dropped memory is more surprising
  • cstr is not the most interesting case because its drop is a no-op
• Interaction with proposed DynSized trait
  • probably not relevant, since custom DST would still be "dynamically sized", not 'no size at all, even at runtime'
• Bottom line: when adding custom DST, the design must accommodate size_of_val and align_of_val with a dropped value
• Question:
  • what would it take to permit align_of_val to require a valid-but-dropped value?
  • would require Weak::from_raw to check for the sentinel value, since that may not be valid-but-dropped
    • that could be as little as if ptr == 0xFFFF_FFFF_FFFF_FFFF…

[ ref count data] <-- from_raw must recover pointer to this
[ padding ]
[ payload ] <-- into_raw gives pointer to this

• Consensus from meeting:
  • We approve the option to make align_of_val_raw require a once-valid-but-dropped value, in order to better support thin objects
    • we believe the sentinel design means that align_of_val_raw is only ever invoked on once-valid-but-dropped values
  • We do not want align_of_val_raw to be forced to work for metadata + thin pointer
  • Implement from_raw to check for sentinel and take some special action if it is observed
    • potential cost: for unsized types (only), there is an extra branch (but if custom dst doesn't require flexibility on alignment, we can change this later)
  • It is not really lang team's call, but we are -1 on adding more fields to Rc/Arc
  • For custom dst, the design will have to accommodate getting the size and alignment from "once-valid-but-dropped" values (values that were once valid but have been dropped); this is a non-issue for known use cases like c-string and thin-objects (which store a vtable)
    • (but could be relevant for dynamically allocated vtables)