Distinguish repr(C) ZSTs from others in ABI compatibility rules

[Jules-Bertholet]

(Split out from compiler implementation in #156112)

Some C ABIs pass and return ZSTs by pointer. But () should never be returned by pointer, as it must match void. To account for this, we have to weaken the present guarantee of "any two types with size 0 and alignment 1 are ABI-compatible" to exclude repr(C).

Fixes rust-lang/unsafe-code-guidelines#552; see also #78586, #155299.
Also related to #155984.

@rustbot label T-lang A-ABI needs-fcp

[rustbot]

r? @Mark-Simulacrum

rustbot has assigned @Mark-Simulacrum.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

Why was this reviewer chosen?

The reviewer was selected based on:

• Owners of files modified in this PR: libs
• libs expanded to 11 candidates
• Random selection from 6 candidates

[Jules-Bertholet]

We exempt arrays for 2 reasons:

• A future version of C, or some other language we would like to do FFI with, might allow passing arrays to functions directly by value, in a way that would conflict with these guarantees
• It would be nice to also use "trivial ABI" in the specification of repr(C), and we need this clause for that. See discussion at repr(ordered_fields) rfcs#3845 (comment)

We could also simplify this clause by saying merely:

Suggested change

	/// - Not an array whose element type fails to satisfy these conditions
	/// - Not an array

The downside would be a larger breaking change.

Note that repr(transparent) will need to be adjusted to account for this change (by rejecting non-trivial arrays as "additional" fields).

View changes since the review

[RalfJung]

I'm fine with saying that "all arrays with trivial-ABI element type have trivial ABI".

However, even that less-breaking version breaks ghost again, right? Similar to dtolnay/ghost#41. ghost uses a zero-length array of *const T, which definitely does not have trivial ABI. I don't know why they do that...

EDIT: Ah, ghost is saved by having a repr(Rust) type around the array, under the rules discussed here.

[Mark-Simulacrum]

Probably mostly an opsem question in my eyes, at least to figure out the right shape. I see there's already been some discussion but I think nailing the language is probably better delegated too.

r? opsem

View changes since this review

[Mark-Simulacrum]

Some C ABIs pass and return ZSTs by pointer. But () should never be returned by pointer, as it must match void. To account for this, we have to weaken the present guarantee of "any two types with size 0 and alignment 1 are ABI-compatible" to exclude repr(C).

I think the implication here is that () is repr(C)? Am I reading that right? Where do we make that guarantee? Or is the thinking that the language here would make () and #[repr(C)] struct Foo; not ABI compatible?

One callout is that ZSTs aren't (I think?) standardized -- C and C++ without extensions both require types to be non-ZST if I remember right (e.g., see https://stackoverflow.com/a/2632075). Maybe that has changed since then though?

It seems like at minimum, it would be nice to avoid weakening this guarantee for Rust ABI even if we do so for C ABIs as a result of the weird platforms.

[Jules-Bertholet]

Or is the thinking that the language here would make () and #[repr(C)] struct Foo; not ABI compatible?

Yes, this.

One callout is that ZSTs aren't (I think?) standardized

Correct.

[RalfJung]

The entire notion of a repr(transparent) type being a wrapper around a particular field doesn't really work for 1-ZST. So I think we should say something more like:

A type has trivial ABI if is satisfies all of the following:

• It has size 0.
• It has alignment 1.
• Further restrictions apply based on what kind of type it is:
  • If it is an array, its element type must also have trivial ABI. (This applies even if the array has length 0!)
  • If is is a struct, enum, or union, then it must not use repr(C) and all fields (of all variants) must have trivial ABI.
  • If it is a tuple, all fields must have trivial ABI.
  • No other type ever has trivial ABI.

[Jules-Bertholet]

all fields (of all variants) must have trivial ABI.

That's more restrictive than the current wording, and I don't see a good motivation for it. We are clawing back a stable guarantee here, so I don't think we should break anything beyond what is absolutely necessary.

[RalfJung]

The current wording isn't even well-defined. This is my best guess at turning this into something well-defined, and it seems pretty natural to me. When could we ever have a non-trivial-ABI type anywhere inside a trivial-ABI type?

Regarding it not being well-defined -- I don't know what you mean by this:
"a repr(transparent) wrapper around a type that fails to satisfy these conditions"
Do you mean to check all fields of the repr(transparent) type, and they must all have trivial ABI?
If not, which field do you want to check? For zero-sized repr(transparent), there is no unique field that this type "wraps".

[RalfJung]

Ah, I saw you landed a commit that clarifies that you mean all fields of repr(transparent) types.

So you are suggesting that a non-C non-transparent ADT that happens to be a 1-ZST always has trivial ABI? I see. Hm...

[RalfJung]

I think I like this. If a type has entirely undefined ABI we can just choose it to be trivial and don't have to recurse further. I would word it as follows:

• It has size 0.
• It has alignment 1.
• Further restrictions apply based on what kind of type it is:
  • If it is an array, its element type must also have trivial ABI. (This applies even if the array has length 0!)
  • If it is a struct, enum, or union, then
    • If it is repr(transparent), all fields must have trivial ABI.
    • If it is repr(C), the type does not have trivial ABI.
    • Otherwise, it is repr(Rust) (possibly with additional flags such as packed), and has a trivial ABI no matter its fields.
  • If it is a tuple, it has trivial ABI no matter its fields.
  • No other type ever has trivial ABI.

Note that this is not equivalent to your wording since your wording implicitly allows closures to have trivial ABI if they happen to be 1-ZST. (We could probably get away with that but I don't think we should do so implicitly.) I also find it quite hard to reason through the logic of this in the way that you framed it (too many negations). So I think we should spell this out in more detail, similar to what I suggested.

[RalfJung]

Suggested change

	/// - A `repr(transparent)` type is ABI-compatible with its unique field that does not have trivial ABI
	/// (as defined above). If there is no such field, the type has trivial ABI.
	/// - A `repr(transparent)` type `T` is ABI-compatible with its unique non-trivial field, i.e., the
	/// unique field that doesn't have size 0 and alignment 1 (if there is such a field).
	/// - A `repr(transparent)` type is ABI-compatible with its unique field that does not have trivial ABI
	/// (as defined above), if such a field exists.

View changes since the review

[RalfJung]

Suggested change

	/// - Any two types fulfilling all the following conditions are ABI-compatible;
	/// such types are said to have "trivial ABI":
	/// - Any two types with "trivial ABI" are ABI-compatible.
	/// A type has trivial ABI if is satisfies all of the following:

View changes since the review

[RalfJung]

I think I do agree with this modulo wording nits, though maybe I misread and my suggestion above (to require "all fields must have trivial ABI" rather than the oddly negated repr(transparent) condition) isn't just a wording change but a substantial change. In the following I am referring to my proposed ruleset. EDIT: Updated ruleset.

@rust-lang/opsem @rust-lang/lang what do you think? I think we do have to take back a bit of what we promised here, since we did promise too much. The current docs are plain wrong for extern "C" on Windows where the following types do not have the same ABI when used in return position:

// This is returned by-ptr.
#[repr(C)]
struct T1 {}

// This matches a C function returning `void`.
type T2 = ();

In argument position they do have the same ABI, but the way we achieve this is silly: we pass a pointer for () arguments.

Note that we also have to adjust the logic for repr(transparent) to ensure that all the types it ignores have "trivial ABI" according to these rules. So there's two breaking changes here:

1. Code that relied on the ABI compatibility we are taking away now has UB.
2. Code that relied on repr(transparent) ignoring a 1-ZST that we no longer consider to be "trivial" will stop compiling.

I cratered the 2nd part and found 0 regressions.

I don't know how to measure the fallout from the 1st breakage. We should implement this in Miri, but even that will just give us a very incomplete partial picture.

That said, given that the current docs are wrong, we have to do something. The only actually open question (in my eyes) is how bespoke we want the rules to be with the aim of breaking less code.

• We could have different ABI compatibility rules for the "Rust" ABI and the "C" ABI, and say that all 1-ZST have "trivial Rust ABI". This would theoretically reduce the amount of case 1 breakage (but we don't know anything about that breakage so it's unclear if there even exists code that would be helped by this). Currently our ABI compatibility notion is independent of the ABI used by the function, so this adds non-trivial extra complexity on the docs side. I don't think we should do this.
• The current rules say that all repr(Rust) 1-ZST have trivial ABI, without recursing further. This means that adding repr(transparent) to a type can make it lose its "trivial ABI" property! The way to intuitively understand this is that by adding repr(transparent) you are forcing the ABI to be compatible with at least one field, and if that field has an ABI that's determined by the C platform then we can't say that it has trivial ABI. The reason we do this is two-fold: this means more types have "trivial ABI", so we reduce the amount of type 1 breakage. And it means we can keep a pattern used by the ghost crate working (specifically, this allows us to consider a ghost-declared phantom type as being "trivial" for repr(transparent)).