Lift the same-signature restriction for extern "tail"

[folkertdev]

tracking issue: #157427

The extern "tail" calling convention uses callee cleanup (i.e. the callee restores the stack, not the caller). Hence, the same-signature restriction that is normally required to codegen tail calls does not apply.

We need the ABI to deviate from extern "Rust" to make sure indirect arguments are passed by stack offset, not via a pointer into the caller's stack frame (the value would potentially be overwritten). For standard tail calls we work around this problem by storing the value in the caller's caller, but for extern "tail" that doesn't work reliably because the signatures can be different.

I'm not sure about unsized arguments. That feature seems really broken, so I'm not sure how much work we should put into trying to do something reasonable there. Fundamentally I don't think we can support unsized arguments in extern "tail" calls.

Also we can't really promote using this yet due to tailcc being a bit broken on LLVM 22 on x86_64.

r? WaffleLapkin
cc @bjorn3

[rustbot]

WaffleLapkin is not on the review rotation at the moment.
They may take a while to respond.

[bjorn3]

I don't think it is possible supporting unsized arguments with tail calls except for the narrow case where every tail call passes the unsized argument it received from its own caller, in which case Indirect { on_stack: false } would work even with extern "tail". I don't think it is worth supporting this narrow of a rule, so I think disallowing unsized arguments on extern "tail" entirely is the right call.

[bjorn3]

Maybe also disallow it for extern "tail"?

View changes since the review

[folkertdev]

This disallows any tail call with unsized arguments (currently that ICEs for e.g. extern "Rust" https://godbolt.org/z/xe888qThd).

Interestingly the test passes locally, and it already has LLVM 22 as the minimum version, so I'm not sure what the problem is here.

[bjorn3]

This disallows it on the tail call itself, but my suggestion is to also disallow it for extern "tail" even if no actual tail calls are involved so we don't need to compute any ABI adjustments for at all for extern "tail" with unsized arguments.

[folkertdev]

@bjorn3 this line here should disallow unsized arguments in extern "tail" signatures. I forgot to push a test file for this, I'll do so shortly.

View changes since the review

[rust-log-analyzer]

The job x86_64-gnu-pre-stabilization failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

test [ui] tests/ui/zero-sized/zero-sized-btreemap-insert.rs ... ok

failures:

---- [ui] tests/ui/explicit-tail-calls/no-unsized-arguments.rs#x86_64 stdout ----
Saved the actual stderr to `/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/explicit-tail-calls/no-unsized-arguments.x86_64/no-unsized-arguments.x86_64.stderr`
diff of stderr:

1 error[E0277]: the size for values of type `[u8]` cannot be known at compilation time
-   --> $DIR/no-unsized-arguments.rs:32:42
+   --> $DIR/no-unsized-arguments.rs:30:42
3    |
4 LL |     extern "tail" fn unsized_argument(x: [u8]) -> u8 {
5    |                                          ^^^^ doesn't have a size known at compile-time

11    |                                          +
12 
13 error: unsized arguments cannot be used in a tail call
---
To only update this specific test, also pass `--test-args explicit-tail-calls/no-unsized-arguments.rs`

error in revision `x86_64`: 1 errors occurred comparing output.
status: exit status: 1
command: env -u RUSTC_LOG_COLOR RUSTC_ICE="0" RUST_BACKTRACE="short" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1/bin/rustc" "/checkout/tests/ui/explicit-tail-calls/no-unsized-arguments.rs" "-Zthreads=1" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Z" "ignore-directory-in-diagnostics-source-blocks=/cargo" "-Z" "ignore-directory-in-diagnostics-source-blocks=/checkout/vendor" "--sysroot" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage1" "--target=x86_64-unknown-linux-gnu" "--cfg" "x86_64" "--check-cfg" "cfg(test,FALSE,x86,x86_64,aarch64)" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zwrite-long-types-to-disk=no" "-Cstrip=debuginfo" "--emit" "metadata" "-C" "prefer-dynamic" "--out-dir" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/ui/explicit-tail-calls/no-unsized-arguments.x86_64" "-A" "unused" "-W" "unused_attributes" "-A" "internal_features" "-A" "incomplete_features" "-A" "unused_parens" "-A" "unused_braces" "-Crpath" "-Cdebuginfo=0" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers"
stdout: none
--- stderr -------------------------------
error[E0277]: the size for values of type `[u8]` cannot be known at compilation time
##[error]  --> /checkout/tests/ui/explicit-tail-calls/no-unsized-arguments.rs:30:42
   |
LL |     extern "tail" fn unsized_argument(x: [u8]) -> u8 {
   |                                          ^^^^ doesn't have a size known at compile-time
   |
   = help: the trait `Sized` is not implemented for `[u8]`
help: function arguments must have a statically known size, borrowed slices always have a known size
   |
---

For more information about this error, try `rustc --explain E0277`.
------------------------------------------

---- [ui] tests/ui/explicit-tail-calls/no-unsized-arguments.rs#x86_64 stdout end ----

failures:
    [ui] tests/ui/explicit-tail-calls/no-unsized-arguments.rs#x86_64

test result: FAILED. 21204 passed; 1 failed; 232 ignored; 0 measured; 0 filtered out; finished in 754.38s

For more information how to resolve CI failures of this job, visit this link.