Reject extern "{abi}" when the target does not support it

[workingjubilee]

What

Promote unsupported_fn_ptr_calling_conventions from a warning to a hard error, making sure edge-cases will not escape. We now emit hard errors for every case we would return Invalid from AbiMap::canonize_abi during AST to HIR lowering. In particular, these architecture-specific ABIs now only compile on their architectures¹:

• amdgpu: "gpu-kernel"
• arm: "aapcs", "C-cmse-nonsecure-entry", "C-cmse-nonsecure-call"
• avr: "avr-interrupt", "avr-non-blocking-interrupt"
• msp430: "msp430-interrupt"
• nvptx64: "gpu-kernel", "ptx-kernel"
• riscv32 and riscv64: "riscv-interrupt-m", "riscv-interrupt-s"
• x86: "thiscall"
• x86 and x86_64: "x86-interrupt"
• x86_64: "sysv64", "win64"

The panoply of ABIs that are logically x86-specific but actually permitted on all Windows targets remain supported on Windows, as they were before. For non-Windows targets they error if the architecture does not match. This only causes using extern "{abi}" to have a consistent result without regard to its position in the grammar.

Moving the check into AST lowering is itself a breaking change in rare cases, above and beyond the cases rustc currently warns about. See "Why or Why Not" for details.

How

We modify rustc_ast_lowering to prevent unsupported ABIs from leaking through the HIR without being checked for target support. Previously ad-hoc checking on various HIR items required making sure we check every HIR item which could contain an extern "{abi}" string. This is a losing proposition compared to gating the lowering itself.

As a consequence, unsupported ABI strings will now hard-error instead of triggering the FCW unsupported_fn_ptr_calling_conventions. The code is also greatly simplified.

However, per #86232 this does cause errors for rare usages of extern "{abi}" that were theoretically possible to write in Rust source, without previous warning or error. For instance, trait declarations without impls were never checked. These are the exact kinds of leakages that this new approach prevents.

This differs from the following PRs:

• Add (back) unsupported_calling_conventions lint to reject more invalid calling conventions #141435 is orthogonal, as it adds a new lint for ABIs we have not warned on and are not touched by this PR
• Reject unsupported, unstable ABIs during AST lowering #141877 is subsumed by this, in that this simply cuts out bad functionality instead of adding epicycles for stable code

Why or Why Not

We already made the decision to issue the unsupported_fn_ptr_calling_conventions future compatibility warning. It has warned in dependencies since #135767, which reached stable with Rust 1.87. That was released on 2025 May 17, and it is now June. As we already had erred on these ABI strings in most other positions, and warn on stable for function pointer types, this breakage has had reasonable foreshadowing.

Upgrading the warning to an error addresses a real problem. In some cases the Rust compiler can attempt to actually compute the ABI for calling a function. We could accept this case and compute unsupported ABIs according to some other ABI, silently². However, this obviously exposes Rust to errors in codegen. We cannot lower directly to the "obvious", target-incorrect ABI and then trust code generators like LLVM to reliably error on these cases, either.

Refactoring the compiler so we could defer more ABI computations would be possible, but seems weakly motivated. Even if we succeeded, we would at minimum risk:

• exposing the "whack-a-mole" problem but "approaching linking" instead of "leaving AST"
• making it harder to reason about functions we can lower further
• complicating the compiler for no clear benefit

A deprecation cycle for the edge-cases could be implemented first, but it is not very useful for such marginal cases, like this trait declaration without a definition:

pub trait UsedToSneakBy {
    pub extern "gpu-kernel" fn sneaky();
}

Upon any impl, even for provided fn within trait declarations, e.g. pub extern "gpu-kernel" fn sneaky() {}, different HIR types were used which would, in fact, get checked. Likewise with anything with function pointers. Thus we would be discussing deprecation cycles for code that is impotent or forewarned³.

Implementing a deprecation cycle is possible, but it would likely require emitting multiple of a functionally identical warning or error on code that would not have multiple warnings or errors before. It is also not clear to me we would not find another, even more marginal edge-case that slipped through, as "things slip through" is the motivation for checking earlier. For this reason, any effort spent on additional warnings, as opposed to merely letting this PR languish for a bit, should require committing to limits in time and cost to user experience.

r? lang

Fixes #86232
Fixes #132430
Fixes #138738
Fixes #142107

Footnotes

1. Some already will not compile, due to reaching ICEs or LLVM errors. ↩

2. We already do this for all AbiStr we cannot parse, pretending they are ExternAbi::Rust, but we also emit an error to prevent reaching too far into codegen. ↩

3. It actually did appear in two cases in rustc's test suite because we are a collection of Rust edge-cases by the simple fact that we don't care if the code actually runs. These cases were excised in c1db989. ↩

[rustbot]

HIR ty lowering was modified

cc @fmease

This PR changes a file inside tests/crashes. If a crash was fixed, please move into the corresponding ui subdir and add 'Fixes #' to the PR description to autoclose the issue upon merge.

These commits modify tests/rustdoc-json.
rustdoc-json is a public (but unstable) interface.

Please ensure that if you've changed the output:

• It's intentional.
• The FORMAT_VERSION in src/librustdoc-json-types is bumped if necessary.

cc @aDotInTheVoid, @obi1kenobi

[workingjubilee]

@bors2 try

[rust-bors]

⌛ Trying commit c1db989 with merge 8b8eff5…

To cancel the try build, run the command @bors2 try cancel.

[rust-bors]

☀️ Try build successful (CI)
Build commit: 8b8eff5 (8b8eff55bd72abbb57167bc42222a7f91d41cb0d)

[workingjubilee]

@craterbot run mode=build-only name=pr-142134-abi-ast-error cap-lints=warn

[craterbot]

🚧 Experiment pr-142134-abi-ast-error is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

👌 Experiment pr-142134-abi-ast-error created and queued.
🤖 Automatically detected try build 8b8eff5
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more