Warn user when the whole function is optimized to a `ud2`

[HKalbasi]

Code

pub fn bad_func1(num: i64) -> i32 {
    let v2 = unsafe { std::mem::transmute::<i64, &i32>(num) };
    *v2
}

Current output

No warning, but the assembly only contains a single `ud2` instruction, which will terminate the program with `SIGILL` if executed.

Desired output

Warn user that this function has undefined behavior and will not work as intended.

Rationale and extra context

This warning should be emitted at very last stages of compiling, and it might be very hard to implement in the current compiler architecture. But it can save time of the developer, specially when there is no test coverage for that function.

It should be a warning and not a hard error, since optimizations can work differently from version to version.

Other cases

Rust Version

rustc 1.86.0 (05f9846f8 2025-03-31)
binary: rustc
commit-hash: 05f9846f893b09a1be1fc8560e33fc3c815cfecb
commit-date: 2025-03-31
host: x86_64-unknown-linux-gnu
release: 1.86.0
LLVM version: 19.1.7

Anything else?

No response

[hanna-kruppe]

I would have expected that at least clippy warns about this kind of transmute, but at least the version currently on the playground doesn’t.

[HKalbasi]

There is a clippy issue for this transmute (rust-lang/rust-clippy#13140) and I agree that a lint is needed for that specific case, but this issue is more general, and in many cases compiler or clippy can't detect the UB, but the optimizer is able to. For example:

pub fn out_of_bound_access() -> i32 {
    let arr = [1, 2, 3];
    unsafe { *arr.get_unchecked(5) }
}

pub fn see_through() {
    fn f() -> i32 {
        5
    }

    if f() == 5 {
        unsafe {
            std::hint::unreachable_unchecked();
        }
    }
}

pub fn complex_control_flow(num: i64) {
    unsafe {
        if num == 5 {
            std::hint::unreachable_unchecked();
        }
        if num != 5 {
            std::hint::unreachable_unchecked();
        }
    }
}

All of these functions optimize to a single ud2 instruction, and I can't imagine a compiler or clippy lint to detect them in reasonable time. It will be even harder to do the detection on more real world scenarios, but an optimizer might be able to do it.

[hanna-kruppe]

Note that "the function compiled down to ud2" is not the most common or most dangerous outcome of UB in incorrectly written unsafe code. Most unsound usage of unsafe constructs results in something less debuggable than the program reliably crashing with SIGILL. Even if a particular function is optimized by LLVM into an unreachable and nothing else, this will likely be inlined into some or all callers, which can also easily result in the inlined function being eliminated entirely. So it's not clear how commonly this lint would catch a real problem in real code that couldn't also be detected just as well by other means. Do you have a less contrived example where the lint you're proposing would have helped?

From the other direction, there are legitimate and 100% safe Rust functions that are basically a single unreachable from the start, which isn't a problem because they won't ever be called. For example, consider all the trait impls for core::convert::Infallible and other empty enums that define the methods as match *self {}. Usually these won't be codegen'd, let alone survive until the end of the optimization pipeline, but exceptions exist, and these would be scary-sounding false positives.

[saethlin]

All of these functions optimize to a single ud2 instruction, and I can't imagine a compiler or clippy lint to detect them in reasonable time. It will be even harder to do the detection on more real world scenarios, but an optimizer might be able to do it.

The optimizer can do it, we can't, and there is no communication pathway back to us from LLVM optimizations.

Also if you think a few false positives out of impls for ! and Infallible are bad, wait until you hear about my favorite feature, panic_immediate_abort.

[saethlin]

My position here is that this should be a lint, and if people want post-optimization linting, you need to work out what the communication pathway is to get either the optimized LLVM IR in a traversable way or some analysis of it back into the Rust part of the compiler.

[hanna-kruppe]

Also if you think a few false positives out of impls for ! and Infallible are bad, wait until you hear about my favorite feature, panic_immediate_abort.

I assume the lint would not be literally about ud2 in the machine code but about LLVM IR functions containing only unreachable. panic_immediate_abort produces calls to @llvm.trap() instead, which would indeed be a total false positive to lint about.

[HKalbasi]

Note that "the function compiled down to ud2" is not the most common or most dangerous outcome of UB in incorrectly written unsafe code.

It is not the most dangerous, but it still saves a gdb invocation and a crash if tests are not covering that function. It is also a nice lint for small example codes and demonstration proposes. Also, it can help newcomers who don't know what to do about and what exactly is UB.

Do you have a less contrived example where the lint you're proposing would have helped?

My trigger for opening this issue was a SIGILL from an int-to-ptr transmute in Zngur. I guess a fair percentage of SIGILLs in Rust to be detected by this lint.

For example, consider all the trait impls for core::convert::Infallible and other empty enums that define the methods as match *self {}

I think we can disable the lint if the function has some uninhabited type in its argument. But I agree that if there is no way to have this lint without many false positives, it doesn't worth it.