Sporadic ICE when compiling in gocryptfs mount

[texel-sensei]

Code

Code is not relevant to the bug, tested example:

fn main() {
    42
}

The repository is located in a gocryptfs mount, running on Arch Linux. The crash happens sporadically, sometimes compilation works as expected, sometimes the code aborts with SIGBUS and sometimes it panics with the listed error message.

Other tools have no issues running in such a mount.

Output of uname -srv: Linux 5.18.12-zen1-1-zen #1 ZEN SMP PREEMPT_DYNAMIC Fri, 15 Jul 2022 15:33:04 +0000

Meta

rustc --version --verbose:

<version>

Error output

thread 'rustc' panicked at 'assertion failed: `(left == right)`
  left: `82`,
 right: `1002111927320821928687967599834759150`', compiler/rustc_query_impl/src/on_disk_cache.rs:540:5

Backtrace

stack backtrace:
   0:     0x7fdf4329dfed - std::backtrace_rs::backtrace::libunwind::trace::hb729d9642bb971eb
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
   1:     0x7fdf4329dfed - std::backtrace_rs::backtrace::trace_unsynchronized::h373bb774579df5c7
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7fdf4329dfed - std::sys_common::backtrace::_print_fmt::hfbd4e92d240c89bb
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/sys_common/backtrace.rs:66:5
   3:     0x7fdf4329dfed - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h8f618991fbf64972
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/sys_common/backtrace.rs:45:22
   4:     0x7fdf432f9c7c - core::fmt::write::hc69b5b640d88cce8
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/fmt/mod.rs:1196:17
   5:     0x7fdf4328f6c1 - std::io::Write::write_fmt::h3403cef06a24a303
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/io/mod.rs:1654:15
   6:     0x7fdf432a0d05 - std::sys_common::backtrace::_print::h368f27cdedea0e52
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/sys_common/backtrace.rs:48:5
   7:     0x7fdf432a0d05 - std::sys_common::backtrace::print::ha105c9cf5a64cd17
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/sys_common/backtrace.rs:35:9
   8:     0x7fdf432a0d05 - std::panicking::default_hook::{{closure}}::h48ed2c3707d5e20e
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:295:22
   9:     0x7fdf432a0979 - std::panicking::default_hook::h8744fc5cea5e3110
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:314:9
  10:     0x7fdf43ae9961 - rustc_driver[cc15f272104f8c81]::DEFAULT_HOOK::{closure#0}::{closure#0}
  11:     0x7fdf432a14d6 - std::panicking::rust_panic_with_hook::hc82286af2030e925
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:702:17
  12:     0x7fdf432a12d7 - std::panicking::begin_panic_handler::{{closure}}::h1c15057c2f09081f
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:588:13
  13:     0x7fdf4329e4a4 - std::sys_common::backtrace::__rust_end_short_backtrace::h65de906a5330f8da
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/sys_common/backtrace.rs:138:18
  14:     0x7fdf432a1009 - rust_begin_unwind
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/panicking.rs:584:5
  15:     0x7fdf43266283 - core::panicking::panic_fmt::h741cfbfc95bc6112
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/core/src/panicking.rs:142:14
  16:     0x7fdf432f6a28 - core::panicking::assert_failed_inner::h36469c68b6fc10f1
  17:     0x7fdf443beb0c - core[6724bec812d80be0]::panicking::assert_failed::<u128, u128>
  18:     0x7fdf443b87de - <rustc_query_impl[37d320545ae1f097]::on_disk_cache::OnDiskCache as rustc_middle[2b9dbf3985a9058b]::ty::context::OnDiskCache>::new
  19:     0x7fdf45ba35c3 - rustc_incremental[36fe853e067954a8]::persist::load::load_query_result_cache::<rustc_query_impl[37d320545ae1f097]::on_disk_cache::OnDiskCache>
  20:     0x7fdf45b84102 - rustc_interface[a10bc87377f6ea4]::passes::create_global_ctxt
  21:     0x7fdf45ba7ead - <rustc_interface[a10bc87377f6ea4]::queries::Queries>::global_ctxt
  22:     0x7fdf45b4ff82 - <rustc_interface[a10bc87377f6ea4]::interface::Compiler>::enter::<rustc_driver[cc15f272104f8c81]::run_compiler::{closure#1}::{closure#2}, core[6724bec812d80be0]::result::Result<core[6724bec812d80be0]::option::Option<rustc_interface[a10bc87377f6ea4]::queries::Linker>, rustc_errors[7cc8b2502942d635]::ErrorGuaranteed>>
  23:     0x7fdf45b786ef - rustc_span[aeddd698d19619f]::with_source_map::<core[6724bec812d80be0]::result::Result<(), rustc_errors[7cc8b2502942d635]::ErrorGuaranteed>, rustc_interface[a10bc87377f6ea4]::interface::create_compiler_and_run<core[6724bec812d80be0]::result::Result<(), rustc_errors[7cc8b2502942d635]::ErrorGuaranteed>, rustc_driver[cc15f272104f8c81]::run_compiler::{closure#1}>::{closure#1}>
  24:     0x7fdf45b64024 - rustc_interface[a10bc87377f6ea4]::interface::create_compiler_and_run::<core[6724bec812d80be0]::result::Result<(), rustc_errors[7cc8b2502942d635]::ErrorGuaranteed>, rustc_driver[cc15f272104f8c81]::run_compiler::{closure#1}>
  25:     0x7fdf45b4e4f2 - <scoped_tls[7f4edc8f5e37700f]::ScopedKey<rustc_span[aeddd698d19619f]::SessionGlobals>>::set::<rustc_interface[a10bc87377f6ea4]::interface::run_compiler<core[6724bec812d80be0]::result::Result<(), rustc_errors[7cc8b2502942d635]::ErrorGuaranteed>, rustc_driver[cc15f272104f8c81]::run_compiler::{closure#1}>::{closure#0}, core[6724bec812d80be0]::result::Result<(), rustc_errors[7cc8b2502942d635]::ErrorGuaranteed>>
  26:     0x7fdf45b6561f - std[62a6aef941a414a5]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[a10bc87377f6ea4]::util::run_in_thread_pool_with_globals<rustc_interface[a10bc87377f6ea4]::interface::run_compiler<core[6724bec812d80be0]::result::Result<(), rustc_errors[7cc8b2502942d635]::ErrorGuaranteed>, rustc_driver[cc15f272104f8c81]::run_compiler::{closure#1}>::{closure#0}, core[6724bec812d80be0]::result::Result<(), rustc_errors[7cc8b2502942d635]::ErrorGuaranteed>>::{closure#0}, core[6724bec812d80be0]::result::Result<(), rustc_errors[7cc8b2502942d635]::ErrorGuaranteed>>
  27:     0x7fdf45b65759 - <<std[62a6aef941a414a5]::thread::Builder>::spawn_unchecked_<rustc_interface[a10bc87377f6ea4]::util::run_in_thread_pool_with_globals<rustc_interface[a10bc87377f6ea4]::interface::run_compiler<core[6724bec812d80be0]::result::Result<(), rustc_errors[7cc8b2502942d635]::ErrorGuaranteed>, rustc_driver[cc15f272104f8c81]::run_compiler::{closure#1}>::{closure#0}, core[6724bec812d80be0]::result::Result<(), rustc_errors[7cc8b2502942d635]::ErrorGuaranteed>>::{closure#0}, core[6724bec812d80be0]::result::Result<(), rustc_errors[7cc8b2502942d635]::ErrorGuaranteed>>::{closure#1} as core[6724bec812d80be0]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  28:     0x7fdf432ab3f3 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hcbca3baf872b7fe4
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/alloc/src/boxed.rs:1872:9
  29:     0x7fdf432ab3f3 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h18790338ce1743e2
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/alloc/src/boxed.rs:1872:9
  30:     0x7fdf432ab3f3 - std::sys::unix::thread::Thread::new::thread_start::hb1067183bad48893
                               at /rustc/a8314ef7d0ec7b75c336af2c9857bfaf43002bfc/library/std/src/sys/unix/thread.rs:108:17
  31:     0x7fdf42e8c54d - <unknown>
  32:     0x7fdf42f11874 - clone
  33:                0x0 - <unknown>

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.62.0 (a8314ef7d 2022-06-27) running on x86_64-unknown-linux-gnu

note: compiler flags: --crate-type bin -C embed-bitcode=no -C debuginfo=2 -C linker=clang -C incremental -C link-arg=-fuse-ld=/usr/bin/mold

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack

[the8472]

The cache uses mmap'ed IO, perhaps other tools aren't using mmap or aren't verifying file contents? SIGBUS points in the same direction.

[Enselic]

Triage: Specific steps on how to reproduce will be needed to make progress here. Preferably a script to be run in a given docker container for maximum reproducibility.

[saethlin]

As far as I can tell, the ICE reported in this issue is caused by another process damaging rustc's build artifacts or by a buggy filesystem. That other process could be anything from an IDE running the compiler at the same time as a manual build to a find-and-replace that accidentally edits files in the target directory.

#124686 contains a mitigation for this situation; the compiler now has a crude way to detect and ignore damaged incremental compilation files. You will still get a warning about the damaged file, but the build should proceed normally. That change shipped in Rust 1.80 which released today, so I am closing this issue and a few others which as far as I can tell all have the same root cause. You shouldn't see this ICE anymore, but please file a new issue if you see something like it on 1.80 or later.