Assertion in on-disk cache code in the compiler fails

[gaultier]

Code

Since this is a caching issue (apparently, since the old code and the new code both build fine on their own) it's hard for me to give code to reproduce it. Basically I had code in a library like this:

// Old code

pub fn foo(u64 x){}

which compiled fine with cargo b --target=x86_64-unknown-linux-gnu.

Then I changed the code to:

// New code

pub fn foo_u64(u64 x){}

pub fn foo_16(u16 x){}

And then re-running cargo b --target=x86_64-unknown-linux-gnu it fails.

Meta

rustc --version --verbose:

rustc 1.76.0 (07dca489a 2024-02-04)

Error output

$ cargo b --target=x86_64-unknown-linux-gnu
   Compiling fmw_rust v0.1.0 (/home/pg/company-code/ [...] /src/src/fmw_rust)
thread 'rustc' panicked at compiler/rustc_middle/src/query/on_disk_cache.rs:525:5:
assertion `left == right` failed
  left: 0
 right: 1002111927320821928687967599834759150
stack backtrace:
   0:     0x7b3863f8b6f6 - std::backtrace_rs::backtrace::libunwind::trace::hbee8a7973eeb6c93
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
   1:     0x7b3863f8b6f6 - std::backtrace_rs::backtrace::trace_unsynchronized::hc8ac75eea3aa6899
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0x7b3863f8b6f6 - std::sys_common::backtrace::_print_fmt::hc7f3e3b5298b1083
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:68:5
   3:     0x7b3863f8b6f6 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hbb235daedd7c6190
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x7b3863fddf40 - core::fmt::rt::Argument::fmt::h76c38a80d925a410
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/rt.rs:142:9
   5:     0x7b3863fddf40 - core::fmt::write::h3ed6aeaa977c8e45
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/fmt/mod.rs:1120:17
   6:     0x7b3863f7f53f - std::io::Write::write_fmt::h78b18af5775fedb5
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/io/mod.rs:1810:15
   7:     0x7b3863f8b4d4 - std::sys_common::backtrace::_print::h5d645a07e0fcfdbb
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x7b3863f8b4d4 - std::sys_common::backtrace::print::h85035a511aafe7a8
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x7b3863f8e267 - std::panicking::default_hook::{{closure}}::hcce8cea212785a25
  10:     0x7b3863f8dfc9 - std::panicking::default_hook::hf5fcb0f213fe709a
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:292:9
  11:     0x7b3866c1461c - std[79729d9c385e1623]::panicking::update_hook::<alloc[6df67106ebca92c0]::boxed::Box<rustc_driver_impl[66ed8fdbde15dc6c]::install_ice_hook::{closure#0}>>::{closure#0}
  12:     0x7b3863f8e9b6 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hbc5ccf4eb663e1e5
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2029:9
  13:     0x7b3863f8e9b6 - std::panicking::rust_panic_with_hook::h095fccf1dc9379ee
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:783:13
  14:     0x7b3863f8e702 - std::panicking::begin_panic_handler::{{closure}}::h032ba12139b353db
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
  15:     0x7b3863f8bbf6 - std::sys_common::backtrace::__rust_end_short_backtrace::h9259bc2ff8fd0f76
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
  16:     0x7b3863f8e460 - rust_begin_unwind
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
  17:     0x7b3863fda645 - core::panicking::panic_fmt::h784f20a50eaab275
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
  18:     0x7b3863fdabdb - core::panicking::assert_failed_inner::hbf94b40c37b92af0
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:342:17
  19:     0x7b3867026247 - core[f975038e3cc9791c]::panicking::assert_failed::<u128, u128>
  20:     0x7b3868e4a531 - <rustc_middle[e0f609162babcb92]::query::on_disk_cache::OnDiskCache>::new
  21:     0x7b3868b4bba2 - rustc_incremental[9a504be5dec41bc5]::persist::load::load_query_result_cache
  22:     0x7b3868b4bdaa - rustc_interface[bf5cafa581ab7832]::passes::create_global_ctxt
  23:     0x7b3868cf2af0 - <rustc_interface[bf5cafa581ab7832]::queries::Queries>::global_ctxt
  24:     0x7b3868afe568 - rustc_interface[bf5cafa581ab7832]::interface::run_compiler::<core[f975038e3cc9791c]::result::Result<(), rustc_span[7a149c27976a99e7]::ErrorGuaranteed>, rustc_driver_impl[66ed8fdbde15dc6c]::run_compiler::{closure#0}>::{closure#0}
  25:     0x7b3868c2c7db - std[79729d9c385e1623]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[bf5cafa581ab7832]::util::run_in_thread_with_globals<rustc_interface[bf5cafa581ab7832]::interface::run_compiler<core[f975038e3cc9791c]::result::Result<(), rustc_span[7a149c27976a99e7]::ErrorGuaranteed>, rustc_driver_impl[66ed8fdbde15dc6c]::run_compiler::{closure#0}>::{closure#0}, core[f975038e3cc9791c]::result::Result<(), rustc_span[7a149c27976a99e7]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[f975038e3cc9791c]::result::Result<(), rustc_span[7a149c27976a99e7]::ErrorGuaranteed>>
  26:     0x7b3868c2c639 - <<std[79729d9c385e1623]::thread::Builder>::spawn_unchecked_<rustc_interface[bf5cafa581ab7832]::util::run_in_thread_with_globals<rustc_interface[bf5cafa581ab7832]::interface::run_compiler<core[f975038e3cc9791c]::result::Result<(), rustc_span[7a149c27976a99e7]::ErrorGuaranteed>, rustc_driver_impl[66ed8fdbde15dc6c]::run_compiler::{closure#0}>::{closure#0}, core[f975038e3cc9791c]::result::Result<(), rustc_span[7a149c27976a99e7]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[f975038e3cc9791c]::result::Result<(), rustc_span[7a149c27976a99e7]::ErrorGuaranteed>>::{closure#1} as core[f975038e3cc9791c]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  27:     0x7b3863f988e5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h12de4fc57affb195
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  28:     0x7b3863f988e5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h3c619f45059d5cf1
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  29:     0x7b3863f988e5 - std::sys::unix::thread::Thread::new::thread_start::hbac657605e4b7389
                               at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
  30:     0x7b3863d64b5a - start_thread
                               at ./nptl/pthread_create.c:444:8
  31:     0x7b3863df55fc - __GI___clone3
                               at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
  32:                0x0 - <unknown>

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.76.0 (07dca489a 2024-02-04) running on x86_64-unknown-linux-gnu

note: compiler flags: --crate-type staticlib --crate-type lib -C embed-bitcode=no -C debuginfo=2 -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
error: could not compile `fmw_rust` (lib)

Backtrace

$  RUST_BACKTRACE=1 cargo b --target=x86_64-unknown-linux-gnu
   Compiling fmw_rust v0.1.0 (/home/pg/company-code/ [...] /src/src/fmw_rust)
thread 'rustc' panicked at compiler/rustc_middle/src/query/on_disk_cache.rs:525:5:
assertion `left == right` failed
  left: 0
 right: 1002111927320821928687967599834759150
stack backtrace:
   0: rust_begin_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   2: core::panicking::assert_failed_inner
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:342:17
   3: core::panicking::assert_failed::<u128, u128>
   4: <rustc_middle::query::on_disk_cache::OnDiskCache>::new
   5: rustc_incremental::persist::load::load_query_result_cache
   6: rustc_interface::passes::create_global_ctxt
   7: <rustc_interface::queries::Queries>::global_ctxt
   8: rustc_interface::interface::run_compiler::<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#0}>::{closure#0}
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.76.0 (07dca489a 2024-02-04) running on x86_64-unknown-linux-gnu

note: compiler flags: --crate-type staticlib --crate-type lib -C embed-bitcode=no -C debuginfo=2 -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
error: could not compile `fmw_rust` **(lib)**

[Noratrieb]

@saethlin

[gaultier]

Confirming that cargo clean and then cargo b --target=x86_64-unknown-linux-gnu makes the problem disappear, so it's probably a caching bug.

Correction: It first works and makes the problem disappear. However, after making a code change (renaming foo_u64 back to foo), and rebuilding, the bug reappears, with a different caching 'tag':

thread 'rustc' panicked at compiler/rustc_middle/src/query/on_disk_cache.rs:525:5:
assertion `left == right` failed
  left: 3733157840821228655755399166
 right: 1002111927320821928687967599834759150

[saethlin]

@Nilstrieb The failing assertion is not the one I recently added. The assertion that's firing here is very old:

rust/compiler/rustc_middle/src/query/on_disk_cache.rs

Line 525 in 07dca48

assert_eq!(actual_tag, expected_tag);

[gaultier]

It happened to me again today, same project, different file, after renaming (through the rust-analyzer language server) a variable.

[saethlin]

As far as I can tell, the ICE reported in this issue is caused by another process damaging rustc's build artifacts or by a buggy filesystem. That other process could be anything from an IDE running the compiler at the same time as a manual build to a find-and-replace that accidentally edits files in the target directory.

#124686 contains a mitigation for this situation; the compiler now has a crude way to detect and ignore damaged incremental compilation files. You will still get a warning about the damaged file, but the build should proceed normally. That change shipped in Rust 1.80 which released today, so I am closing this issue and a few others which as far as I can tell all have the same root cause. You shouldn't see this ICE anymore, but please file a new issue if you see something like it on 1.80 or later.