`test-fast` on `ubuntu-24.04-arm` intermittent SIGSEGV or SIGBUS in rustc

[EliahKagan]

Current behavior 😯

Since #1777, a test-fast CI job runs on ubuntu-24.04-arm, which is one of the newly more available 64-bit ARM (AArch64/ARM64) Linux runners. This job initially worked with no problems in this job: the ARM failures mentioned in #1777 and #1778 and tracked in #1780 apply to a test-32bit job, occur only with Docker, and do not affect test-fast.

However, the ARM64 test-fast now intermittently fails with SIGBUS or SIGSEGV in rustc. This is probably a bug in rustsc or another component of the Rust toolchain for ARM64, but I have not reproduced it locally or otherwise ruled out a problem on the runner image. The job, like the other test-fast jobs, uses a stable toolchain.

Expected behavior 🤔

The compilation should complete, or give an error, but not crash. SIGSEGV and SIGBUS should not occur.

The underlying bug is not in gitoxide, but I'm opening this issue in gitoxide to track the problem with the affected CI job here, which may need to be removed, skipped, or made continue-on-error.

Git behavior

Not applicable.

Steps to reproduce 🕹

Run or rerun the test-fast (ubuntu-24.04-arm) job on any commit. It seems less likely to happen if rust-cache is able to retrieve cache dependencies, since there is less to build, but it happens even when caching retrieves everything except what is built from this repository's workspace. It may be necessary to rerun a job multiple times to observe the problem.

A few runs that show this are:

• https://github.com/EliahKagan/gitoxide/actions/runs/12874716014/job/35967617793
• https://github.com/EliahKagan/gitoxide/actions/runs/12899025438/job/35967112460
• https://github.com/EliahKagan/gitoxide/actions/runs/12898120687/job/35967580551

The first link is to a run on the main branch in my fork. The following are relevant pieces of the output of that one run (not separate runs):

Compiling gix-date v0.9.3 (/home/runner/work/gitoxide/gitoxide/gix-date)
error: rustc interrupted by SIGSEGV, printing backtrace

/home/runner/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-bedc4a794a543ce8.so(+0xbf78ec)[0xff6a483f78ec]
linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xff6a517ba7e0]

note: we would appreciate a report at https://github.com/rust-lang/rust
help: you can increase rustc's stack size by setting RUST_MIN_STACK=[16](https://github.com/EliahKagan/gitoxide/actions/runs/12874716014/job/35967617793#step:8:17)777216

error: could not compile `gix-trace` (lib)

Caused by:
  process didn't exit successfully: `/home/runner/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/bin/rustc --crate-name gix_trace --edition=2021 gix-trace/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 '--warn=clippy::pedantic' '--allow=clippy::wildcard_imports' '--allow=clippy::used_underscore_binding' '--allow=clippy::unused_self' '--allow=clippy::unreadable_literal' '--allow=clippy::unnecessary_wraps' '--allow=clippy::unnecessary_join' '--allow=clippy::trivially_copy_pass_by_ref' '--allow=clippy::transmute_ptr_to_ptr' '--allow=clippy::too_many_lines' '--allow=clippy::too_long_first_doc_paragraph' '--allow=clippy::struct_field_names' '--allow=clippy::struct_excessive_bools' '--allow=clippy::stable_sort_primitive' '--allow=clippy::single_match_else' '--allow=clippy::similar_names' '--allow=clippy::should_panic_without_expect' '--allow=clippy::return_self_not_must_use' '--allow=clippy::redundant_else' '--allow=clippy::range_plus_one' '--allow=clippy::option_option' '--allow=clippy::no_effect_underscore_binding' '--allow=clippy::needless_raw_string_hashes' '--allow=clippy::needless_pass_by_value' '--allow=clippy::needless_for_each' '--allow=clippy::naive_bytecount' '--allow=clippy::mut_mut' '--allow=clippy::must_use_candidate' '--allow=clippy::module_name_repetitions' '--allow=clippy::missing_panics_doc' '--allow=clippy::missing_errors_doc' '--allow=clippy::match_wildcard_for_single_variants' '--allow=clippy::match_wild_err_arm' '--allow=clippy::match_same_arms' '--allow=clippy::match_bool' '--allow=clippy::many_single_char_names' '--allow=clippy::manual_string_new' '--allow=clippy::manual_let_else' '--allow=clippy::manual_is_variant_and' '--allow=clippy::manual_assert' '--allow=clippy::large_stack_arrays' '--allow=clippy::iter_without_into_iter' '--allow=clippy::iter_not_returning_iterator' '--allow=clippy::items_after_statements' '--allow=clippy::inline_always' '--allow=clippy::inefficient_to_string' '--allow=clippy::inconsistent_struct_constructor' '--allow=clippy::implicit_clone' '--allow=clippy::ignored_unit_patterns' '--allow=clippy::if_not_else' '--allow=clippy::from_iter_instead_of_collect' '--allow=clippy::fn_params_excessive_bools' '--allow=clippy::filter_map_next' '--allow=clippy::explicit_iter_loop' '--allow=clippy::explicit_into_iter_loop' '--allow=clippy::explicit_deref_methods' '--allow=clippy::enum_glob_use' '--allow=clippy::empty_docs' '--allow=clippy::doc_markdown' '--allow=clippy::default_trait_access' '--allow=clippy::copy_iterator' '--allow=clippy::checked_conversions' '--allow=clippy::cast_sign_loss' '--allow=clippy::cast_precision_loss' '--allow=clippy::cast_possible_wrap' '--allow=clippy::cast_possible_truncation' '--allow=clippy::cast_lossless' '--allow=clippy::borrow_as_ptr' '--allow=clippy::bool_to_int_with_if' --cfg 'feature="default"' --cfg 'feature="tracing"' --cfg 'feature="tracing-detail"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("default", "document-features", "tracing", "tracing-detail"))' -C metadata=70615eb61[26](https://github.com/EliahKagan/gitoxide/actions/runs/12874716014/job/35967617793#step:8:27)64f03 -C extra-filename=-70615eb612664f03 --out-dir /home/runner/work/gitoxide/gitoxide/target/debug/deps -L dependency=/home/runner/work/gitoxide/gitoxide/target/debug/deps --extern tracing_core=/home/runner/work/gitoxide/gitoxide/target/debug/deps/libtracing_core-90c3db60dd8d72e4.rmeta` (signal: 11, SIGSEGV: invalid memory reference)

error: could not compile `gix-hash` (lib)

Caused by:
  process didn't exit successfully: `/home/runner/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/bin/rustc --crate-name gix_hash --edition=2021 gix-hash/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=2 '--warn=clippy::pedantic' '--allow=clippy::wildcard_imports' '--allow=clippy::used_underscore_binding' '--allow=clippy::unused_self' '--allow=clippy::unreadable_literal' '--allow=clippy::unnecessary_wraps' '--allow=clippy::unnecessary_join' '--allow=clippy::trivially_copy_pass_by_ref' '--allow=clippy::transmute_ptr_to_ptr' '--allow=clippy::too_many_lines' '--allow=clippy::too_long_first_doc_paragraph' '--allow=clippy::struct_field_names' '--allow=clippy::struct_excessive_bools' '--allow=clippy::stable_sort_primitive' '--allow=clippy::single_match_else' '--allow=clippy::similar_names' '--allow=clippy::should_panic_without_expect' '--allow=clippy::return_self_not_must_use' '--allow=clippy::redundant_else' '--allow=clippy::range_plus_one' '--allow=clippy::option_option' '--allow=clippy::no_effect_underscore_binding' '--allow=clippy::needless_raw_string_hashes' '--allow=clippy::needless_pass_by_value' '--allow=clippy::needless_for_each' '--allow=clippy::naive_bytecount' '--allow=clippy::mut_mut' '--allow=clippy::must_use_candidate' '--allow=clippy::module_name_repetitions' '--allow=clippy::missing_panics_doc' '--allow=clippy::missing_errors_doc' '--allow=clippy::match_wildcard_for_single_variants' '--allow=clippy::match_wild_err_arm' '--allow=clippy::match_same_arms' '--allow=clippy::match_bool' '--allow=clippy::many_single_char_names' '--allow=clippy::manual_string_new' '--allow=clippy::manual_let_else' '--allow=clippy::manual_is_variant_and' '--allow=clippy::manual_assert' '--allow=clippy::large_stack_arrays' '--allow=clippy::iter_without_into_iter' '--allow=clippy::iter_not_returning_iterator' '--allow=clippy::items_after_statements' '--allow=clippy::inline_always' '--allow=clippy::inefficient_to_string' '--allow=clippy::inconsistent_struct_constructor' '--allow=clippy::implicit_clone' '--allow=clippy::ignored_unit_patterns' '--allow=clippy::if_not_else' '--allow=clippy::from_iter_instead_of_collect' '--allow=clippy::fn_params_excessive_bools' '--allow=clippy::filter_map_next' '--allow=clippy::explicit_iter_loop' '--allow=clippy::explicit_into_iter_loop' '--allow=clippy::explicit_deref_methods' '--allow=clippy::enum_glob_use' '--allow=clippy::empty_docs' '--allow=clippy::doc_markdown' '--allow=clippy::default_trait_access' '--allow=clippy::copy_iterator' '--allow=clippy::checked_conversions' '--allow=clippy::cast_sign_loss' '--allow=clippy::cast_precision_loss' '--allow=clippy::cast_possible_wrap' '--allow=clippy::cast_possible_truncation' '--allow=clippy::cast_lossless' '--allow=clippy::borrow_as_ptr' '--allow=clippy::bool_to_int_with_if' -C debug-assertions=on --cfg 'feature="serde"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("document-features", "serde"))' -C metadata=afbae439064e8dcf -C extra-filename=-afbae439064e8dcf --out-dir /home/runner/work/gitoxide/gitoxide/target/debug/deps -L dependency=/home/runner/work/gitoxide/gitoxide/target/debug/deps --extern faster_hex=/home/runner/work/gitoxide/gitoxide/target/debug/deps/libfaster_hex-3224d66e2cae46b5.rmeta --extern serde=/home/runner/work/gitoxide/gitoxide/target/debug/deps/libserde-fc1536d1[29](https://github.com/EliahKagan/gitoxide/actions/runs/12874716014/job/35967617793#step:8:30)a55ca3.rmeta --extern thiserror=/home/runner/work/gitoxide/gitoxide/target/debug/deps/libthiserror-0af4056d81b4b441.rmeta` (signal: 7, SIGBUS: access to undefined memory)
error: rustc interrupted by SIGSEGV, printing backtrace

/home/runner/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/bin/../lib/librustc_driver-bedc4a794a543ce8.so(+0xbf78ec)[0xff5a2c7f78ec]
linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xff5a35c3f7e0]

note: we would appreciate a report at https://github.com/rust-lang/rust
help: you can increase rustc's stack size by setting RUST_MIN_STACK=16777216

error: could not compile `gix-utils` (lib)

Caused by:
  process didn't exit successfully: `/home/runner/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/bin/rustc --crate-name gix_utils --edition=2021 gix-utils/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 '--warn=clippy::pedantic' '--allow=clippy::wildcard_imports' '--allow=clippy::used_underscore_binding' '--allow=clippy::unused_self' '--allow=clippy::unreadable_literal' '--allow=clippy::unnecessary_wraps' '--allow=clippy::unnecessary_join' '--allow=clippy::trivially_copy_pass_by_ref' '--allow=clippy::transmute_ptr_to_ptr' '--allow=clippy::too_many_lines' '--allow=clippy::too_long_first_doc_paragraph' '--allow=clippy::struct_field_names' '--allow=clippy::struct_excessive_bools' '--allow=clippy::stable_sort_primitive' '--allow=clippy::single_match_else' '--allow=clippy::similar_names' '--allow=clippy::should_panic_without_expect' '--allow=clippy::return_self_not_must_use' '--allow=clippy::redundant_else' '--allow=clippy::range_plus_one' '--allow=clippy::option_option' '--allow=clippy::no_effect_underscore_binding' '--allow=clippy::needless_raw_string_hashes' '--allow=clippy::needless_pass_by_value' '--allow=clippy::needless_for_each' '--allow=clippy::naive_bytecount' '--allow=clippy::mut_mut' '--allow=clippy::must_use_candidate' '--allow=clippy::module_name_repetitions' '--allow=clippy::missing_panics_doc' '--allow=clippy::missing_errors_doc' '--allow=clippy::match_wildcard_for_single_variants' '--allow=clippy::match_wild_err_arm' '--allow=clippy::match_same_arms' '--allow=clippy::match_bool' '--allow=clippy::many_single_char_names' '--allow=clippy::manual_string_new' '--allow=clippy::manual_let_else' '--allow=clippy::manual_is_variant_and' '--allow=clippy::manual_assert' '--allow=clippy::large_stack_arrays' '--allow=clippy::iter_without_into_iter' '--allow=clippy::iter_not_returning_iterator' '--allow=clippy::items_after_statements' '--allow=clippy::inline_always' '--allow=clippy::inefficient_to_string' '--allow=clippy::inconsistent_struct_constructor' '--allow=clippy::implicit_clone' '--allow=clippy::ignored_unit_patterns' '--allow=clippy::if_not_else' '--allow=clippy::from_iter_instead_of_collect' '--allow=clippy::fn_params_excessive_bools' '--allow=clippy::filter_map_next' '--allow=clippy::explicit_iter_loop' '--allow=clippy::explicit_into_iter_loop' '--allow=clippy::explicit_deref_methods' '--allow=clippy::enum_glob_use' '--allow=clippy::empty_docs' '--allow=clippy::doc_markdown' '--allow=clippy::default_trait_access' '--allow=clippy::copy_iterator' '--allow=clippy::checked_conversions' '--allow=clippy::cast_sign_loss' '--allow=clippy::cast_precision_loss' '--allow=clippy::cast_possible_wrap' '--allow=clippy::cast_possible_truncation' '--allow=clippy::cast_lossless' '--allow=clippy::borrow_as_ptr' '--allow=clippy::bool_to_int_with_if' --cfg 'feature="bstr"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("bstr"))' -C metadata=78d5054a26f3e2ca -C extra-filename=-78d5054a26f3e2ca --out-dir /home/runner/work/gitoxide/gitoxide/target/debug/deps -L dependency=/home/runner/work/gitoxide/gitoxide/target/debug/deps --extern bstr=/home/runner/work/gitoxide/gitoxide/target/debug/deps/libbstr-36fa1f333109ab1c.rmeta --extern fastrand=/home/runner/work/gitoxide/gitoxide/target/debug/deps/libfastrand-e2bb9a6311991097.rmeta --extern unicode_normalization=/home/runner/work/gitoxide/gitoxide/target/debug/deps/libunicode_normalization-6d4aa053d3ce5a81.rmeta` (signal: 11, SIGSEGV: invalid memory reference)

error: could not compile `gix-date` (lib)

Caused by:
  process didn't exit successfully: `/home/runner/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/bin/rustc --crate-name gix_date --edition=2021 gix-date/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 '--warn=clippy::pedantic' '--allow=clippy::wildcard_imports' '--allow=clippy::used_underscore_binding' '--allow=clippy::unused_self' '--allow=clippy::unreadable_literal' '--allow=clippy::unnecessary_wraps' '--allow=clippy::unnecessary_join' '--allow=clippy::trivially_copy_pass_by_ref' '--allow=clippy::transmute_ptr_to_ptr' '--allow=clippy::too_many_lines' '--allow=clippy::too_long_first_doc_paragraph' '--allow=clippy::struct_field_names' '--allow=clippy::struct_excessive_bools' '--allow=clippy::stable_sort_primitive' '--allow=clippy::single_match_else' '--allow=clippy::similar_names' '--allow=clippy::should_panic_without_expect' '--allow=clippy::return_self_not_must_use' '--allow=clippy::redundant_else' '--allow=clippy::range_plus_one' '--allow=clippy::option_option' '--allow=clippy::no_effect_underscore_binding' '--allow=clippy::needless_raw_string_hashes' '--allow=clippy::needless_pass_by_value' '--allow=clippy::needless_for_each' '--allow=clippy::naive_bytecount' '--allow=clippy::mut_mut' '--allow=clippy::must_use_candidate' '--allow=clippy::module_name_repetitions' '--allow=clippy::missing_panics_doc' '--allow=clippy::missing_errors_doc' '--allow=clippy::match_wildcard_for_single_variants' '--allow=clippy::match_wild_err_arm' '--allow=clippy::match_same_arms' '--allow=clippy::match_bool' '--allow=clippy::many_single_char_names' '--allow=clippy::manual_string_new' '--allow=clippy::manual_let_else' '--allow=clippy::manual_is_variant_and' '--allow=clippy::manual_assert' '--allow=clippy::large_stack_arrays' '--allow=clippy::iter_without_into_iter' '--allow=clippy::iter_not_returning_iterator' '--allow=clippy::items_after_statements' '--allow=clippy::inline_always' '--allow=clippy::inefficient_to_string' '--allow=clippy::inconsistent_struct_constructor' '--allow=clippy::implicit_clone' '--allow=clippy::ignored_unit_patterns' '--allow=clippy::if_not_else' '--allow=clippy::from_iter_instead_of_collect' '--allow=clippy::fn_params_excessive_bools' '--allow=clippy::filter_map_next' '--allow=clippy::explicit_iter_loop' '--allow=clippy::explicit_into_iter_loop' '--allow=clippy::explicit_deref_methods' '--allow=clippy::enum_glob_use' '--allow=clippy::empty_docs' '--allow=clippy::doc_markdown' '--allow=clippy::default_trait_access' '--allow=clippy::copy_iterator' '--allow=clippy::checked_conversions' '--allow=clippy::cast_sign_loss' '--allow=clippy::cast_precision_loss' '--allow=clippy::cast_possible_wrap' '--allow=clippy::cast_possible_truncation' '--allow=clippy::cast_lossless' '--allow=clippy::borrow_as_ptr' '--allow=clippy::bool_to_int_with_if' --cfg 'feature="serde"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("document-features", "serde"))' -C metadata=d3f2fa21d502802c -C extra-filename=-d3f2fa21d502802c --out-dir /home/runner/work/gitoxide/gitoxide/target/debug/deps -L dependency=/home/runner/work/gitoxide/gitoxide/target/debug/deps --extern bstr=/home/runner/work/gitoxide/gitoxide/target/debug/deps/libbstr-36fa1f333109ab1c.rmeta --extern itoa=/home/runner/work/gitoxide/gitoxide/target/debug/deps/libitoa-5181ef3238623c92.rmeta --extern jiff=/home/runner/work/gitoxide/gitoxide/target/debug/deps/libjiff-4e2aca16ca2f1b49.rmeta --extern serde=/home/runner/work/gitoxide/gitoxide/target/debug/deps/libserde-fc1536d129a55ca3.rmeta --extern thiserror=/home/runner/work/gitoxide/gitoxide/target/debug/deps/libthiserror-0af4056d81b4b[44](https://github.com/EliahKagan/gitoxide/actions/runs/12874716014/job/35967617793#step:8:45)1.rmeta` (signal: 11, SIGSEGV: invalid memory reference)

[Byron]

Thanks for reporting!

I didn't notice this yet and hope that this will be a rare occurrence. If not, like proposed, it could be made 'non-blocking'.

[Byron]

Actually, it just failed on main: https://github.com/GitoxideLabs/gitoxide/actions/runs/12903033259/job/35977612391

Maybe it's best to just make it non-blocking right away.

[EliahKagan]

The error messages suggest to force a minimum stack size for rustc using an environment variable. This is part of the messages from before as well. This is in the same message as its recommendation to open a bug report, so if this works then it's a workaround and there's still a bug that should be reported for rustc, unless it's somehow due to a problem in the runner image. I'm not sure if that suggestion is given anytime rustc fails with SIGSEGV, anytime it fails with SIGSEGV where SIGBUS occurred, or more specifically such that it means there is actually an indication of insufficient stack.

I am about to look into whether setting that helps. Then I'll open a PR to improve the situation one way or another. (Making it non-blocking for PRs by splitting it out of test-fast and having it not be a dependency of a required check is something I hadn't thought to do, but it is a good idea.)

Rerunning the check may make it pass, since the failure is intermittent. But I am not recommending that as a substitute for a change that would make the check fail less often (or not at all) or that would change how we treat failures of that check.

[EliahKagan]

Maybe it's best to just make it non-blocking right away.

I've gone ahead and done this in #1792. I suspect it can be adjusted and made blocking again, but I'm not done with the research to figure out how, so I think it does make sense to make it non-blocking temporarily.

[EliahKagan]

In both e71b0cf (results) and 5a71963 (results), a pattern emerges: the memory errors on ARM64 Linux CI runners seem only ever to happen on the ubuntu-24.04-arm runner, not the ubuntu-22.04-arm runner, and seem only ever to happen with the stable channel, not the beta channel (and also probably not the nightly channel, but I didn't test as much with that--only in e71b0cf--and there are ordinary build errors that happen on it).

This remains the case even if we regard panics that suggest but do not prove memory errors, such as large range start indices that are much bigger than the range, to be memory errors. I observed these in 5a71963.

A third kind of error, which is a memory error even if narrowly defined, also happened only in 5a71963, and only once: in dtolnay/rust-toolchain, running rustup failed with free(): invalid next size (fast). This was also on 24.04 with the stable toolchain.

Another kind of error that I had originally assumed was unrelated was that actions/checkout failed a few times. This happened occasionally in both e71b0cf and 5a71963. This is unrelated to Rust. It was strange, because no error message was written to the log, suggesting something crashed. It also only happened on 24.04. But it didn't happen often enough that this is decisive; it's possible that Rust is essential to this problem, or even that there are two or more separate problems some of which are entirely in the Rust toolchain. I suspect something is going on with the 24.04 image, though.

For jobs where the actions/checkout error happened, I reran those jobs. (It did not recur.) I did not rerun any other jobs in either e71b0cf or 5a71963 (the repetition to distinguish correlation from happenstance is in the CI matrix definition itself, rather than requiring checks to be manually rerun).

So it looks like it would be sufficient to change ubuntu-24.04-arm to ubuntu-22.04-arm in the test-fast job. Alternatively, it would likely be sufficient to use the beta toolchain instead of the stable one (which could be special-cased even if the affected ARM64 jobs remain part of, or are put back into, the test-fast matrix).

Why do (22.04, stable), (22.04, beta), and (24.04, beta) all work, and it is only (24.04, stable) that does not? I think #1792 may still be the best first move, until I've investigated that question more. So far I've look at issues related to the runner images, and the only issue that seems maybe related, indirectly, is actions/partner-runner-images#36. I have not yet looked at what has changed between the stable and beta channels of the toolchain the ARM64 jobs are using.

[EliahKagan]

The SIGSEGV and "range start index" errors are rust-lang/rust#135867.