Nondeterministic(?) SIGSEGV&co. in CI

[workingjubilee]

See #89167, getting lovely stuff like core dumps here:

x86_64 -> ppc64le dist fail, first couple of tries

2021-11-11T05:45:43.4343491Z    Compiling proc-macro2 v1.0.30
2021-11-11T05:45:43.4347792Z    Compiling unicode-xid v0.2.2
2021-11-11T05:45:43.4349026Z    Compiling syn v1.0.80
2021-11-11T05:45:43.4350026Z    Compiling memchr v2.4.1
2021-11-11T05:45:43.4351359Z    Compiling version_check v0.9.3
2021-11-11T05:45:43.4352483Z    Compiling autocfg v1.0.0
2021-11-11T05:45:43.4353539Z    Compiling lazy_static v1.4.0
2021-11-11T05:45:43.4354559Z    Compiling libc v0.2.106
2021-11-11T05:45:43.4360979Z    Compiling cfg-if v1.0.0
2021-11-11T05:45:43.4394877Z    Compiling serde_derive v1.0.125
2021-11-11T05:45:43.4439872Z    Compiling regex-automata v0.1.10
2021-11-11T05:45:43.4470847Z    Compiling log v0.4.14
2021-11-11T05:45:43.4472199Z    Compiling serde v1.0.125
2021-11-11T05:45:43.4473313Z    Compiling regex-syntax v0.6.25
2021-11-11T05:45:43.4474103Z    Compiling ryu v1.0.5
2021-11-11T05:45:43.4474733Z    Compiling serde_json v1.0.59
2021-11-11T05:45:43.4831568Z    Compiling same-file v1.0.6
2021-11-11T05:45:43.5221357Z    Compiling fnv v1.0.7
2021-11-11T05:45:43.5364027Z    Compiling itoa v0.4.6
2021-11-11T05:45:43.5924101Z    Compiling unicode-width v0.1.8
2021-11-11T05:45:43.6321886Z    Compiling cc v1.0.69
2021-11-11T05:45:43.6517998Z    Compiling bootstrap v0.0.0 (/checkout/src/bootstrap)
2021-11-11T05:45:43.6676262Z    Compiling build_helper v0.1.0 (/checkout/src/build_helper)
2021-11-11T05:45:43.7870653Z    Compiling once_cell v1.7.2
2021-11-11T05:45:43.8081458Z    Compiling thread_local v1.0.1
2021-11-11T05:45:43.8487642Z    Compiling walkdir v2.3.1
2021-11-11T05:45:43.8989047Z    Compiling getopts v0.2.21
2021-11-11T05:45:43.9527579Z    Compiling proc-macro-error-attr v1.0.4
2021-11-11T05:45:43.9528752Z    Compiling proc-macro-error v1.0.4
2021-11-11T05:45:43.9632438Z    Compiling num-traits v0.2.12
2021-11-11T05:45:43.9710731Z    Compiling crossbeam-utils v0.8.3
2021-11-11T05:45:44.1733708Z    Compiling cmake v0.1.44
2021-11-11T05:45:44.4300525Z    Compiling bstr v0.2.13
2021-11-11T05:45:44.4301914Z    Compiling aho-corasick v0.7.18
2021-11-11T05:45:44.4471608Z    Compiling quote v1.0.7
2021-11-11T05:45:44.6246359Z    Compiling num_cpus v1.13.0
2021-11-11T05:45:44.6247547Z    Compiling time v0.1.43
2021-11-11T05:45:44.6249485Z    Compiling filetime v0.2.14
2021-11-11T05:45:45.0687445Z    Compiling opener v0.5.0
2021-11-11T05:45:45.4587158Z error: could not compile `aho-corasick`
2021-11-11T05:45:45.4587753Z 
2021-11-11T05:45:45.4588202Z Caused by:
2021-11-11T05:45:45.4600450Z   process didn't exit successfully: `/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustc --crate-name aho_corasick --edition=2018 /cargo/registry/src/github.com-1ecc6299db9ec823/aho-corasick-0.7.18/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=0 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=6acdc16a2f6d6338 -C extra-filename=-6acdc16a2f6d6338 --out-dir /checkout/obj/build/bootstrap/debug/deps -L dependency=/checkout/obj/build/bootstrap/debug/deps --extern memchr=/checkout/obj/build/bootstrap/debug/deps/libmemchr-9d65e5e3abdde0ba.rmeta --cap-lints allow -Wrust_2018_idioms -Wunused_lifetimes -Wsemicolon_in_expressions_from_macros -Dwarnings` (signal: 11, SIGSEGV: invalid memory reference)
2021-11-11T05:45:45.4605387Z warning: build failed, waiting for other jobs to finish...
2021-11-11T05:45:48.1296579Z error: build failed
2021-11-11T05:45:48.1342142Z failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
2021-11-11T05:45:48.1344587Z Build completed unsuccessfully in 0:00:55
2021-11-11T05:45:48.1485824Z make: *** [prepare] Error 1
2021-11-11T05:45:49.1577087Z Command failed. Attempt 2/5:
2021-11-11T05:45:49.4859700Z    Compiling aho-corasick v0.7.18
2021-11-11T05:45:49.4861068Z    Compiling serde_derive v1.0.125
2021-11-11T05:45:49.4862658Z    Compiling proc-macro-error v1.0.4
2021-11-11T05:45:49.7244965Z    Compiling merge_derive v0.1.0
2021-11-11T05:45:50.2771981Z    Compiling regex v1.5.4
2021-11-11T05:45:50.3023062Z fatal runtime error: Rust cannot catch foreign exceptions
2021-11-11T05:45:50.4491990Z error: could not compile `regex`
2021-11-11T05:45:50.4492882Z 
2021-11-11T05:45:50.4493631Z Caused by:
2021-11-11T05:45:50.4509899Z   process didn't exit successfully: `/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustc --crate-name regex --edition=2018 /cargo/registry/src/github.com-1ecc6299db9ec823/regex-1.5.4/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=0 --cfg 'feature="aho-corasick"' --cfg 'feature="default"' --cfg 'feature="memchr"' --cfg 'feature="perf"' --cfg 'feature="perf-cache"' --cfg 'feature="perf-dfa"' --cfg 'feature="perf-inline"' --cfg 'feature="perf-literal"' --cfg 'feature="std"' --cfg 'feature="unicode"' --cfg 'feature="unicode-age"' --cfg 'feature="unicode-bool"' --cfg 'feature="unicode-case"' --cfg 'feature="unicode-gencat"' --cfg 'feature="unicode-perl"' --cfg 'feature="unicode-script"' --cfg 'feature="unicode-segment"' -C metadata=b0c184cb911fb168 -C extra-filename=-b0c184cb911fb168 --out-dir /checkout/obj/build/bootstrap/debug/deps -L dependency=/checkout/obj/build/bootstrap/debug/deps --extern aho_corasick=/checkout/obj/build/bootstrap/debug/deps/libaho_corasick-6acdc16a2f6d6338.rmeta --extern memchr=/checkout/obj/build/bootstrap/debug/deps/libmemchr-9d65e5e3abdde0ba.rmeta --extern regex_syntax=/checkout/obj/build/bootstrap/debug/deps/libregex_syntax-74207c8016af6cf7.rmeta --cap-lints allow -Wrust_2018_idioms -Wunused_lifetimes -Wsemicolon_in_expressions_from_macros -Dwarnings` (signal: 6, SIGABRT: process abort signal)
2021-11-11T05:45:50.4523349Z warning: build failed, waiting for other jobs to finish...
2021-11-11T05:45:53.0676231Z error: build failed
2021-11-11T05:45:53.0706948Z failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
2021-11-11T05:45:53.0708269Z Build completed unsuccessfully in 0:00:03
2021-11-11T05:45:53.0833472Z make: *** [prepare] Error 1

x86_64 -> ppc64le dist fail, last try

2021-11-11T05:46:12.9222679Z + python3 ../x.py dist --host powerpc64le-unknown-linux-gnu --target powerpc64le-unknown-linux-gnu
2021-11-11T05:46:13.2203039Z     Finished dev [unoptimized] target(s) in 0.21s
2021-11-11T05:46:13.5549550Z [TIMING] Mingw { host: TargetSelection { triple: "powerpc64le-unknown-linux-gnu", file: None } } -- 0.000
2021-11-11T05:46:13.5553431Z [TIMING] Assemble { target_compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } } -- 0.000
2021-11-11T05:46:13.5557531Z [TIMING] StartupObjects { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } -- 0.000
2021-11-11T05:46:13.5561344Z [TIMING] Sysroot { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } } -- 0.000
2021-11-11T05:46:13.5565149Z [TIMING] Libdir { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } -- 0.000
2021-11-11T05:46:13.5568511Z Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2021-11-11T05:46:13.9338957Z �[0m�[0m�[1m�[32m Downloading�[0m crates ...
2021-11-11T05:46:14.0320028Z �[0m�[0m�[1m�[32m  Downloaded�[0m rustc-demangle v0.1.21
2021-11-11T05:46:14.0338932Z �[0m�[0m�[1m�[32m  Downloaded�[0m adler v0.2.3
2021-11-11T05:46:14.0358889Z �[0m�[0m�[1m�[32m  Downloaded�[0m miniz_oxide v0.4.0
2021-11-11T05:46:14.0385982Z �[0m�[0m�[1m�[32m  Downloaded�[0m hashbrown v0.11.0
2021-11-11T05:46:14.0436912Z �[0m�[0m�[1m�[32m  Downloaded�[0m addr2line v0.16.0
2021-11-11T05:46:14.0467713Z �[0m�[0m�[1m�[32m  Downloaded�[0m object v0.26.2
2021-11-11T05:46:14.0595396Z �[0m�[0m�[1m�[32m  Downloaded�[0m gimli v0.25.0
2021-11-11T05:46:14.0837740Z �[0m�[0m�[1m�[32m  Downloaded�[0m compiler_builtins v0.1.49
2021-11-11T05:46:14.0966269Z �[0m�[0m�[1m�[32m  Downloaded�[0m cfg-if v0.1.10
2021-11-11T05:46:14.2035928Z �[0m�[0m�[1m�[32m   Compiling�[0m cc v1.0.69
2021-11-11T05:46:14.2037277Z �[0m�[0m�[1m�[32m   Compiling�[0m core v0.0.0 (/checkout/library/core)
2021-11-11T05:46:14.2038453Z �[0m�[0m�[1m�[32m   Compiling�[0m libc v0.2.106
2021-11-11T05:46:14.2039821Z �[0m�[0m�[1m�[32m   Compiling�[0m memchr v2.4.1
2021-11-11T05:46:14.2041265Z �[0m�[0m�[1m�[32m   Compiling�[0m std v0.0.0 (/checkout/library/std)
2021-11-11T05:46:14.4328299Z [RUSTC-TIMING] build_script_build test:false 0.220
2021-11-11T05:46:14.4329847Z rustc exited with signal: 11 (core dumped)
2021-11-11T05:46:14.4331456Z �[0m�[0m�[1m�[31merror�[0m�[1m:�[0m could not compile `std`
2021-11-11T05:46:14.4332223Z 
2021-11-11T05:46:14.4332951Z Caused by:
2021-11-11T05:46:14.4343428Z   process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc --crate-name build_script_build --edition=2018 library/std/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=1 -C debug-assertions=off --cfg 'feature="addr2line"' --cfg 'feature="backtrace"' --cfg 'feature="compiler-builtins-c"' --cfg 'feature="gimli-symbolize"' --cfg 'feature="miniz_oxide"' --cfg 'feature="object"' --cfg 'feature="panic_unwind"' --cfg 'feature="std_detect_dlsym_getauxval"' --cfg 'feature="std_detect_file_io"' -C metadata=4fbeca988c122c63 -C extra-filename=-4fbeca988c122c63 --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/release/build/std-4fbeca988c122c63 -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-std/release/deps -Z binary-dep-depinfo` (exit status: 254)
2021-11-11T05:46:14.4352471Z �[0m�[0m�[1m�[33mwarning�[0m�[1m:�[0m build failed, waiting for other jobs to finish...
2021-11-11T05:46:14.5062557Z [RUSTC-TIMING] build_script_build test:false 0.295
2021-11-11T05:46:14.5584858Z [RUSTC-TIMING] build_script_build test:false 0.347
2021-11-11T05:46:14.9907233Z [RUSTC-TIMING] cc test:false 0.781
2021-11-11T05:46:30.4861238Z [RUSTC-TIMING] core test:false 16.277
2021-11-11T05:46:30.5039886Z �[0m�[0m�[1m�[31merror�[0m�[1m:�[0m build failed
2021-11-11T05:46:30.5069716Z Build completed unsuccessfully in 0:00:17
2021-11-11T05:46:30.5208193Z == clock drift check ==
2021-11-11T05:46:30.5219374Z   local time: Thu Nov 11 05:46:30 UTC 2021
2021-11-11T05:46:30.5963718Z   network time: Wed, 10 Nov 2021 17:28:17 GMT
2021-11-11T05:46:30.5968760Z == end clock drift check ==
2021-11-11T05:46:31.0960411Z ##[error]Process completed with exit code 1.

And in #90792:

x86_64 -> s390x dist fail, first few times

2021-11-11T08:57:21.2424040Z    Compiling proc-macro2 v1.0.30
2021-11-11T08:57:21.2427063Z    Compiling unicode-xid v0.2.2
2021-11-11T08:57:21.2428595Z    Compiling syn v1.0.80
2021-11-11T08:57:21.2429719Z    Compiling version_check v0.9.3
2021-11-11T08:57:21.2430787Z    Compiling memchr v2.4.1
2021-11-11T08:57:21.2431804Z    Compiling autocfg v1.0.0
2021-11-11T08:57:21.2432852Z    Compiling lazy_static v1.4.0
2021-11-11T08:57:21.2433967Z    Compiling serde_derive v1.0.125
2021-11-11T08:57:21.2453507Z    Compiling cfg-if v1.0.0
2021-11-11T08:57:21.2454684Z    Compiling libc v0.2.106
2021-11-11T08:57:21.2455701Z    Compiling serde v1.0.125
2021-11-11T08:57:21.2564334Z    Compiling log v0.4.14
2021-11-11T08:57:21.2565776Z    Compiling regex-automata v0.1.10
2021-11-11T08:57:21.2566454Z    Compiling ryu v1.0.5
2021-11-11T08:57:21.2567251Z    Compiling regex-syntax v0.6.25
2021-11-11T08:57:21.2573481Z    Compiling same-file v1.0.6
2021-11-11T08:57:21.2914141Z    Compiling serde_json v1.0.59
2021-11-11T08:57:21.3383315Z    Compiling fnv v1.0.7
2021-11-11T08:57:21.3542478Z    Compiling itoa v0.4.6
2021-11-11T08:57:21.4114984Z    Compiling unicode-width v0.1.8
2021-11-11T08:57:21.4145028Z    Compiling bootstrap v0.0.0 (/checkout/src/bootstrap)
2021-11-11T08:57:21.4687993Z    Compiling cc v1.0.69
2021-11-11T08:57:21.4841645Z    Compiling build_helper v0.1.0 (/checkout/src/build_helper)
2021-11-11T08:57:21.5830518Z error: could not compile `autocfg`
2021-11-11T08:57:21.5831003Z 
2021-11-11T08:57:21.5831470Z Caused by:
2021-11-11T08:57:21.5836507Z   process didn't exit successfully: `/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustc --crate-name autocfg /cargo/registry/src/github.com-1ecc6299db9ec823/autocfg-1.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=0 -C metadata=fa5fb2f81f71684d -C extra-filename=-fa5fb2f81f71684d --out-dir /checkout/obj/build/bootstrap/debug/deps -L dependency=/checkout/obj/build/bootstrap/debug/deps --cap-lints allow -Wrust_2018_idioms -Wunused_lifetimes -Wsemicolon_in_expressions_from_macros -Dwarnings` (signal: 11, SIGSEGV: invalid memory reference)
2021-11-11T08:57:21.5840643Z warning: build failed, waiting for other jobs to finish...
2021-11-11T08:57:23.8821586Z error: build failed
2021-11-11T08:57:23.8867676Z failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
2021-11-11T08:57:23.8869967Z Build completed unsuccessfully in 0:00:53
2021-11-11T08:57:23.8980776Z make: *** [prepare] Error 1
2021-11-11T08:57:24.9016567Z Command failed. Attempt 2/5:
2021-11-11T08:57:25.2947855Z    Compiling autocfg v1.0.0
2021-11-11T08:57:25.2951386Z    Compiling once_cell v1.7.2
2021-11-11T08:57:25.2953741Z    Compiling proc-macro2 v1.0.30
2021-11-11T08:57:25.2954845Z    Compiling syn v1.0.80
2021-11-11T08:57:25.2956436Z    Compiling proc-macro-error-attr v1.0.4
2021-11-11T08:57:25.2958157Z    Compiling proc-macro-error v1.0.4
2021-11-11T08:57:25.3013492Z    Compiling memchr v2.4.1
2021-11-11T08:57:25.3014163Z    Compiling libc v0.2.106
2021-11-11T08:57:25.3015198Z    Compiling serde_derive v1.0.125
2021-11-11T08:57:25.3025089Z    Compiling log v0.4.14
2021-11-11T08:57:25.3051701Z    Compiling serde v1.0.125
2021-11-11T08:57:25.3063135Z    Compiling ryu v1.0.5
2021-11-11T08:57:25.3065652Z    Compiling walkdir v2.3.1
2021-11-11T08:57:25.3082080Z    Compiling serde_json v1.0.59
2021-11-11T08:57:25.3099137Z    Compiling thread_local v1.0.1
2021-11-11T08:57:25.3116734Z    Compiling cmake v0.1.44
2021-11-11T08:57:25.3130212Z    Compiling bootstrap v0.0.0 (/checkout/src/bootstrap)
2021-11-11T08:57:25.3133424Z    Compiling getopts v0.2.21
2021-11-11T08:57:25.6113273Z error: could not compile `thread_local`
2021-11-11T08:57:25.6113780Z 
2021-11-11T08:57:25.6114225Z Caused by:
2021-11-11T08:57:25.6120130Z   process didn't exit successfully: `/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustc --crate-name thread_local /cargo/registry/src/github.com-1ecc6299db9ec823/thread_local-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=0 -C metadata=ae9887d7ee5a3068 -C extra-filename=-ae9887d7ee5a3068 --out-dir /checkout/obj/build/bootstrap/debug/deps -L dependency=/checkout/obj/build/bootstrap/debug/deps --extern lazy_static=/checkout/obj/build/bootstrap/debug/deps/liblazy_static-1f5d3923e02f6398.rmeta --cap-lints allow -Wrust_2018_idioms -Wunused_lifetimes -Wsemicolon_in_expressions_from_macros -Dwarnings` (signal: 11, SIGSEGV: invalid memory reference)
2021-11-11T08:57:25.6125005Z warning: build failed, waiting for other jobs to finish...
2021-11-11T08:57:25.9821848Z error: build failed
2021-11-11T08:57:25.9850086Z failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
2021-11-11T08:57:25.9851438Z Build completed unsuccessfully in 0:00:01
2021-11-11T08:57:25.9955610Z make: *** [prepare] Error 1
2021-11-11T08:57:27.9975727Z Command failed. Attempt 3/5:
2021-11-11T08:57:28.3254402Z    Compiling crossbeam-utils v0.8.3
2021-11-11T08:57:28.3256081Z    Compiling num-traits v0.2.12
2021-11-11T08:57:28.3257225Z    Compiling thread_local v1.0.1
2021-11-11T08:57:28.3258278Z    Compiling quote v1.0.7
2021-11-11T08:57:28.3259640Z    Compiling aho-corasick v0.7.18
2021-11-11T08:57:28.3260723Z    Compiling bstr v0.2.13
2021-11-11T08:57:28.3289966Z    Compiling num_cpus v1.13.0
2021-11-11T08:57:28.3307921Z    Compiling filetime v0.2.14
2021-11-11T08:57:28.3315903Z    Compiling time v0.1.43
2021-11-11T08:57:28.5433392Z    Compiling syn v1.0.80
2021-11-11T08:57:28.7009450Z error: could not compile `quote`
2021-11-11T08:57:28.7010012Z 
2021-11-11T08:57:28.7010462Z Caused by:
2021-11-11T08:57:28.7016512Z   process didn't exit successfully: `/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustc --crate-name quote --edition=2018 /cargo/registry/src/github.com-1ecc6299db9ec823/quote-1.0.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=0 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=8a2da347e4bf40b5 -C extra-filename=-8a2da347e4bf40b5 --out-dir /checkout/obj/build/bootstrap/debug/deps -L dependency=/checkout/obj/build/bootstrap/debug/deps --extern proc_macro2=/checkout/obj/build/bootstrap/debug/deps/libproc_macro2-2398ddf6b8c4f5ba.rmeta --cap-lints allow -Wrust_2018_idioms -Wunused_lifetimes -Wsemicolon_in_expressions_from_macros -Dwarnings` (signal: 11, SIGSEGV: invalid memory reference)
2021-11-11T08:57:28.7021187Z warning: build failed, waiting for other jobs to finish...
2021-11-11T08:57:31.9186543Z error: build failed
2021-11-11T08:57:31.9210050Z failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
2021-11-11T08:57:31.9212317Z Build completed unsuccessfully in 0:00:03
2021-11-11T08:57:31.9315372Z make: *** [prepare] Error 1

x86_64 -> s390x dist fail, last time

   Compiling build_helper v0.1.0 (/checkout/src/build_helper)
error: could not compile `autocfg`

Caused by:
  process didn't exit successfully: `/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustc --crate-name autocfg /cargo/registry/src/github.com-1ecc6299db9ec823/autocfg-1.0.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=0 -C metadata=fa5fb2f81f71684d -C extra-filename=-fa5fb2f81f71684d --out-dir /checkout/obj/build/bootstrap/debug/deps -L dependency=/checkout/obj/build/bootstrap/debug/deps --cap-lints allow -Wrust_2018_idioms -Wunused_lifetimes -Wsemicolon_in_expressions_from_macros -Dwarnings` (signal: 11, SIGSEGV: invalid memory reference)
error: build failed
failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
Build completed unsuccessfully in 0:00:53
make: *** [prepare] Error 1
---
   Compiling getopts v0.2.21
error: could not compile `thread_local`

Caused by:
  process didn't exit successfully: `/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustc --crate-name thread_local /cargo/registry/src/github.com-1ecc6299db9ec823/thread_local-1.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=0 -C metadata=ae9887d7ee5a3068 -C extra-filename=-ae9887d7ee5a3068 --out-dir /checkout/obj/build/bootstrap/debug/deps -L dependency=/checkout/obj/build/bootstrap/debug/deps --extern lazy_static=/checkout/obj/build/bootstrap/debug/deps/liblazy_static-1f5d3923e02f6398.rmeta --cap-lints allow -Wrust_2018_idioms -Wunused_lifetimes -Wsemicolon_in_expressions_from_macros -Dwarnings` (signal: 11, SIGSEGV: invalid memory reference)
error: build failed
failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
Build completed unsuccessfully in 0:00:01
make: *** [prepare] Error 1
---
   Compiling syn v1.0.80
error: could not compile `quote`

Caused by:
  process didn't exit successfully: `/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/rustc --crate-name quote --edition=2018 /cargo/registry/src/github.com-1ecc6299db9ec823/quote-1.0.7/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=0 --cfg 'feature="default"' --cfg 'feature="proc-macro"' -C metadata=8a2da347e4bf40b5 -C extra-filename=-8a2da347e4bf40b5 --out-dir /checkout/obj/build/bootstrap/debug/deps -L dependency=/checkout/obj/build/bootstrap/debug/deps --extern proc_macro2=/checkout/obj/build/bootstrap/debug/deps/libproc_macro2-2398ddf6b8c4f5ba.rmeta --cap-lints allow -Wrust_2018_idioms -Wunused_lifetimes -Wsemicolon_in_expressions_from_macros -Dwarnings` (signal: 11, SIGSEGV: invalid memory reference)
error: build failed
failed to run: /checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /checkout/src/bootstrap/Cargo.toml --locked
Build completed unsuccessfully in 0:00:03
make: *** [prepare] Error 1
---
   Compiling opaque-debug v0.3.0
[RUSTC-TIMING] build_script_build test:false 0.393
   Compiling rustc_graphviz v0.0.0 (/checkout/compiler/rustc_graphviz)
[RUSTC-TIMING] either test:false 0.305
rustc exited with signal: 11 (core dumped)
error: could not compile `either`
Caused by:
Caused by:
  process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc --crate-name either /cargo/registry/src/github.com-1ecc6299db9ec823/either-1.6.0/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C debuginfo=0 --cfg 'feature="default"' --cfg 'feature="use_std"' -C metadata=22689a2d6fcd3999 -C extra-filename=-22689a2d6fcd3999 --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/x86_64-unknown-linux-gnu/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/release/deps --cap-lints allow --cfg=bootstrap -Zsymbol-mangling-version=v0 -Zmacro-backtrace '-Clink-args=-Wl,-rpath,$ORIGIN/../lib' -Ztls-model=initial-exec -Zunstable-options '-Wrustc::internal' -Cprefer-dynamic -Z binary-dep-depinfo` (exit status: 254)
[RUSTC-TIMING] opaque_debug test:false 0.059
[RUSTC-TIMING] build_script_build test:false 0.472
[RUSTC-TIMING] build_script_build test:false 0.457
[RUSTC-TIMING] autocfg test:false 0.511

I don't know if this is going to be a useful issue, but I figured I would note this down as happening over the last day-ish.
For all I know it may have been cosmic rays.

[jfrimmel]

One more is #90041.

[nikic]

#90589 in x86_64-gnu-distcheck

[hkratz]

Going systematically through the bors CI failures of the last 24 hours:

Rustc segfaults:

• https://github.com/rust-lang-ci/rust/runs/4183298667?check_suite_focus=true
• https://github.com/rust-lang-ci/rust/runs/4181602467?check_suite_focus=true
• https://github.com/rust-lang-ci/rust/runs/4186005517?check_suite_focus=true
• https://github.com/rust-lang-ci/rust/runs/4185976885?check_suite_focus=true
• https://github.com/rust-lang-ci/rust/runs/4185303624?check_suite_focus=true
• https://github.com/rust-lang-ci/rust/runs/4187076557?check_suite_focus=true

8 other (inconspicuous) failures.

[workingjubilee]

Once again:
https://github.com/rust-lang-ci/rust/runs/4195144401?check_suite_focus=true

dist-armhf fail

2021-11-12T22:34:46.0890595Z �[0m�[0m�[1m�[32m   Compiling�[0m autocfg v1.0.0
2021-11-12T22:34:46.0892018Z �[0m�[0m�[1m�[32m   Compiling�[0m libc v0.2.106
2021-11-12T22:34:46.0893233Z �[0m�[0m�[1m�[32m   Compiling�[0m proc-macro2 v1.0.30
2021-11-12T22:34:46.0894476Z �[0m�[0m�[1m�[32m   Compiling�[0m unicode-xid v0.2.2
2021-11-12T22:34:46.0895652Z �[0m�[0m�[1m�[32m   Compiling�[0m cfg-if v0.1.10
2021-11-12T22:34:46.0897143Z �[0m�[0m�[1m�[32m   Compiling�[0m syn v1.0.80
2021-11-12T22:34:46.0898332Z �[0m�[0m�[1m�[32m   Compiling�[0m lazy_static v1.4.0
2021-11-12T22:34:46.0899500Z �[0m�[0m�[1m�[32m   Compiling�[0m cfg-if v1.0.0
2021-11-12T22:34:46.0900691Z �[0m�[0m�[1m�[32m   Compiling�[0m scopeguard v1.1.0
2021-11-12T22:34:46.0901841Z �[0m�[0m�[1m�[32m   Compiling�[0m log v0.4.14
2021-11-12T22:34:46.0960995Z �[0m�[0m�[1m�[32m   Compiling�[0m smallvec v1.7.0
2021-11-12T22:34:46.1008915Z �[0m�[0m�[1m�[32m   Compiling�[0m cc v1.0.69
2021-11-12T22:34:46.1017627Z �[0m�[0m�[1m�[32m   Compiling�[0m instant v0.1.6
2021-11-12T22:34:46.1039324Z �[0m�[0m�[1m�[32m   Compiling�[0m maybe-uninit v2.0.0
2021-11-12T22:34:46.1040354Z �[0m�[0m�[1m�[32m   Compiling�[0m getrandom v0.2.0
2021-11-12T22:34:46.1041268Z �[0m�[0m�[1m�[32m   Compiling�[0m hashbrown v0.11.0
2021-11-12T22:34:46.1398646Z [RUSTC-TIMING] cfg_if test:false 0.040
2021-11-12T22:34:46.1402728Z �[0m�[0m�[1m�[32m   Compiling�[0m bitflags v1.2.1
2021-11-12T22:34:46.1563110Z [RUSTC-TIMING] cfg_if test:false 0.043
2021-11-12T22:34:46.1568409Z �[0m�[0m�[1m�[32m   Compiling�[0m ppv-lite86 v0.2.8
2021-11-12T22:34:46.1621281Z [RUSTC-TIMING] scopeguard test:false 0.058
2021-11-12T22:34:46.1628826Z �[0m�[0m�[1m�[32m   Compiling�[0m pin-project-lite v0.2.4
2021-11-12T22:34:46.1813489Z [RUSTC-TIMING] lazy_static test:false 0.080
2021-11-12T22:34:46.1836993Z [RUSTC-TIMING] instant test:false 0.057
2021-11-12T22:34:46.1841694Z �[0m�[0m�[1m�[32m   Compiling�[0m typenum v1.12.0
2021-11-12T22:34:46.1844092Z �[0m�[0m�[1m�[32m   Compiling�[0m version_check v0.9.3
2021-11-12T22:34:46.2124837Z [RUSTC-TIMING] unicode_xid test:false 0.115
2021-11-12T22:34:46.2129790Z �[0m�[0m�[1m�[32m   Compiling�[0m either v1.6.0
2021-11-12T22:34:46.2448673Z [RUSTC-TIMING] pin_project_lite test:false 0.076
2021-11-12T22:34:46.2453805Z �[0m�[0m�[1m�[32m   Compiling�[0m rustc-hash v1.1.0
2021-11-12T22:34:46.3323728Z [RUSTC-TIMING] build_script_build test:false 0.221
2021-11-12T22:34:46.3327818Z rustc exited with signal: 11 (core dumped)
2021-11-12T22:34:46.3335183Z �[0m�[0m�[1m�[31merror�[0m�[1m:�[0m could not compile `getrandom`
2021-11-12T22:34:46.3336017Z 
2021-11-12T22:34:46.3337149Z Caused by:
2021-11-12T22:34:46.3344316Z   process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc --crate-name build_script_build --edition=2018 /cargo/registry/src/github.com-1ecc6299db9ec823/getrandom-0.2.0/build.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=0 -C debug-assertions=off --cfg 'feature="std"' -C metadata=301bf632479934c1 -C extra-filename=-301bf632479934c1 --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/release/build/getrandom-301bf632479934c1 -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-rustc/release/deps --cap-lints allow -Z binary-dep-depinfo` (exit status: 254)
2021-11-12T22:34:46.3348971Z �[0m�[0m�[1m�[33mwarning�[0m�[1m:�[0m build failed, waiting for other jobs to finish...
2021-11-12T22:34:46.3478946Z [RUSTC-TIMING] rustc_hash test:false 0.096
2021-11-12T22:34:46.3712607Z [RUSTC-TIMING] either test:false 0.153
2021-11-12T22:34:46.3963633Z [RUSTC-TIMING] smallvec test:false 0.288
2021-11-12T22:34:46.4509201Z [RUSTC-TIMING] autocfg test:false 0.355
2021-11-12T22:34:46.4656422Z [RUSTC-TIMING] build_script_build test:false 0.355
2021-11-12T22:34:46.4867457Z [RUSTC-TIMING] version_check test:false 0.294
2021-11-12T22:34:46.5465364Z [RUSTC-TIMING] build_script_build test:false 0.418
2021-11-12T22:34:46.5740029Z [RUSTC-TIMING] build_script_build test:false 0.477
2021-11-12T22:34:46.5827616Z [RUSTC-TIMING] build_script_build test:false 0.423
2021-11-12T22:34:46.5871691Z [RUSTC-TIMING] ppv_lite86 test:false 0.425
2021-11-12T22:34:46.5902842Z [RUSTC-TIMING] build_script_build test:false 0.494
2021-11-12T22:34:46.5997624Z [RUSTC-TIMING] build_script_build test:false 0.500
2021-11-12T22:34:46.6786739Z [RUSTC-TIMING] hashbrown test:false 0.568
2021-11-12T22:34:46.7561868Z [RUSTC-TIMING] build_script_main test:false 0.565
2021-11-12T22:34:46.9026757Z [RUSTC-TIMING] cc test:false 0.796
2021-11-12T22:34:46.9186028Z �[0m�[0m�[1m�[31merror�[0m�[1m:�[0m build failed
2021-11-12T22:34:46.9220757Z Build completed unsuccessfully in 0:02:51
2021-11-12T22:34:46.9343164Z == clock drift check ==
2021-11-12T22:34:46.9351870Z   local time: Fri Nov 12 22:34:46 UTC 2021
2021-11-12T22:34:47.2017603Z   network time: Thu, 11 Nov 2021 23:29:52 GMT
2021-11-12T22:34:47.2018813Z == end clock drift check ==
2021-11-12T22:34:49.3059588Z ##[error]Process completed with exit code 1.

[workingjubilee]

This is way too often for cosmic rays to be the answer at this point...
...unless GitHub's backing hosting doesn't believe in ECC RAM at all, in which case maybe. But they run Intel Xeons, so.

[ehuss]

#90881 (comment) failed building memchr for stage0 std (mingw-check).

I investigated a bit to see what, if anything has changed. stage0 rustc has not changed in a long while. The docker images haven't changed, either.

The first error appears to be in #89167 (https://github.com/rust-lang-ci/rust/actions/runs/1447510130) at around 2021-11-11T05:46.

The only thing that I've see that has changed recently is the outer GitHub Ubuntu 18 image was updated at 2021-11-09T11:16. The new image is https://github.com/actions/virtual-environments/releases/tag/ubuntu18%2F20211108.1. I don't see anything in there that would be suspicious. There's also about 43 hours of successful builds between the image being updated and the first SIGSEGV appearing. It would also be surprising that the outer image would introduce SIGSEGV.

[Mark-Simulacrum]

We can try backing out #89167, but I'd really rather not. It introduced very little code that's even getting compiled at stage0 though (since it has cfg(not(bootstrap)) on the whole module it adds, IIRC).

[RalfJung]

I am seeing this SIGSEGV locally. In fact I currently get it each time I run nice ./x.py test src/test/ui --test-args 89765 --bless (so I can't even run ui tests), even if I previously did ./x.py clean -- though that same command did still work a few hours ago, so I suspect some kind of race condition makes it non-deterministic whether a correct or a broken rustc is being produced.

/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-47d0b6818de97398.so(+0x73cac3)[0x7fd19f513ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x3cef0)[0x7fd19eaa0ef0]
/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-47d0b6818de97398.so(_RNvMs1Q_NtNtCseIS259k75xz_12rustc_middle2ty7contextNtB6_6TyCtxt13intern_layout+0x24)[0x7fd1a1501b84]
/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-47d0b6818de97398.so(+0xade075)[0x7fd19f8b5075]
[a lot of errors like this]
/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-47d0b6818de97398.so(+0xb851a4)[0x7fd19f95c1a4]
/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-47d0b6818de97398.so(+0xb9202e)[0x7fd19f96902e]
/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-47d0b6818de97398.so(+0xbd7c45)[0x7fd19f9aec45]
/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1/lib/librustc_driver-47d0b6818de97398.so(+0xb851a4)[0x7fd19f95c1a4]
rustc exited with signal: 11 (core dumped)
error: could not compile `core`

Caused by:
  process didn't exit successfully: `/home/r/src/rust/rustc/build/bootstrap/debug/rustc --crate-name core --edition=2018 library/core/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C codegen-units=8 -C debuginfo=1 -C metadata=d2ad3ad0cb571346 -C extra-filename=-d2ad3ad0cb571346 --out-dir /home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -C incremental=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/incremental -L dependency=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps -L dependency=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-std/release/deps -Zsymbol-mangling-version=legacy -Zmacro-backtrace '-Clink-args=-Wl,-rpath,$ORIGIN/../lib' -Zsave-analysis -Cprefer-dynamic -Cllvm-args=-import-instr-limit=10 -Cembed-bitcode=yes '-Zcrate-attr=doc(html_root_url="https://doc.rust-lang.org/nightly/")' -Z binary-dep-depinfo` (exit status: 254)
Build completed unsuccessfully in 0:00:32

EDIT: Now I got this even after doing rm build/x86_64-unknown-linux-gnu/ -rf, then ./x.py clean, and then nice ./x.py test src/test/ui --test-args 89765 --bless.

For the record, this is my config.toml with comments removed:

changelog-seen = 2

[llvm]
download-ci-llvm = true
ninja = true

[build]
submodules = false
extended = true
tools = ["miri"]

[rust]
codegen-units = 0
debug-assertions = true
debug-assertions-std = false
debuginfo-level = 1
incremental = true
deny-warnings = false

The error seems to happen every single time when I build RalfJung@4869098d160 or RalfJung@a125e39ac4b. When I try master (1b12d01), that works fine. The problematic commit also fails on CI.

[RalfJung]

@ehuss points out that my PR 'just' introduces infinite recursion, leading to a stack overflow. I should have thought of that earlier.^^

[Mark-Simulacrum]

Running under valgrind with the stage0 compiler, I am seeing a bunch of the following (or similar) on thread_local compilation as part of bootstrap (due to #90812 (comment))

==624266== Mismatched free() / delete / delete []
==624266==    at 0x4841E81: operator delete(void*) (vg_replace_malloc.c:923)
==624266==    by 0x6CBD81D: rustc_codegen_ssa::ModuleCodegen<M>::into_compiled_module (in /home/mark/Build/rust/obj/build/x86_64-unknown-linux-gnu/stage0/lib/librustc_driver-24f36aeef90a945b.so)
==624266==    by 0x6D08862: rustc_codegen_llvm::back::write::codegen (in /home/mark/Build/rust/obj/build/x86_64-unknown-linux-gnu/stage0/lib/librustc_driver-24f36aeef90a945b.so)
==624266==    by 0x6C9BF1A: rustc_codegen_ssa::back::write::finish_intra_module_work (in /home/mark/Build/rust/obj/build/x86_64-unknown-linux-gnu/stage0/lib/librustc_driver-24f36aeef90a945b.so)
==624266==    by 0x6C958AA: rustc_codegen_ssa::back::write::execute_work_item (in /home/mark/Build/rust/obj/build/x86_64-unknown-linux-gnu/stage0/lib/librustc_driver-24f36aeef90a945b.so)
==624266==    by 0x6CC4FBF: std::sys_common::backtrace::__rust_begin_short_backtrace (in /home/mark/Build/rust/obj/build/x86_64-unknown-linux-gnu/stage0/lib/librustc_driver-24f36aeef90a945b.so)
==624266==    by 0x6CEFE4B: core::ops::function::FnOnce::call_once{{vtable.shim}} (in /home/mark/Build/rust/obj/build/x86_64-unknown-linux-gnu/stage0/lib/librustc_driver-24f36aeef90a945b.so)
==624266==    by 0x8753852: std::sys::unix::thread::Thread::new::thread_start (boxed.rs:1691)
==624266==    by 0x8A32608: start_thread (pthread_create.c:477)
==624266==    by 0x8B81292: clone (clone.S:95)
==624266==  Address 0x1625b6e0 is 0 bytes inside a block of size 8 alloc'd
==624266==    at 0x483F085: malloc (vg_replace_malloc.c:381)
==624266==    by 0x6D246A7: operator new(unsigned long) (new_op.cc:50)
==624266==    by 0x6D1B9CE: LLVMRustContextCreate (in /home/mark/Build/rust/obj/build/x86_64-unknown-linux-gnu/stage0/lib/librustc_driver-24f36aeef90a945b.so)
==624266==    by 0x5FE1046: rustc_codegen_llvm::base::compile_codegen_unit::module_codegen (in /home/mark/Build/rust/obj/build/x86_64-unknown-linux-gnu/stage0/lib/librustc_driver-24f36aeef90a945b.so)
==624266==    by 0x6CB71D4: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task (in /home/mark/Build/rust/obj/build/x86_64-unknown-linux-gnu/stage0/lib/librustc_driver-24f36aeef90a945b.so)
==624266==    by 0x6CED485: rustc_codegen_llvm::base::compile_codegen_unit (in /home/mark/Build/rust/obj/build/x86_64-unknown-linux-gnu/stage0/lib/librustc_driver-24f36aeef90a945b.so)
==624266==    by 0x6CC263E: rustc_codegen_ssa::base::codegen_crate (in /home/mark/Build/rust/obj/build/x86_64-unknown-linux-gnu/stage0/lib/librustc_driver-24f36aeef90a945b.so)
==624266==    by 0x6CE2F89: <rustc_codegen_llvm::LlvmCodegenBackend as rustc_codegen_ssa::traits::backend::CodegenBackend>::codegen_crate (in /home/mark/Build/rust/obj/build/x86_64-unknown-linux-gnu/stage0/lib/librustc_driver-24f36aeef90a945b.so)
==624266==    by 0x6C889D6: rustc_session::utils::<impl rustc_session::session::Session>::time (in /home/mark/Build/rust/obj/build/x86_64-unknown-linux-gnu/stage0/lib/librustc_driver-24f36aeef90a945b.so)
==624266==    by 0x6C66D58: rustc_interface::queries::Queries::ongoing_codegen (in /home/mark/Build/rust/obj/build/x86_64-unknown-linux-gnu/stage0/lib/librustc_driver-24f36aeef90a945b.so)
==624266==    by 0x6C45D26: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter (in /home/mark/Build/rust/obj/build/x86_64-unknown-linux-gnu/stage0/lib/librustc_driver-24f36aeef90a945b.so)
==624266==    by 0x6C3641E: rustc_span::with_source_map (in /home/mark/Build/rust/obj/build/x86_64-unknown-linux-gnu/stage0/lib/librustc_driver-24f36aeef90a945b.so)