Cannot bootstrap rust with system compiler and customized libdir from 1.85 (stage0-sysroot)

[lucabonnix]

Summary

Starting from rust 1.85, if using the system compiler, the boostrap system seems no more able to identify "libdir" of the compiler, so, if it is different than "lib" (e.g. "lib64"), the system compiler cannot find "core" and/or "std" crates while "Building compiler artifacts (stage0 -> stage1, x86_64-unknown-linux-gnu)"

Command used

python3 x.py build

Expected behaviour

We expect to find "rustlib" in the directory "stage0-sysroot/lib64"

Actual behaviour

"rustlib" is wrongly placed in the directory "stage0-sysroot/lib"

Bootstrap configuration (bootstrap.toml)

# Avoid triggering "git --local" on non-git vendored sources:
profile = "dist"
 
[llvm]
link-shared = true
download-ci-llvm = false
 
[build]
build = "x86_64-unknown-linux-gnu"
host = ["x86_64-unknown-linux-gnu"]
target = ["x86_64-unknown-linux-gnu"]
tools = ["analysis", "cargo", "clippy", "rls", "rustfmt", "src", "rust-analyzer", "rust-demangler"]
submodules = false
vendor = true
extended = true
rustc = "/usr/bin/rustc"
cargo = "/usr/bin/cargo"
profiler = true
sanitizers = true
# Do not query new versions of dependencies online.
locked-deps = true
 
jobs =  16
 
[install]
prefix = "/usr"
docdir = "doc/rust-1.85.0"
libdir = "lib64"
mandir = "man"
 
[rust]
codegen-units = 0
channel = "stable"
rpath = false
codegen-tests = false
 
# Add this stuff to build with the system LLVM:
[target.i586-unknown-linux-gnu]
llvm-config = "/usr/bin/llvm-config"
 
[target.i686-unknown-linux-gnu]
llvm-config = "/usr/bin/llvm-config"
 
[target.x86_64-unknown-linux-gnu]
llvm-config = "/usr/bin/llvm-config"
 
[target.arm-unknown-linux-gnueabi]
llvm-config = "/usr/bin/llvm-config"
 
[target.armv5te-unknown-linux-gnueabi]
llvm-config = "/usr/bin/llvm-config"
 
[target.armv7-unknown-linux-gnueabihf]
llvm-config = "/usr/bin/llvm-config"
 
[target.aarch64-unknown-linux-gnu]
llvm-config = "/usr/bin/llvm-config"
 
[target.powerpc-unknown-linux-gnu]
llvm-config = "/usr/bin/llvm-config"
 
[target.powerpc64-unknown-linux-gnu]
llvm-config = "/usr/bin/llvm-config"
 
[target.powerpc64le-unknown-linux-gnu]
llvm-config = "/usr/bin/llvm-config"
 
[target.mips-unknown-linux-gnu]
llvm-config = "/usr/bin/llvm-config"
 
[target.mipsel-unknown-linux-gnu]
llvm-config = "/usr/bin/llvm-config"
 
[target.mips64el-unknown-linux-gnuabi64]
llvm-config = "/usr/bin/llvm-config"
 
[target.riscv64gc-unknown-linux-gnu]
llvm-config = "/usr/bin/llvm-config"
 
[target.s390x-unknown-linux-gnu]
llvm-config = "/usr/bin/llvm-config"
 
[target.loongarch64-unknown-linux-gnu]
llvm-config = "/usr/bin/llvm-config"

Operating system

Slackware 15.0 / Slackware current (developing branch 15.1)

HEAD

4d91de4

Additional context

If the system compiler was built with standard libdir = "lib", all works fine.

Build Log

Building bootstrap
   Compiling proc-macro2 v1.0.89
   Compiling memchr v2.7.4
   Compiling unicode-ident v1.0.13
   Compiling version_check v0.9.5
   Compiling typenum v1.17.0
   Compiling libc v0.2.167
   Compiling shlex v1.3.0
   Compiling serde v1.0.215
   Compiling rustix v0.38.40
   Compiling crossbeam-utils v0.8.20
   Compiling regex-syntax v0.8.5
   Compiling bitflags v2.6.0
   Compiling linux-raw-sys v0.4.14
   Compiling pkg-config v0.3.31
   Compiling cfg-if v1.0.0
   Compiling anstyle v1.0.10
   Compiling clap_lex v0.7.2
   Compiling cc v1.2.0
   Compiling heck v0.5.0
   Compiling log v0.4.22
   Compiling serde_json v1.0.132
   Compiling clap_builder v4.5.20
   Compiling generic-array v0.14.7
   Compiling semver v1.0.23
   Compiling same-file v1.0.6
   Compiling itoa v1.0.11
   Compiling bootstrap v0.0.0 (/root/tmp/rustc-1.85.0-src/src/bootstrap)
   Compiling walkdir v2.5.0
   Compiling aho-corasick v1.1.3
   Compiling cpufeatures v0.2.15
   Compiling ryu v1.0.18
   Compiling object v0.36.5
   Compiling home v0.5.9
   Compiling termcolor v1.4.1
   Compiling crossbeam-epoch v0.9.18
   Compiling filetime v0.2.25
   Compiling cmake v0.1.48
   Compiling quote v1.0.37
   Compiling crossbeam-deque v0.8.5
   Compiling syn v2.0.87
   Compiling lzma-sys v0.1.20
   Compiling xz2 v0.1.7
   Compiling block-buffer v0.10.4
   Compiling crypto-common v0.1.6
   Compiling digest v0.10.7
   Compiling regex-automata v0.4.9
   Compiling xattr v1.3.1
   Compiling fd-lock v4.0.2
   Compiling sha2 v0.10.8
   Compiling tar v0.4.43
   Compiling toml v0.5.11
   Compiling bstr v1.10.0
   Compiling clap_derive v4.5.18
   Compiling serde_derive v1.0.215
   Compiling globset v0.4.15
   Compiling opener v0.5.2
   Compiling ignore v0.4.23
   Compiling clap v4.5.20
   Compiling clap_complete v4.5.37
   Compiling build_helper v0.1.0 (/root/tmp/rustc-1.85.0-src/src/build_helper)
    Finished `dev` profile [unoptimized] target(s) in 31.10s
WARNING: The `change-id` is missing in the `config.toml`. This means that you will not be able to track the major changes made to the bootstrap configurations.
NOTE: to silence this warning, add `change-id = 134650` at the top of `config.toml`
Building stage0 library artifacts (x86_64-unknown-linux-gnu)
   Compiling shlex v1.3.0
   Compiling core v0.0.0 (/root/tmp/rustc-1.85.0-src/library/core)
   Compiling libc v0.2.169
   Compiling object v0.36.7
   Compiling std v0.0.0 (/root/tmp/rustc-1.85.0-src/library/std)
   Compiling cc v1.2.0
   Compiling compiler_builtins v0.1.140
   Compiling profiler_builtins v0.0.0 (/root/tmp/rustc-1.85.0-src/library/profiler_builtins)
   Compiling rustc-std-workspace-core v1.99.0 (/root/tmp/rustc-1.85.0-src/library/rustc-std-workspace-core)
   Compiling alloc v0.0.0 (/root/tmp/rustc-1.85.0-src/library/alloc)
   Compiling cfg-if v1.0.0
   Compiling memchr v2.7.4
   Compiling adler v1.0.2
   Compiling rustc-demangle v0.1.24
   Compiling unwind v0.0.0 (/root/tmp/rustc-1.85.0-src/library/unwind)
   Compiling rustc-std-workspace-alloc v1.99.0 (/root/tmp/rustc-1.85.0-src/library/rustc-std-workspace-alloc)
   Compiling panic_abort v0.0.0 (/root/tmp/rustc-1.85.0-src/library/panic_abort)
   Compiling panic_unwind v0.0.0 (/root/tmp/rustc-1.85.0-src/library/panic_unwind)
   Compiling gimli v0.29.0
   Compiling std_detect v0.1.5 (/root/tmp/rustc-1.85.0-src/library/stdarch/crates/std_detect)
   Compiling miniz_oxide v0.7.4
   Compiling hashbrown v0.15.2
   Compiling addr2line v0.22.0
   Compiling rustc-std-workspace-std v1.99.0 (/root/tmp/rustc-1.85.0-src/library/rustc-std-workspace-std)
   Compiling proc_macro v0.0.0 (/root/tmp/rustc-1.85.0-src/library/proc_macro)
   Compiling unicode-width v0.1.14
   Compiling getopts v0.2.21
   Compiling test v0.0.0 (/root/tmp/rustc-1.85.0-src/library/test)
   Compiling sysroot v0.0.0 (/root/tmp/rustc-1.85.0-src/library/sysroot)
    Finished `release` profile [optimized] target(s) in 38.97s
Building compiler artifacts (stage0 -> stage1, x86_64-unknown-linux-gnu)
   Compiling proc-macro2 v1.0.92
   Compiling unicode-ident v1.0.14
   Compiling libc v0.2.169
   Compiling cfg-if v1.0.0
   Compiling stable_deref_trait v1.2.0
   Compiling crossbeam-utils v0.8.21
   Compiling shlex v1.3.0
   Compiling version_check v0.9.5
   Compiling smallvec v1.13.2
   Compiling autocfg v1.4.0
   Compiling once_cell v1.20.2
   Compiling either v1.13.0
   Compiling bitflags v2.6.0
   Compiling memchr v2.7.4
   Compiling foldhash v0.1.4
   Compiling equivalent v1.0.1
error[E0463]: can't find crate for `core`
  |
  = note: the `x86_64-unknown-linux-gnu` target may not be installed
  = help: consider downloading the target with `rustup target add x86_64-unknown-linux-gnu`
  = help: consider building the standard library from source with `cargo build -Zbuild-std`

For more information about this error, try `rustc --explain E0463`.
error: could not compile `cfg-if` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...
error: could not compile `smallvec` (lib) due to 1 previous error
error[E0463]: can't find crate for `std`
  |
  = note: the `x86_64-unknown-linux-gnu` target may not be installed
  = help: consider downloading the target with `rustup target add x86_64-unknown-linux-gnu`
  = help: consider building the standard library from source with `cargo build -Zbuild-std`

error: could not compile `once_cell` (lib) due to 1 previous error
error: could not compile `bitflags` (lib) due to 1 previous error
error: could not compile `either` (lib) due to 1 previous error
error: could not compile `stable_deref_trait` (lib) due to 1 previous error
error: could not compile `equivalent` (lib) due to 1 previous error
error: could not compile `foldhash` (lib) due to 1 previous error
error: could not compile `memchr` (lib) due to 1 previous error
Build completed unsuccessfully in 0:01:11

[jieyouxu]

This libdir calculation is wrong I think

rust/src/bootstrap/src/utils/helpers.rs

Lines 130 to 134 in 38081f2

	/// Returns the corresponding relative library directory that the compiler's
	/// dylibs will be found in.
	pub fn libdir(target: TargetSelection) -> &'static str {
	if target.is_windows() || target.contains("cygwin") { "bin" } else { "lib" }
	}

when used for

rust/src/bootstrap/src/core/build_steps/compile.rs

Line 1305 in 38081f2

let libdir_relative = builder.config.libdir_relative().unwrap_or_else(|| Path::new("lib"));

? I'll need to double-check.

[jieyouxu]

We seem to be setting config.libdir from install.libdir

rust/src/bootstrap/src/core/config/config.rs

Line 1913 in 38081f2

config.libdir = libdir.map(PathBuf::from);

Is a workaround to set install.libdir = "lib" feasible for you?

[onur-ozkan]

This could be related as well:

rust/src/bootstrap/src/lib.rs

Lines 366 to 367 in 38081f2

	let initial_target_libdir_str =
	config.initial_sysroot.join("lib/rustlib").join(config.build).join("lib");

In fact it def. should be related as they don't have lib in their system rust toolchain.

[jieyouxu]

Yeah, I think there's several places that straight up explicitly or implicitly assume lib is the only possible libdir name

[lucabonnix]

We seem to be setting config.libdir from install.libdir

rust/src/bootstrap/src/core/config/config.rs

Line 1913 in 38081f2
config.libdir = libdir.map(PathBuf::from);

It seems that it works only for stage1 and stage2, but not for stage0.

Is a workaround to set install.libdir = "lib" feasible for you?

No, it will not work, because I need to set libdir = "lib64" in stage0-sysroot.

[onur-ozkan]

Yeah, I think there's several places that straight up explicitly or implicitly assume lib is the only possible libdir name

I think all we need is to update the logic I linked above, the rest will be handled properly as we always look and copy on lib unconditionally.

[onur-ozkan]

Can you replace lib with lib64 here and try again? @lucabonnix

[onur-ozkan]

Please test #141729 once you have time and let us know if that fixes the problem you get.

[lucabonnix]

Please test #141729 once you have time and let us know if that fixes the problem you get.

All works fine with this patch, with both system and pre-compiled (downloaded one) rustc.

Many thanks!!