rust_extern_with_linkage___dso_handle+0x0): undefined reference to `__dso_handle'

[tkennedy1-godaddy]

I tried this code:

Trying to cross compile (from aarch64 to x86_86 on linux)

I expected to see this happen: explanation

The software compiled correctly.

Instead, this happened: explanation

The linker failed on trying to link Rust's libstd.

  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: /usr/x86_64-linux-gnu/bin/ld: warning: cannot find entry symbol _start; defaulting to 0000000000005030
          /usr/x86_64-linux-gnu/bin/ld: /home/debian/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-f6265b21db1f990f.rlib(std-f6265b21db1f990f.std.d9e466a2d75004a2-cgu.0.rcgu.o):(.data._
rust_extern_with_linkage___dso_handle+0x0): undefined reference to `__dso_handle'
          /usr/x86_64-linux-gnu/bin/ld: /home/debian/envoy-cli/target/x86_64-unknown-linux-gnu/debug/deps/oktaplz-a2cfe24976663532: hidden symbol `__dso_handle' isn't defined
          /usr/x86_64-linux-gnu/bin/ld: final link failed: bad value

Meta

rustc --version --verbose:

1.81

Backtrace

<backtrace>

This does work correctly on rust 1.74.0 on the same system. I have been unable to test other versions yet, nor create a self-contained example.

The build command for this is:

RUSTFLAGS="-Clinker=/usr/x86_64-linux-gnu/bin/ld -L/lib/gcc-cross/x86_64-linux-gnu/12 -L/lib/gcc-cross/x86_64-linux-gnu/11" PKG_CONFIG_SYSROOT_DIR=/usr/x86_64-linux-gnu CC=clang cargo build --target x86_64-unknown-linux-gnu

[tkennedy1-godaddy]

The list of dependencies for the project are:

[dependencies]
base64 = "0.21"
chrono = "0.4"
color-eyre = "0.6"
ctrlc = "3.1"
dirs = "3.0"
futures = "0.3"
futures-channel = "0.3"
keyring = "0.10"
open = "1.4"
otpcli-tkennedy1 = "1.0"
promptly = "0.3"
rpassword = "5.0"
rusoto_core = "0.47"
rusoto_sts = "0.47"
rusoto_iam = "0.47"
rusoto_ssm = "0.47"
rusoto_secretsmanager = "0.47"
select = "0.6"
sentry = "0.23"
sentry-slog = "0.23"
serde-xml-rs = "0"
serde_json = "1.0"
skim = "0.10"
slog = "2.7"
slog-stdlog = "4.0"
slog-term = "2.7"
toml = "0.5"
url = "2"
warp = "0.3"
indicatif = "0.16"
rust-ini = "0.21.0"

[mati865]

You didn't provide crt objects to the linker. When calling linker directly you have to take care of it.

[tkennedy1-godaddy]

I didn’t on the version that compiled. Same exact build command, same dependencies installed. Builds on 1.74, fails on 1.81.

What would be the proper library to include on the linker for version 1.81?

[moxian]

@tkennedy1-godaddy would you like to run cargo bisect-rustc to narrow down the rustc commit range that intoduced this change in behavior? (I don't know much about CRT myself, but bisected PR description is often insightful in these cases, in my experience)

@rustbot label: +A-linkage +A-cross

[tkennedy1-godaddy]

@moxian yes i can get to that this afternoon pdt

[tkennedy1-godaddy]

@moxian

searched toolchains nightly-2023-10-01 through nightly-2025-05-13


********************************************************************************
Regression in nightly-2024-03-18
********************************************************************************

fetching https://static.rust-lang.org/dist/2024-03-17/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2024-03-17: 40 B / 40 B [===============================================================================================================================================================================] 100.00 % 17.86 KB/s
converted 2024-03-17 to 766bdce744d531267d53ba2a3f9ffcda69fb9b17
fetching https://static.rust-lang.org/dist/2024-03-18/channel-rust-nightly-git-commit-hash.txt
nightly manifest 2024-03-18: 40 B / 40 B [==============================================================================================================================================================================] 100.00 % 250.13 KB/s
converted 2024-03-18 to eb45c844407968ea54df0d9870ebce9e3235b706
looking for regression commit between 2024-03-17 and 2024-03-18
fetching (via remote github) commits from max(766bdce744d531267d53ba2a3f9ffcda69fb9b17, 2024-03-15) to eb45c844407968ea54df0d9870ebce9e3235b706
ending github query because we found starting sha: 766bdce744d531267d53ba2a3f9ffcda69fb9b17
get_commits_between returning commits, len: 10
  commit[0] 2024-03-16: Auto merge of #122602 - ChrisDenton:rollup-f0vumtg, r=ChrisDenton
  commit[1] 2024-03-16: Auto merge of #122594 - Mark-Simulacrum:bump-version, r=Mark-Simulacrum
  commit[2] 2024-03-17: Auto merge of #122607 - fmease:rollup-ozl1eeq, r=fmease
  commit[3] 2024-03-17: Auto merge of #121885 - reitermarkus:generic-nonzero-inner, r=oli-obk,wesleywiser
  commit[4] 2024-03-17: Auto merge of #122321 - majaha:mingw_ci_new, r=Mark-Simulacrum
  commit[5] 2024-03-17: Auto merge of #122611 - rust-lang:cargo_update, r=clubby789
  commit[6] 2024-03-17: Auto merge of #122625 - matthiaskrgr:rollup-ue4dmnx, r=matthiaskrgr
  commit[7] 2024-03-17: Auto merge of #122628 - lnicola:sync-from-ra, r=lnicola
  commit[8] 2024-03-17: Auto merge of #122637 - matthiaskrgr:rollup-bczj5bp, r=matthiaskrgr
  commit[9] 2024-03-17: Auto merge of #122653 - matthiaskrgr:rollup-28h37ym, r=matthiaskrgr
ERROR: no CI builds available between 766bdce744d531267d53ba2a3f9ffcda69fb9b17 and eb45c844407968ea54df0d9870ebce9e3235b706 within last 167 days

[tkennedy1-godaddy]

Amazingly this build system was finalized in Feb 2024, so it looks like I really dodged a bullet there :)

[mati865]

I didn’t on the version that compiled. Same exact build command, same dependencies installed. Builds on 1.74, fails on 1.81.

I think something else must have pulled the necessary objects.
The fact it worked doesn't imply it was supported.

What would be the proper library to include on the linker for version 1.81?

Just as with C.
Try to link C executable but pass -### to GCC or Clang, it'll print all the libraries and objects that are required. Rust should pull required libraries via https://github.com/rust-lang/libc/blob/a1e464928440edefb96f1d3131210a98e0e9eb54/src/unix/mod.rs#L465-L470, but you'll have to supply the objects (crt*.o) to the linker.

[tkennedy1-godaddy]

The crt*.0 libraries are in the path being provided to the linker though.

debian@debian:~/envoy-cli$ ls /lib/gcc-cross/x86_64-linux-gnu/12/*crt*
/lib/gcc-cross/x86_64-linux-gnu/12/crtbegin.o   /lib/gcc-cross/x86_64-linux-gnu/12/crtend.o       /lib/gcc-cross/x86_64-linux-gnu/12/crtoffloadbegin.o  /lib/gcc-cross/x86_64-linux-gnu/12/crtprec32.o
/lib/gcc-cross/x86_64-linux-gnu/12/crtbeginS.o  /lib/gcc-cross/x86_64-linux-gnu/12/crtendS.o      /lib/gcc-cross/x86_64-linux-gnu/12/crtoffloadend.o    /lib/gcc-cross/x86_64-linux-gnu/12/crtprec64.o
/lib/gcc-cross/x86_64-linux-gnu/12/crtbeginT.o  /lib/gcc-cross/x86_64-linux-gnu/12/crtfastmath.o  /lib/gcc-cross/x86_64-linux-gnu/12/crtoffloadtable.o  /lib/gcc-cross/x86_64-linux-gnu/12/crtprec80.o
debian@debian:~/envoy-cli$ ls /lib/gcc-cross/x86_64-linux-gnu/11/*crt*
/lib/gcc-cross/x86_64-linux-gnu/11/crtbegin.o   /lib/gcc-cross/x86_64-linux-gnu/11/crtend.o       /lib/gcc-cross/x86_64-linux-gnu/11/crtoffloadbegin.o  /lib/gcc-cross/x86_64-linux-gnu/11/crtprec32.o
/lib/gcc-cross/x86_64-linux-gnu/11/crtbeginS.o  /lib/gcc-cross/x86_64-linux-gnu/11/crtendS.o      /lib/gcc-cross/x86_64-linux-gnu/11/crtoffloadend.o    /lib/gcc-cross/x86_64-linux-gnu/11/crtprec64.o
/lib/gcc-cross/x86_64-linux-gnu/11/crtbeginT.o  /lib/gcc-cross/x86_64-linux-gnu/11/crtfastmath.o  /lib/gcc-cross/x86_64-linux-gnu/11/crtoffloadtable.o  /lib/gcc-cross/x86_64-linux-gnu/11/crtprec80.o

These paths are passed in via the -L/lib/gcc-cross/x86_64-linux-gnu/12 -L/lib/gcc-cross/x86_64-linux-gnu/11 parts of the RUSTFLAGS environment variable.

And more importantly, if I leave them out, the linker is unable to link to gcc_s, which is is one of the libraries located in those paths. Here's the error if I omit those -L flags:

  = note: /usr/x86_64-linux-gnu/bin/ld: cannot find -lgcc_s: No such file or directory

And here's the library it's looking for.

debian@debian:~/envoy-cli$ ls /lib/gcc-cross/x86_64-linux-gnu/*/libgcc_s.so
/lib/gcc-cross/x86_64-linux-gnu/11/libgcc_s.so  /lib/gcc-cross/x86_64-linux-gnu/12/libgcc_s.so

The fact it worked doesn't imply it was supported.

The fact that we're referring to __dso_handle, which is both present in the all versions of the stdlib:

debian@debian:~/.rustup/toolchains$ for LIB in $(find -type f -name *libstd-*rlib); do echo $LIB; readelf -s $LIB | grep __dso_handle; done
./1.74-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-84a09a48c3c6b54d.rlib
  7808: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./1.74-aarch64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-b149a04e58514815.rlib
  5020: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./1.69-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-946b15357ac77df4.rlib
  7520: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./1.61-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-7d5a620ab77655ab.rlib
  7663: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-d99d33bfbb8d1b5c.rlib
  6953: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./stable-aarch64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-f6265b21db1f990f.rlib
  4486: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./1.81-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-2bf0b2a5e0a60917.rlib
  6475: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./1.81-aarch64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-1c4b19562077c20d.rlib
  4189: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./1.67-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-349359eac2fa563a.rlib
  7659: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./bisector-nightly-2025-04-07-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libstd-998e3cd732077b76.rlib
  6919: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle
./bisector-nightly-2025-04-07-aarch64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-77b6a10324684199.rlib
  4462: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __dso_handle

And has been an issue in the past, specifically in Rust makes me think something else might be going on here other than "it wasn't supported and now it's not supported at all".

It might be a "it wasn't supposed to be supported and so it was an accident that it was supported" type thing, but given that even providing the correct libraries still causes this to break and the information in that gist, I'm not thoroughly convinced.