macos: unable to get debug information from dylib dependencies

[ehuss]

On macOS, the debug information for the dependencies of a dylib do not seem to be working with split-debuginfo=unpacked.

One way to demonstrate this is building rustc itself with debug enabled, and split-debuginfo as unpacked (the default). Backtraces are unable to find the source file information for anything past rustc_query_system. lldb is also unable to find any debug information (it can't even find the symbols).

Example of what the backtrace looks like:

Backtrace

   0: std::panicking::begin_panic
   1: rustc_errors::HandlerInner::emit_diagnostic
   2: rustc_errors::Handler::emit_diagnostic
   3: rustc_errors::diagnostic_builder::DiagnosticBuilder::emit
   4: rustc_resolve::late::lifetimes::LifetimeContext::resolve_lifetime_ref
   5: <rustc_resolve::late::lifetimes::LifetimeContext as rustc_hir::intravisit::Visitor>::visit_lifetime
   6: <rustc_resolve::late::lifetimes::LifetimeContext as rustc_hir::intravisit::Visitor>::visit_path
   7: rustc_hir::intravisit::walk_expr
   8: rustc_resolve::late::lifetimes::LifetimeContext::with
   9: <rustc_resolve::late::lifetimes::LifetimeContext as rustc_hir::intravisit::Visitor>::visit_nested_body
  10: rustc_resolve::late::lifetimes::LifetimeContext::visit_early_late
  11: <rustc_resolve::late::lifetimes::LifetimeContext as rustc_hir::intravisit::Visitor>::visit_item
  12: rustc_resolve::late::lifetimes::do_resolve
  13: rustc_resolve::late::lifetimes::resolve_lifetimes
  14: rustc_query_impl::<impl rustc_query_system::query::config::QueryAccessors<rustc_query_impl::plumbing::QueryCtxt> for rustc_query_impl::queries::resolve_lifetimes>::compute
  15: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl
  16: rustc_data_structures::stack::ensure_sufficient_stack
  17: rustc_query_system::query::plumbing::force_query_with_job
  18: rustc_query_system::query::plumbing::get_query_impl
  19: rustc_query_system::query::plumbing::get_query
  20: rustc_resolve::late::lifetimes::resolve_lifetimes_for
  21: core::ops::function::FnOnce::call_once
  22: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl
  23: rustc_data_structures::stack::ensure_sufficient_stack
  24: rustc_query_system::query::plumbing::force_query_with_job
  25: rustc_query_system::query::plumbing::get_query_impl
  26: rustc_query_system::query::plumbing::get_query
  27: rustc_middle::ty::context::TyCtxt::named_region
  28: <dyn rustc_typeck::astconv::AstConv>::ast_region_to_region
  29: <<dyn rustc_typeck::astconv::AstConv>::create_substs_for_ast_path::SubstsForAstPathCtxt as rustc_typeck::astconv::CreateSubstsForGenericArgsCtxt>::provided_kind
  30: rustc_typeck::astconv::generics::<impl dyn rustc_typeck::astconv::AstConv>::create_substs_for_generic_args
  31: <dyn rustc_typeck::astconv::AstConv>::create_substs_for_ast_path
  32: <dyn rustc_typeck::astconv::AstConv>::ast_path_substs_for_ty
  33: <dyn rustc_typeck::astconv::AstConv>::ast_path_to_ty
  34: <dyn rustc_typeck::astconv::AstConv>::res_to_ty
  35: rustc_typeck::check::fn_ctxt::checks::<impl rustc_typeck::check::fn_ctxt::FnCtxt>::check_struct_path
  36: rustc_typeck::check::expr::<impl rustc_typeck::check::fn_ctxt::FnCtxt>::check_expr_kind
  37: rustc_typeck::check::expr::<impl rustc_typeck::check::fn_ctxt::FnCtxt>::check_expr_with_expectation
  38: rustc_typeck::check::fn_ctxt::_impl::<impl rustc_typeck::check::fn_ctxt::FnCtxt>::with_breakable_ctxt
  39: rustc_typeck::check::fn_ctxt::checks::<impl rustc_typeck::check::fn_ctxt::FnCtxt>::check_block_with_expected
  40: rustc_typeck::check::expr::<impl rustc_typeck::check::fn_ctxt::FnCtxt>::check_expr_kind
  41: rustc_typeck::check::expr::<impl rustc_typeck::check::fn_ctxt::FnCtxt>::check_expr_with_expectation
  42: rustc_typeck::check::expr::<impl rustc_typeck::check::fn_ctxt::FnCtxt>::check_return_expr
  43: rustc_typeck::check::check::check_fn
  44: rustc_infer::infer::InferCtxtBuilder::enter
  45: rustc_typeck::check::inherited::InheritedBuilder::enter
  46: rustc_typeck::check::typeck_with_fallback
  47: rustc_typeck::check::typeck
  48: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl
  49: rustc_data_structures::stack::ensure_sufficient_stack
  50: rustc_query_system::query::plumbing::force_query_with_job
  51: rustc_query_system::query::plumbing::get_query_impl
  52: rustc_query_system::query::plumbing::get_query
  53: rustc_middle::ty::<impl rustc_middle::ty::context::TyCtxt>::par_body_owners
  54: rustc_typeck::check::typeck_item_bodies
  55: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl
  56: rustc_data_structures::stack::ensure_sufficient_stack
  57: rustc_query_system::query::plumbing::force_query_with_job
  58: rustc_query_system::query::plumbing::get_query_impl
  59: rustc_query_system::query::plumbing::get_query
  60: rustc_session::utils::<impl rustc_session::session::Session>::time
  61: rustc_typeck::check_crate
  62: rustc_interface::passes::analysis
  63: rustc_query_system::dep_graph::graph::DepGraph<K>::with_task_impl
  64: rustc_query_system::dep_graph::graph::DepGraph<K>::with_eval_always_task
  65: rustc_data_structures::stack::ensure_sufficient_stack
  66: rustc_query_system::query::plumbing::force_query_with_job
  67: rustc_query_system::query::plumbing::get_query_impl
  68: rustc_query_system::query::plumbing::get_query
  69: rustc_middle::ty::query::TyCtxtAt::analysis
             at /Users/eric/Proj/rust/rust2/compiler/rustc_middle/src/ty/query/mod.rs:205:17
  70: rustc_middle::ty::query::<impl rustc_middle::ty::context::TyCtxt>::analysis
             at /Users/eric/Proj/rust/rust2/compiler/rustc_middle/src/ty/query/mod.rs:186:17
  71: rustc_driver::run_compiler::{{closure}}::{{closure}}::{{closure}}
             at /Users/eric/Proj/rust/rust2/compiler/rustc_driver/src/lib.rs:436:59
  72: rustc_interface::passes::QueryContext::enter::{{closure}}
             at /Users/eric/Proj/rust/rust2/compiler/rustc_interface/src/passes.rs:754:42
  73: rustc_middle::ty::context::tls::enter_context::{{closure}}
             at /Users/eric/Proj/rust/rust2/compiler/rustc_middle/src/ty/context.rs:1726:50
  74: rustc_middle::ty::context::tls::set_tlv
             at /Users/eric/Proj/rust/rust2/compiler/rustc_middle/src/ty/context.rs:1710:9
  75: rustc_middle::ty::context::tls::enter_context
             at /Users/eric/Proj/rust/rust2/compiler/rustc_middle/src/ty/context.rs:1726:9
  76: rustc_interface::passes::QueryContext::enter
             at /Users/eric/Proj/rust/rust2/compiler/rustc_interface/src/passes.rs:754:9
  77: rustc_driver::run_compiler::{{closure}}::{{closure}}
             at /Users/eric/Proj/rust/rust2/compiler/rustc_driver/src/lib.rs:436:13
  78: rustc_interface::queries::<impl rustc_interface::interface::Compiler>::enter
             at /Users/eric/Proj/rust/rust2/compiler/rustc_interface/src/queries.rs:422:19
  79: rustc_driver::run_compiler::{{closure}}
             at /Users/eric/Proj/rust/rust2/compiler/rustc_driver/src/lib.rs:337:22
  80: rustc_interface::interface::create_compiler_and_run::{{closure}}
             at /Users/eric/Proj/rust/rust2/compiler/rustc_interface/src/interface.rs:208:13
  81: rustc_span::with_source_map
             at /Users/eric/Proj/rust/rust2/compiler/rustc_span/src/lib.rs:788:5
  82: rustc_interface::interface::create_compiler_and_run
             at /Users/eric/Proj/rust/rust2/compiler/rustc_interface/src/interface.rs:202:5
  83: rustc_interface::interface::run_compiler::{{closure}}
             at /Users/eric/Proj/rust/rust2/compiler/rustc_interface/src/interface.rs:224:12
  84: rustc_interface::util::setup_callbacks_and_run_in_thread_pool_with_globals::{{closure}}::{{closure}}
             at /Users/eric/Proj/rust/rust2/compiler/rustc_interface/src/util.rs:155:13
  85: scoped_tls::ScopedKey<T>::set
             at /Users/eric/.cargo/registry/src/github.com-1ecc6299db9ec823/scoped-tls-1.0.0/src/lib.rs:137:9
  86: rustc_span::with_session_globals
             at /Users/eric/Proj/rust/rust2/compiler/rustc_span/src/lib.rs:105:5
  87: rustc_interface::util::setup_callbacks_and_run_in_thread_pool_with_globals::{{closure}}
             at /Users/eric/Proj/rust/rust2/compiler/rustc_interface/src/util.rs:153:9
  88: rustc_interface::util::scoped_thread::{{closure}}
             at /Users/eric/Proj/rust/rust2/compiler/rustc_interface/src/util.rs:128:24

From what I can tell, the debug information for everything directly in rustc_driver can be found, but anything from a dependency cannot. Note that generic functions are part of rustc_driver, so that is why things like rustc_middle functions appear in the backtrace above.

Minimal example

I created a simple demo project at https://github.com/ehuss/macos-dylib-debug.

This has an executable foo depends on dylib driver which depends on somedep.

If you build on macOS with the latest nightly (2021-03-31) which has split-debuginfo enabled by default, and run with RUST_BACKTRACE=1, you can see that the information is missing from the dependency:

   0: std::panicking::begin_panic
   1: somedep::do_something
   2: driver::run_driver
             at ./driver/src/lib.rs:2:5
   3: foo::main
             at ./src/main.rs:2:5
   4: core::ops::function::FnOnce::call_once
             at /rustc/74874a690bc95443292496ff5df5cc5c8cb56e0b/library/core/src/ops/function.rs:227:5

If you disable split-debuginfo (CARGO_PROFILE_DEV_SPLIT_DEBUGINFO=packed), then it gets the full information:

   0: std::panicking::begin_panic
             at /rustc/74874a690bc95443292496ff5df5cc5c8cb56e0b/library/std/src/panicking.rs:519:12
   1: somedep::do_something
             at ./somedep/src/lib.rs:3:5
   2: driver::run_driver
             at ./driver/src/lib.rs:2:5
   3: foo::main
             at ./src/main.rs:2:5
   4: core::ops::function::FnOnce::call_once
             at /rustc/74874a690bc95443292496ff5df5cc5c8cb56e0b/library/core/src/ops/function.rs:227:5

Similarly, trying to debug with lldb is unable to see anything in somedep. Running lldb with a dylib is a little tricky due to system integrity protection, it is something like this:

DYLD_FALLBACK_LIBRARY_PATH=$(rustc --print=sysroot)/lib/rustlib/x86_64-apple-darwin/lib \
    /Applications/Xcode.app/Contents/Developer/usr/bin/lldb ./target/debug/foo

b do_something fails to find the symbol. Setting a breakpoint on run_driver and running and then trying to step into do_something is unable to work. Doing all of the above with packed split-debuginfo works as expected.

Meta

rustc 1.53.0-nightly (74874a690 2021-03-30)

[alexcrichton]

Aha I see what's happening here. Instrumenting this function it looks like it's attemping to load a file named /var/folders/b0/wd3mrtcj36l61jkqrzpdjds00000gn/T/rustc2vbk9u/libsomedep-3b1888eeb49387a1.rlib(somedep-3b1888eeb49387a1.4c36handi6gb1nd7.rcgu.o). That file was present during linking but is naturally in a temporary location that rustc later deletes.

This temporary file is created here in the compiler and while most of the time we skip that we don't skip it if the crate is a dylib. I forget the exact reason we generate a new rlib if there's a dylib input...

In any case that's the reason debuginfo doesn't work, it's because the file with the debuginfo was deleted (it was a temporary file). The fix here would probably look like one of:

1. Remove the need to create a separate rlib. This is not only causing issues with debuginfo but it's also just a generally really slow operation to copy everything around.
2. Preserve the temporary rlib if split-debuginfo is requested. This would probably mean moving the temporary rlib out of a temporary folder into --out-dir.
3. Automatically run dsymutil despite the request for split-debuginfo for dylibs because we'll be deleting the source of debuginfo otherwise.

After writing this I'm also now recalling that the reason for this is we call link_whole_rlib for dylibs which means we need to remove metadata from rlib files since otherwise the linker rightfully complains that the file isn't an object file.

[ehuss]

A related question, I can't seem to get lldb to find symbols in any rlib dependencies. Is split-debuginfo intended to work for that?

Example: build cargo and set a breakpoint for join_paths which is in the cargo_util crate. It can't find it (or step into any dependency). It can find things in the cargo package, though:

(lldb) b join_paths
Breakpoint 1: no locations (pending).
WARNING:  Unable to resolve breakpoint to any actual locations.
(lldb) b fill_env
Breakpoint 2: where = cargo`cargo::core::compiler::compilation::Compilation::fill_env::ha5affff1c0a6224b + 143 at compilation.rs:262:13, address = 0x00000001000cf1af

[alexcrichton]

That sounds like a bug in something, although I have no idea what. The issue seems likely unrelated to this though? Do you know if the backtrace crate can see the join_paths frame and LLDB can't?

Also, I was struck with inspiration about this last night. The only reason we create temporary rlibs is to remove bytecode and metadata but bytecode now lives in adjacent sections in object files and we could presumably do the same thing with an empty object file for metadata as well. By ensuring that everything in an archive is an object file we can guarantee that raw rlibs can always be passed to the linker, even with --whole-archive. This would involve updating how we store metadata in archives, but I don't think there's anything fundamental about that change other than it's just some work that needs to be done.

Once we change the format of metadata in archives (to be a section of an empty object instead of just a raw binary file) then I think we have no need to create temporary rlibs and we can pass the rlibs as-is to the linker (basically delete all that copying-around code in the compiler related to linking)

[ehuss]

I believe backtrace is able to find the frames. It is able to print things like this:

   2: cargo_util::paths::join_paths
             at /Users/eric/Proj/rust/cargo/crates/cargo-util/src/paths.rs:21:14

In general I've noticed that lldb is having trouble with any rlib dependency (even with "packed"). I tried older versions of rust and it seems to have a lot of trouble dealing with them. Even if lldb can find the symbols, it can't find the source files, which is surprising because the absolute paths are clearly in the DWARF file.

It is pretty rare that I use a debugger, so I'm not familiar with what has and has not been working in the past. It would be nice if it were a little more reliable.

Anyways, whatever you said sounds good even if I don't fully understand it.