Problem

Tock is an operating system written in Rust for embedded platforms (e.g. thumbv7em-none-eabi target). We've been trying to make builds reproducible, with good progress (tock/tock#1666). In particular:

• pinning a version of the Rust compiler,
• using --remap-path-prefix,
• using a Cargo workspace so that local dependencies are identified by a relative path (Tock is split into multiple crates inside the workspace),
• using cargo rustc to avoid passing custom linker arguments (that include paths on the filesystem) to the dependencies.

With these we've managed to make the builds reproducible across various Linux machines.

However, the builds are different on Linux (nightly-2020-02-03-x86_64-unknown-linux-gnu) and MacOS (nightly-2020-02-03-x86_64-apple-darwin), as evidenced by CI on a GitHub workflow (google/OpenSK#94 (comment)).

In particular, the issue seems to stem from a different -C metadata= (and -C extra-filename=) passed to rustc (while the steps that I mentioned above make sure that the same metadata hash is passed across Linux machines for each crate in the project).

These are built with the --verbose parameter passed to cargo, and an example rustc invocation is the following (tock_cells crate of the Tock project).

• On Linux (https://github.com/google/OpenSK/pull/94/checks?check_run_id=602439620): rustc --crate-name tock_cells libraries/tock-cells/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=z -C panic=abort -C debuginfo=2 -C metadata=92487154152022d3 -C extra-filename=-92487154152022d3 --out-dir /home/runner/work/OpenSK/OpenSK/third_party/tock/target/thumbv7em-none-eabi/release/deps --target thumbv7em-none-eabi -L dependency=/home/runner/work/OpenSK/OpenSK/third_party/tock/target/thumbv7em-none-eabi/release/deps -L dependency=/home/runner/work/OpenSK/OpenSK/third_party/tock/target/release/deps -C link-arg=-Tlayout.ld -C linker=rust-lld -C linker-flavor=ld.lld -C relocation-model=dynamic-no-pic -C link-arg=-zmax-page-size=512 -C link-arg=-icf=all --remap-path-prefix=/home/runner/work/OpenSK/OpenSK/third_party/tock/= -D warnings
• On OSX (https://github.com/google/OpenSK/pull/94/checks?check_run_id=602439631): rustc --crate-name tock_cells libraries/tock-cells/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=z -C panic=abort -C debuginfo=2 -C metadata=9fc982d890c0358d -C extra-filename=-9fc982d890c0358d --out-dir /Users/runner/runners/2.169.0/work/OpenSK/OpenSK/third_party/tock/target/thumbv7em-none-eabi/release/deps --target thumbv7em-none-eabi -L dependency=/Users/runner/runners/2.169.0/work/OpenSK/OpenSK/third_party/tock/target/thumbv7em-none-eabi/release/deps -L dependency=/Users/runner/runners/2.169.0/work/OpenSK/OpenSK/third_party/tock/target/release/deps -C link-arg=-Tlayout.ld -C linker=rust-lld -C linker-flavor=ld.lld -C relocation-model=dynamic-no-pic -C link-arg=-zmax-page-size=512 -C link-arg=-icf=all --remap-path-prefix=/Users/runner/runners/2.169.0/work/OpenSK/OpenSK/third_party/tock/= -D warnings

Steps

1. git clone https://github.com/tock/tock
2. cd tock/boards/nordic/nrf52840dk
3. V=1 make

This will invoke cargo with the following parameters: RUSTFLAGS="-C link-arg=-Tlayout.ld -C linker=rust-lld -C linker-flavor=ld.lld -C relocation-model=dynamic-no-pic -C link-arg=-zmax-page-size=512 -C link-arg=-icf=all --remap-path-prefix=/path/to/tock/= " cargo rustc --verbose --target=thumbv7em-none-eabi --package nrf52840dk --bin nrf52840dk --release -- -C link-arg=-L/path/to/tock/boards/nordic/nrf52840dk. One of the resulting rustc invocations is detailed above.
The exact /path/to/tock/ depends on the machine, but this has no impact across the Linux machines we've tested.

More detailed steps are available in the GitHub workflow (https://github.com/google/OpenSK/pull/94/checks?check_run_id=602439620, https://github.com/google/OpenSK/pull/94/checks?check_run_id=602439631)

Possible Solution(s)

The various steps that we've taken have allowed to obtain the same -C metadata=hash parameter across Linux machines, at which points the builds are reproducible across these machines. Given that when running cargo in --verbose mode we observe different metadata hashes, the computation of such hashes is likely the problem.

• Maybe this hash (incorrectly) depends on the exact toolchain (nightly-2020-02-03-x86_64-unknown-linux-gnu vs. nightly-2020-02-03-x86_64-apple-darwin)? I'd assume it should only depend on the nightly-2020-02-03 part and the --target thumbv7em-none-eabi, but not on the OS we compile from.
• Maybe some other parameters are taken into account in computing the hash?

Notes

See also rust-lang/rust#71361 and google/OpenSK#94

Output of cargo version (GitHub workflow machines with actions-rs/toolchain@v1):

• Linux:

  /usr/share/rust/.cargo/bin/rustc -V
  rustc 1.42.0-nightly (f43c34a13 2020-02-02)
  /usr/share/rust/.cargo/bin/cargo -V
  cargo 1.42.0-nightly (9d32b7b01 2020-01-26)
  /usr/share/rust/.cargo/bin/rustup -V
  rustup 1.21.1 (7832b2ebe 2019-12-20)
  

• OSX:

  /Users/runner/.cargo/bin/rustc -V
  rustc 1.42.0-nightly (f43c34a13 2020-02-02)
  /Users/runner/.cargo/bin/cargo -V
  cargo 1.42.0-nightly (9d32b7b01 2020-01-26)
  /Users/runner/.cargo/bin/rustup -V
  rustup 1.21.1 (2020-01-08)