x.py: `--keep-stage` is broken

[RalfJung]

Say you are working on libcore, did some changes and now want to see of the test suite still passes. So you do

./x.py --stage 1 test src/libcore

Unfortunately, even if all changes are inside libcore, this leads to the entire stage0 rustc being rebuilt. So, this takes forever.

So, instead I am told I should do

./x.py --stage 1 --keep-stage 0 test src/libcore

But that fails:

$ nice ./x.py --stage 1 --keep-stage 0 test src/libcore
Updating only changed submodules
Submodules updated in 0.03 seconds
    Finished dev [unoptimized] target(s) in 0.21s
Warning: Using a potentially old librustc. This may not behave well.
Warning: Using a potentially old libtest. This may not behave well.
Warning: Using a potentially old librustc. This may not behave well.
Copying stage0 rustc from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage0 codegen artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu, llvm)
    Finished release [optimized] target(s) in 0.23s
Assembling stage1 compiler (x86_64-unknown-linux-gnu)
Warning: Using a potentially old libtest. This may not behave well.
Copying stage1 test from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building rustdoc for stage1 (x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.20s
Testing core stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
   Compiling core v0.0.0 (file:///home/r/src/rust/rustc/src/libcore)
/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1/bin/rustc: error while loading shared libraries: libstd-dca726f5ff6a5f1b.so: cannot open shared object file: No such file or directory
error: Could not compile `core`.

To learn more, run the command again with --verbose.


command did not execute successfully: "/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "test" "--target" "x86_64-unknown-linux-gnu" "-j" "8" "--release" "--features" "panic-unwind debug-jemalloc jemalloc backtrace" "--manifest-path" "/home/r/src/rust/rustc/src/libstd/Cargo.toml" "-p" "core" "--" "--quiet"
expected success, got: exit code: 101


failed to run: /home/r/src/rust/rustc/build/bootstrap/debug/bootstrap --stage 1 --keep-stage 0 test src/libcore
Build completed unsuccessfully in 0:00:01

(EDIT: Originally I thought this was a missing feature, but it really is a bug in --keep-stage.)

[RalfJung]

To make things even better, even changes in libcore/tests/slice.rs will trigger a full re-build -- but if I have a type error in there, I will only get an error after waiting for 15min because the test suite is only compiled in stage 1.

This makes developing tests an exercise in patience...

[tinco]

I have a similar problem. If I make changes in libsyntax, and I run:

/x.py test --stage 1 --keep-stage 0 src/test/pretty

I get an exception about not being able to find std. I'm currently compiling and the exact error is lost in the backlog, but when I get the chance I'll try again and post the error here.

I also have the problem that even changes in tests trigger compilation of both stages.

[tinco]

Ah, my compilation failed, so I tried with --keep-stage 0 again, this is the full error:

  rust git:(issue_12590) ✗ ./x.py test --keep-stage 0 --stage 1 src/test/pretty ; say "I am done. I am really done. ... Really"
Updating only changed submodules
Submodules updated in 0.12 seconds
    Finished dev [unoptimized] target(s) in 0.29s
Warning: Using a potentially old librustc. This may not behave well.
Warning: Using a potentially old librustc. This may not behave well.
Warning: Using a potentially old libtest. This may not behave well.
Warning: Using a potentially old librustc. This may not behave well.
Copying stage0 rustc from stage0 (x86_64-apple-darwin -> x86_64-apple-darwin / x86_64-apple-darwin)
Building stage0 codegen artifacts (x86_64-apple-darwin -> x86_64-apple-darwin, llvm)
   Compiling serialize v0.0.0 (file:///Users/tinco/Source/rust/src/libserialize)
   Compiling rustc_target v0.0.0 (file:///Users/tinco/Source/rust/src/librustc_target)
   Compiling syntax v0.0.0 (file:///Users/tinco/Source/rust/src/libsyntax)
   Compiling rustc v0.0.0 (file:///Users/tinco/Source/rust/src/librustc)
error[E0463]: can't find crate for `std`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: Could not compile `serialize`.

Caused by:
  process didn't exit successfully: `/Users/tinco/Source/rust/build/bootstrap/debug/rustc --crate-name serialize libserialize/lib.rs --error-format json --crate-type dylib --crate-type rlib --emit=dep-info,link -C prefer-dynamic -C opt-level=2 -C metadata=b5d7af8c38a39a93 -C extra-filename=-b5d7af8c38a39a93 --out-dir /Users/tinco/Source/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps --target x86_64-apple-darwin -L dependency=/Users/tinco/Source/rust/build/x86_64-apple-darwin/stage0-rustc/x86_64-apple-darwin/release/deps -L dependency=/Users/tinco/Source/rust/build/x86_64-apple-darwin/stage0-rustc/release/deps` (exit code: 101)
warning: build failed, waiting for other jobs to finish...
error: build failed
command did not execute successfully: "/Users/tinco/Source/rust/build/x86_64-apple-darwin/stage0/bin/cargo" "build" "--target" "x86_64-apple-darwin" "-j" "4" "--release" "--manifest-path" "/Users/tinco/Source/rust/src/librustc_codegen_llvm/Cargo.toml" "--features" " jemalloc" "--message-format" "json"
expected success, got: exit code: 101
thread 'main' panicked at 'cargo must succeed', bootstrap/compile.rs:1117:9
note: Run with `RUST_BACKTRACE=1` for a backtrace.
failed to run: /Users/tinco/Source/rust/build/bootstrap/debug/bootstrap test --keep-stage 0 --stage 1 src/test/pretty
Build completed unsuccessfully in 0:00:03

[RalfJung]

Actually --keep-stage seems entirely broken: I just did ./x.py build, then edited something in librustc_mir, and then did ./x.py build --keep-stage 1. The result:

$ ./x.py build --keep-stage 1
Updating only changed submodules
Submodules updated in 0.03 seconds
    Finished dev [unoptimized] target(s) in 0.17s
Warning: Using a potentially old librustc. This may not behave well.
Warning: Using a potentially old libstd. This may not behave well.
Warning: Using a potentially old libtest. This may not behave well.
Warning: Using a potentially old librustc. This may not behave well.
Building stage0 std artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.17s
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage0 test artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
    Finished release [optimized] target(s) in 0.16s
Copying stage0 test from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage0 compiler artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
   Compiling rustc_mir v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_mir)
   Compiling rustc_lint v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_lint)
   Compiling rustc_passes v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_passes)
   Compiling rustc_codegen_utils v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_codegen_utils)
   Compiling rustc_borrowck v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_borrowck)
   Compiling rustc_driver v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_driver)
   Compiling rustc-main v0.0.0 (file:///home/r/src/rust/rustc/src/rustc)
    Finished release [optimized] target(s) in 13.38s
Copying stage0 rustc from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage0 codegen artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu, llvm)
   Compiling rustc_codegen_llvm v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_codegen_llvm)
    Finished release [optimized] target(s) in 6.31s
Assembling stage1 compiler (x86_64-unknown-linux-gnu)
Warning: Using a potentially old librustc. This may not behave well.
Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage1 codegen artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu, llvm)
   Compiling rustc_mir v0.0.0 (file:///home/r/src/rust/rustc/src/librustc_mir)
error[E0463]: can't find crate for `std`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: Could not compile `rustc_mir`.

Caused by:
  process didn't exit successfully: `/home/r/src/rust/rustc/build/bootstrap/debug/rustc --crate-name rustc_mir librustc_mir/lib.rs --error-format json --crate-type dylib --emit=dep-info,link -C prefer-dynamic -C opt-level=2 -C metadata=8e9da67c28005143 -C extra-filename=-8e9da67c28005143 --out-dir /home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps -L dependency=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps --extern arena=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libarena-417926933fe788a8.so --extern bitflags=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libbitflags-7a49a1619dd10bb0.rlib --extern byteorder=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libbyteorder-4ca9f7fb66199358.rlib --extern either=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libeither-bdd04aa2a3a1f20d.rlib --extern graphviz=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libgraphviz-b71a70c8a4ed7469.so --extern log=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/liblog-0eb143a377f6f226.rlib --extern log_settings=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/liblog_settings-6bf5cebbd6300a28.rlib --extern polonius_engine=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libpolonius_engine-9a343edcb4233c18.rlib --extern rustc=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc-efa0afc4091e90c9.so --extern rustc_apfloat=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_apfloat-ac01335060c9f611.rlib --extern rustc_data_structures=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_data_structures-80c75f7b514397ee.so --extern rustc_errors=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_errors-2464ee6ca699f10d.so --extern rustc_target=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_target-635e8fb1260aff1c.so --extern serialize=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libserialize-6ea93882817b1dda.so --extern serialize=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libserialize-6ea93882817b1dda.rlib --extern syntax=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libsyntax-ac5f2f7cd567ecc7.so --extern syntax_pos=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libsyntax_pos-358f903881d0285f.so -L native=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/backtrace-sys-daf5c28be2095534/out -L native=/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/miniz-sys-5b64b3ed0b67f8cb/out` (exit code: 101)
command did not execute successfully: "/home/r/src/rust/rustc/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-j" "8" "--release" "--manifest-path" "/home/r/src/rust/rustc/src/librustc_codegen_llvm/Cargo.toml" "--features" " jemalloc" "--message-format" "json"
expected success, got: exit code: 101
thread 'main' panicked at 'cargo must succeed', bootstrap/compile.rs:1117:9
note: Run with `RUST_BACKTRACE=1` for a backtrace.
failed to run: /home/r/src/rust/rustc/build/bootstrap/debug/bootstrap build --keep-stage 1
Build completed unsuccessfully in 0:00:21

This is really bad, it extends the edit-compile cycle for miri from <3min to >30min...

[RalfJung]

I think some of this was my misunderstanding about --keep-stage 1 implying --keep-stage 0 (it does not, and anyway there are very few cases where you want to skip both stage 0 and 1). Also, the part about --stage 0 test src/libcore is a duplicate of #52186. So I think I am going to close this issue.