diff --git a/posts/inside-rust/2024-10-10-test-infra-oct-2024.md b/posts/inside-rust/2024-10-10-test-infra-oct-2024.md new file mode 100644 index 000000000..f47e6dd18 --- /dev/null +++ b/posts/inside-rust/2024-10-10-test-infra-oct-2024.md @@ -0,0 +1,213 @@ +--- +layout: post +title: "This Month in Our Test Infra: September 2024" +author: Jieyou Xu +team: the Bootstrap Team +--- + +# This Month in Our Test Infra: September 2024 + +This is a quick summary of the changes in the test infrastructure for the +[rust-lang/rust][rust] repository[^scope] for September 2024. It also includes +brief descriptions of on-going work. + +[^scope]: The test infra here refers to the test harness [compiletest] and +supporting components in our build system [bootstrap]. This test infra is used +mainly by rustc and rustdoc. Other tools like cargo, miri or rustfmt maintain +their own test infra. + +As usual, if you encounter bugs or UX issues when using our test infrastrucutre, +please [file an issue][new-issue]. Bugs and papercuts can't be fixed if we don't +know about them! + +## Changes + +### `run-make` test suite now has access to a properly-staged cargo + +[bootstrap] now builds a properly-staged cargo and makes it available for +`run-make` tests. Previously, `run-make` tests just used whatever initial cargo +[bootstrap] had access to, but this caused problems if a `run-make` test uses a +cargo feature that's present in nightly but not in beta. We encountered some +interesting build cache invalidation issues related to differing `RUSTFLAGS` in +the process, but were able to fix them. We want to add mechanisms to [bootstrap] +to make it harder to misuse `RUSTFLAGS` which may lead to hard-to-diagnose build +cache invalidation in the future. + +- bootstrap: [Fix cargo staging for run-make tests #130739](https://github.com/rust-lang/rust/pull/130739) +- bootstrap/compiletest: [Pass the current cargo to run-make tests #130642](https://github.com/rust-lang/rust/pull/130642) +- bootstrap: [Prevent building cargo from invalidating build cache of other + tools due to conditionally applied `-Zon-broken-pipe=kill` via tracked + `RUSTFLAGS` #131155](https://github.com/rust-lang/rust/pull/131155) + +Why `-Zon-broken-pipe=kill` is required when building rustc was its [own entire +rabbit hole](https://github.com/rust-lang/rust/issues/131436), in case you were +curious. + +### More `run-make` migrations and fixes + +The `emit-to-stdout` `run-make` test was ported to `rmake.rs`, only [10 more to +go]. The remaining ones are stuck on being quite tricky. See the [tracking +issue][port-run-make] for why we are transitioning away from `Makefile`s in +`run-make` tests. + +- run-make: [port `emit-to-stdout` to rmake.rs #131355](https://github.com/rust-lang/rust/pull/131355) + +Misc: + +- run-make: [Add missing `needs-llvm-components` directives for run-make tests + that need target-specific codegen + #129605](https://github.com/rust-lang/rust/pull/129605) + +[remaining-run-make-tests]: https://github.com/rust-lang/rust/blob/883f9a2c8f8923eafafbeba8b18361424b148f05/src/tools/tidy/src/allowed_run_make_makefiles.txt#L1C1-L10C30 +[port-run-make]: https://github.com/rust-lang/rust/issues/121876 + +### `run_make_support` library updates + +`run_make_support` is the support library built and made available to `run-make` +tests. + +- run_make_support: [Add `llvm-pdbutil` wrapper #130048](https://github.com/rust-lang/rust/pull/130048) +- run_make_support: [Rename `Command::stdin` to `stdin_buf` and add + `std{in,out,err}` config helpers + #129973](https://github.com/rust-lang/rust/pull/129973) +- run_make_support: [Rectify symlink handling + #130427](https://github.com/rust-lang/rust/pull/130427) + +### [compiletest] improvements and fixes + +We dropped [compiletest]'s legacy directive check (e.g. `// ignore-test hello` +no longer warns). This was originally added when we migrated from `//` to `//@` +to help test writers notice the new directives, but now a long time has passed +so we can remove it as it was causing friction in adding new directives and +authoring tests. For example, the [Specification Team][t-spec] wanted to add a +`//@ reference` directive, but the legacy directive check would trigger on: + +```rust,no_run +// So what if we added a +// reference to the <- `reference` is a known directive, and +// `// reference` looks suspcious! +// rustc-dev-guide? +``` + +This was added to initially to help migration from `//` to `//@`, but since a +long time has passed we no longer need this check to help contributors know that +legacy directives are being phased out. + +- compiletest: [Drop compiletest legacy directive check + #131392](https://github.com/rust-lang/rust/pull/131392) + +[t-spec]: https://www.rust-lang.org/governance/teams/lang#team-spec + +We updated some `compiletest` normalizations and directive renaming. In +particular, we restricted `//@ ignore-mode-*` directives to not accept *all* +test suites, and later converted `//@ ignore-mode-coverage-map` and `//@ +ignore-mode-coverage-run` to `//@ ignore-coverage-map` and `//@ +ignore-coverage-run` because only `coverage-map` and `coverage-run` were special +in that the same test source files ran under two test suite configurations. + +- compiletest: [Add `{{rust-src-base}}` (for sysroot src base) + #129687](https://github.com/rust-lang/rust/pull/129687) +- compiletest: [Restrict `ignore-mode-*` directives + #131346](https://github.com/rust-lang/rust/pull/131346) +- compiletest: [Simplify the compiletest directives for ignoring coverage-test + modes #131400](https://github.com/rust-lang/rust/pull/131400) + +We broke up [compiletest]'s `runtest.rs` as it was [previously +*massive*][prev-runtest], clocking in at 4710 lines. It's now around 2700 lines, +so still massive, but at least slightly less so. + +- compiletest: [Break up compiletest `runtest.rs` into smaller helper modules + #130566](https://github.com/rust-lang/rust/pull/130566) + +[prev-runtest]: https://github.com/rust-lang/rust/blob/b7b9453ea7354ee39b15390ffd0b4f9e2000076b/src/tools/compiletest/src/runtest.rs + +We added a help message upon `crashes` test failure that you can set +`COMPILETEST_VERBOSE_CRASHES=1` to get compiler stderr/stdout output from trying +to build the failing `crashes` test. + +- compiletest: [Mention `COMPILETEST_VERBOSE_CRASHES` on crashes test failure + #130793](https://github.com/rust-lang/rust/pull/130793) + +We also registered [tool docs][compiletest-tool-docs] for [compiletest]. There +currently isn't much doc comments in [compiletest], but having them getting +built and made available as part of nightly rustc docs is a good first step. + +- bootstrap: [Register tool docs for compiletest + #130567](https://github.com/rust-lang/rust/pull/130567) + +[compiletest-tool-docs]: https://doc.rust-lang.org/nightly/nightly-rustc/compiletest/index.html + +Misc: + +- compiletest: [Rename "runtest/crash.rs" to "runtest/crashes.rs" to be in line + with the test directory + #130973](https://github.com/rust-lang/rust/pull/130973) + +### Testing documentation improvements + +We improved testing docs in [rustc-dev-guide][dev-guide-testing]. We added a +[testing best practices chapter][dev-guide-testing-best-practices], and updated +the [compiletest directives listing][dev-guide-directives-listing]. + +- rustc-dev-guide: [Revise testing chapters excluding the directives chapter + #2088](https://github.com/rust-lang/rustc-dev-guide/pull/2088) +- rustc-dev-guide: [Revise directives docs + #2089](https://github.com/rust-lang/rustc-dev-guide/pull/2089) +- rustc-dev-guide: [Revise test naming advice to discourage using issue numbers + alone #2090](https://github.com/rust-lang/rustc-dev-guide/pull/2090) +- rustc-dev-guide: [Document compiletest directives `ignore-coverage-map` and + `ignore-coverage-run` + #2094](https://github.com/rust-lang/rustc-dev-guide/pull/2094) +- rustc-dev-guide: [Small follow-up to my "internal #[rustc_*] TEST attributes" + PR #2082](https://github.com/rust-lang/rustc-dev-guide/pull/2082) +- rustc-dev-guide: [Note lldb debuginfo requires `python310.dll` to be present + in `PATH` on Windows + #2076](https://github.com/rust-lang/rustc-dev-guide/pull/2076) +- rustc-dev-guide: [Document crashes test suite + #2075](https://github.com/rust-lang/rustc-dev-guide/pull/2075) +- rustc-dev-guide: [Purge `run-pass-valgrind mentions` + #2091](https://github.com/rust-lang/rustc-dev-guide/pull/2091) +- rustc-dev-guide: [Add documentation for `{{rust-src-base}}` + #2079](https://github.com/rust-lang/rustc-dev-guide/pull/2079) + +There's still a lot of room for improvement in our testing docs -- in +[compiletest], [bootstrap] and [rustc-dev-guide], but one step at a time. + +[dev-guide-testing]: https://rustc-dev-guide.rust-lang.org/tests/intro.html +[dev-guide-testing-best-practices]: https://rustc-dev-guide.rust-lang.org/tests/best-practices.html +[dev-guide-directives-listing]: https://rustc-dev-guide.rust-lang.org/tests/directives.html + +### Test suite cleanups + +We deleted an entire test suite `run-pass-valgrind` because it was never +properly wired up and properly implemented, and was not used. It turns out +deleting the test suite actually fixes a bug from 2017 [run-pass-valgrind tests +don't actually run in valgrind +#44816](https://github.com/rust-lang/rust/issues/44816) because you can't have a +test suite related bug if the test suite doesn't exist! + +- bootstrap/compiletest/opt-dist: [Remove valgrind test suite and support from + compiletest, bootstrap and opt-dist + #131351](https://github.com/rust-lang/rust/pull/131351) + +Misc: + +- rustdoc: [Rename `issue-\d+.rs` tests to have meaningful names (part 9) + #130287](https://github.com/rust-lang/rust/pull/130287) + +## On-going developments + +- Add test infrastructure support for a `minicore` test auxiliary, so that + `#![no_std]` cross-compiling build-only tests don't need to reinvent and + reimplement `core` prelude stubs again and again. See + . +- We want to make `RUSTFLAGS` harder to misuse that can lead to tool build cache + invalidation, leading to unnecessary rebuilds. +- There's on-going effort to redesign stage0 std, to help make [bootstrap] + staging more consistent and more intuitive. + +[rust]: https://github.com/rust-lang/rust +[compiletest]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest +[bootstrap]: https://github.com/rust-lang/rust/tree/master/src/bootstrap +[new-issue]: https://github.com/rust-lang/rust/issues/new +[rustc-dev-guide]: https://rustc-dev-guide.rust-lang.org/getting-started.html