From f00643aa1cb4ee35f619406fbd7934d6a1093d50 Mon Sep 17 00:00:00 2001 From: binarycat Date: Tue, 18 Mar 2025 12:04:47 -0500 Subject: [PATCH 1/8] add new section on the `rustdoc` test suite --- src/rustdoc-internals/htmldocck.md | 101 +++++++++++++++++++++++++++++ src/rustdoc.md | 14 ++-- src/tests/compiletest.md | 2 +- 3 files changed, 109 insertions(+), 8 deletions(-) create mode 100644 src/rustdoc-internals/htmldocck.md diff --git a/src/rustdoc-internals/htmldocck.md b/src/rustdoc-internals/htmldocck.md new file mode 100644 index 000000000..d69627f46 --- /dev/null +++ b/src/rustdoc-internals/htmldocck.md @@ -0,0 +1,101 @@ +# `rustdoc` tests + +This page is specifically about the `rustdoc` test suite, for other test suites used for testing rustdoc, see [Rustdoc § Tests](../rustdoc.md#tests). + +`htmldocck.py` is a custom checker script that uses [XPath] to verify the HTML output of rustdoc. + +[XPath]: https://en.wikipedia.org/wiki/XPath + +## Directives +Directives to htmldocck are similar to those given to `compiletest` in that they take the form of `//@` comments. + +All `PATH`s in directives are relative to the the rustdoc output directory (`build/TARGET/test/rustdoc/TESTNAME`), +so it is conventional to use a `#![crate_name = "foo"]` attribute to avoid writing paths. +To avoid repetion, `-` can be used in any `PATH` argument to re-use the previous `PATH` argument. + +All arguments take the form of quoted strings, +with the exception of `COUNT` and the special `-` form of `PATH`. + +Directives are assertions that place constraints on the generated HTML. + +All directives (except `files`) can be negated by putting a `!` in front of their name. + +Similar to shell commands, +directives can extend across multiple lines if their last char is `\`. +In this case, the start of the next line should be `//`, with no `@`. + +For example, `//@ !has 'foo/struct.Bar.html'` checks that crate `foo` does not have a page for a struct named `Bar` in the crate root. + +### `has` + +Usage 1: `//@ has PATH` +Usage 2: `//@ has PATH XPATH PATTERN` + +In the first form, `has` checks that a given file exists. + +In the second form, `has` is an alias for `matches`, +except `PATTERN` is a whitespace-normalized[^1] string instead of a regex. + +### `matches` + +Usage: `//@ matches PATH XPATH PATTERN` + +Checks that the text of each element selected by `XPATH` in `PATH` matches the python-flavored regex `PATTERN`. + +### `matchesraw` + +Usage: `//@ matchesraw PATH PATTERN` + +Checks that the contents of the file `PATH` matches the regex `PATTERN`. + +### `hasraw` + +Usage: `//@ hasraw PATH PATTERN` + +Same as `matchesraw`, except `PATTERN` is a whitespace-normalized[^1] string instead of a regex. + +### `count` + +Usage: `//@ count PATH XPATH COUNT` + +Checks that there are exactly `COUNT` matches for `XPATH` within the file `PATH`. + +### `snapshot` + +Usage: `//@ snapshot NAME PATH XPATH` + +Creates a snapshot test named NAME. +A snapshot test captures a subtree of the DOM, at the location +determined by the XPath, and compares it to a pre-recorded value +in a file. The file's name is the test's name with the `.rs` extension +replaced with `.NAME.html`, where NAME is the snapshot's name. + +htmldocck supports the `--bless` option to accept the current subtree +as expected, saving it to the file determined by the snapshot's name. +compiletest's `--bless` flag is forwarded to htmldocck. + +### `has-dir` + +Usage: `//@ has-dir PATH` + +Checks for the existance of directory `PATH`. + +### `files` + +Usage: `//@ files PATH ENTRIES` + +Checks that the directory `PATH` contains exactly `ENTRIES`. +`ENTRIES` is a python list of strings inside a quoted string, +as if it were to be parsed by `eval`. +(note that the list is actually parsed by `shlex.split`, +so it cannot contain arbitrary python expressions). + +Example: `//@ files "foo/bar" '["index.html", "sidebar-items.js"]'` + +[^1]: Whitespace normalization means that all spans of consecutive whitespace are replaced with a single space. The files themselves are also whitespace-normalized. + +## Limitations +All `XPATH` arguments must start with `//` due to a flaw in the implemention. + +Only well-formed HTML can be parsed (hopefully rustdoc doesn't output mismatched tags). + diff --git a/src/rustdoc.md b/src/rustdoc.md index 356698148..5ac62aa2c 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -77,27 +77,27 @@ does is call the `main()` that's in this crate's `lib.rs`, though.) `doctest.rs`. * The Markdown renderer is loaded up in `html/markdown.rs`, including functions for extracting doctests from a given block of Markdown. -* The tests on the structure of rustdoc HTML output are located in `tests/rustdoc`, where - they're handled by the test runner of bootstrap and the supplementary script - `src/etc/htmldocck.py`. * Frontend CSS and JavaScript are stored in `html/static/`. ## Tests -* All paths in this section are relative to `tests` in the rust-lang/rust repository. -* Tests on search engine and index are located in `rustdoc-js` and `rustdoc-js-std`. +* Tests on search engine and index are located in `tests/rustdoc-js` and `tests/rustdoc-js-std`. The format is specified [in the search guide](rustdoc-internals/search.md#testing-the-search-engine). * Tests on the "UI" of rustdoc (the terminal output it produces when run) are in - `rustdoc-ui` + `tests/rustdoc-ui` * Tests on the "GUI" of rustdoc (the HTML, JS, and CSS as rendered in a browser) - are in `rustdoc-gui`. These use a [NodeJS tool called + are in `tests/rustdoc-gui`. These use a [NodeJS tool called browser-UI-test](https://github.com/GuillaumeGomez/browser-UI-test/) that uses puppeteer to run tests in a headless browser and check rendering and interactivity. * Additionally, JavaScript type annotations are written using [TypeScript-flavored JSDoc] comments and an external d.ts file. The code itself is plain, valid JavaScript; we only use tsc as a linter. +* The tests on the structure of rustdoc HTML output are located in `tests/rustdoc`, + where they're handled by the test runner of bootstrap and + the supplementary script `src/etc/htmldocck.py`. + [These tests have several extra directives available to them](./rustdoc-internals/htmldocck.md). [TypeScript-flavored JSDoc]: https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html diff --git a/src/tests/compiletest.md b/src/tests/compiletest.md index 2c35381ea..ee66b787b 100644 --- a/src/tests/compiletest.md +++ b/src/tests/compiletest.md @@ -78,7 +78,7 @@ The following test suites are available, with links for more information: ### Rustdoc test suites -See [Rustdoc tests](../rustdoc.md#tests) for more details. +See [Rustdoc § Tests](../rustdoc.md#tests) for more details. | Test suite | Purpose | |------------------|--------------------------------------------------------------------------| From 8b501562a8e0b21fb304394a3cb6df18c0f54cc3 Mon Sep 17 00:00:00 2001 From: binarycat Date: Tue, 18 Mar 2025 12:10:43 -0500 Subject: [PATCH 2/8] add htmldocck.md to SUMMARY.md --- src/SUMMARY.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 9e2d0774a..fb4687774 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -95,7 +95,7 @@ - [Parallel Compilation](./parallel-rustc.md) - [Rustdoc internals](./rustdoc-internals.md) - [Search](./rustdoc-internals/search.md) - + - [`rustdoc` tests](./rustdoc-internals/htmldocck.md) # Source Code Representation - [Prologue](./part-3-intro.md) From f248d2f57cd99ae6f4962582b9ecc574b60ceb05 Mon Sep 17 00:00:00 2001 From: binarycat Date: Tue, 18 Mar 2025 13:10:26 -0500 Subject: [PATCH 3/8] htmldocck: expand limitations and mention compiletest directives --- src/rustdoc-internals/htmldocck.md | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/src/rustdoc-internals/htmldocck.md b/src/rustdoc-internals/htmldocck.md index d69627f46..b37c1ac3d 100644 --- a/src/rustdoc-internals/htmldocck.md +++ b/src/rustdoc-internals/htmldocck.md @@ -9,6 +9,10 @@ This page is specifically about the `rustdoc` test suite, for other test suites ## Directives Directives to htmldocck are similar to those given to `compiletest` in that they take the form of `//@` comments. +In addition to the directives listed here, +`rustdoc` tests also support most +[compiletest directives](../tests/directives.html). + All `PATH`s in directives are relative to the the rustdoc output directory (`build/TARGET/test/rustdoc/TESTNAME`), so it is conventional to use a `#![crate_name = "foo"]` attribute to avoid writing paths. To avoid repetion, `-` can be used in any `PATH` argument to re-use the previous `PATH` argument. @@ -95,7 +99,9 @@ Example: `//@ files "foo/bar" '["index.html", "sidebar-items.js"]'` [^1]: Whitespace normalization means that all spans of consecutive whitespace are replaced with a single space. The files themselves are also whitespace-normalized. ## Limitations -All `XPATH` arguments must start with `//` due to a flaw in the implemention. - -Only well-formed HTML can be parsed (hopefully rustdoc doesn't output mismatched tags). +`htmldocck.py` uses the xpath implementation from the standard library. +This leads to several limitations: +* All `XPATH` arguments must start with `//` due to a flaw in the implemention. +* Many XPath features (functions, axies, etc.) are not supported. +* Only well-formed HTML can be parsed (hopefully rustdoc doesn't output mismatched tags). From 72aa06dff1dcfd3516aded79399336932c290976 Mon Sep 17 00:00:00 2001 From: binarycat Date: Tue, 18 Mar 2025 13:19:33 -0500 Subject: [PATCH 4/8] rustdoc test suite: clean up wording and intro --- src/rustdoc-internals/htmldocck.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/src/rustdoc-internals/htmldocck.md b/src/rustdoc-internals/htmldocck.md index b37c1ac3d..97c40c8a8 100644 --- a/src/rustdoc-internals/htmldocck.md +++ b/src/rustdoc-internals/htmldocck.md @@ -1,8 +1,10 @@ -# `rustdoc` tests +# the `rustdoc` test suite This page is specifically about the `rustdoc` test suite, for other test suites used for testing rustdoc, see [Rustdoc § Tests](../rustdoc.md#tests). -`htmldocck.py` is a custom checker script that uses [XPath] to verify the HTML output of rustdoc. +The `rustdoc` test suite is specifically used to test the HTML output of rustdoc. + +This is achived by means of `htmldocck.py`, a custom checker script that leverages [XPath]. [XPath]: https://en.wikipedia.org/wiki/XPath @@ -14,7 +16,8 @@ In addition to the directives listed here, [compiletest directives](../tests/directives.html). All `PATH`s in directives are relative to the the rustdoc output directory (`build/TARGET/test/rustdoc/TESTNAME`), -so it is conventional to use a `#![crate_name = "foo"]` attribute to avoid writing paths. +so it is conventional to use a `#![crate_name = "foo"]` attribute to avoid +having to write a long crate name multiple times. To avoid repetion, `-` can be used in any `PATH` argument to re-use the previous `PATH` argument. All arguments take the form of quoted strings, From 174678da355231d76700710caca84b5439474c9c Mon Sep 17 00:00:00 2001 From: binarycat Date: Tue, 18 Mar 2025 13:21:15 -0500 Subject: [PATCH 5/8] rename htmldocck.md -> rustdoc-test-suite.md --- src/SUMMARY.md | 2 +- src/rustdoc-internals/{htmldocck.md => rustdoc-test-suite.md} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename src/rustdoc-internals/{htmldocck.md => rustdoc-test-suite.md} (100%) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index fb4687774..8ea5e86cd 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -95,7 +95,7 @@ - [Parallel Compilation](./parallel-rustc.md) - [Rustdoc internals](./rustdoc-internals.md) - [Search](./rustdoc-internals/search.md) - - [`rustdoc` tests](./rustdoc-internals/htmldocck.md) + - [the `rustdoc` test suite](./rustdoc-internals/rustdoc-test-suite.md) # Source Code Representation - [Prologue](./part-3-intro.md) diff --git a/src/rustdoc-internals/htmldocck.md b/src/rustdoc-internals/rustdoc-test-suite.md similarity index 100% rename from src/rustdoc-internals/htmldocck.md rename to src/rustdoc-internals/rustdoc-test-suite.md From 2e1c4999c812e7c8f5af44a9f902d908c672dfe9 Mon Sep 17 00:00:00 2001 From: binarycat Date: Tue, 18 Mar 2025 13:23:37 -0500 Subject: [PATCH 6/8] clean up wording/grammar and mention double quotes --- src/SUMMARY.md | 2 +- src/rustdoc-internals/rustdoc-test-suite.md | 7 ++++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/src/SUMMARY.md b/src/SUMMARY.md index 8ea5e86cd..075d5af4c 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -95,7 +95,7 @@ - [Parallel Compilation](./parallel-rustc.md) - [Rustdoc internals](./rustdoc-internals.md) - [Search](./rustdoc-internals/search.md) - - [the `rustdoc` test suite](./rustdoc-internals/rustdoc-test-suite.md) + - [The `rustdoc` test suite](./rustdoc-internals/rustdoc-test-suite.md) # Source Code Representation - [Prologue](./part-3-intro.md) diff --git a/src/rustdoc-internals/rustdoc-test-suite.md b/src/rustdoc-internals/rustdoc-test-suite.md index 97c40c8a8..fb4be51c2 100644 --- a/src/rustdoc-internals/rustdoc-test-suite.md +++ b/src/rustdoc-internals/rustdoc-test-suite.md @@ -1,6 +1,6 @@ -# the `rustdoc` test suite +# The `rustdoc` test suite -This page is specifically about the `rustdoc` test suite, for other test suites used for testing rustdoc, see [Rustdoc § Tests](../rustdoc.md#tests). +This page is specifically about the test suite named `rustdoc`, for other test suites used for testing rustdoc, see [Rustdoc Tests](../rustdoc.md#tests). The `rustdoc` test suite is specifically used to test the HTML output of rustdoc. @@ -20,7 +20,8 @@ so it is conventional to use a `#![crate_name = "foo"]` attribute to avoid having to write a long crate name multiple times. To avoid repetion, `-` can be used in any `PATH` argument to re-use the previous `PATH` argument. -All arguments take the form of quoted strings, +All arguments take the form of quoted strings +(both single and double quotes are supported), with the exception of `COUNT` and the special `-` form of `PATH`. Directives are assertions that place constraints on the generated HTML. From 871280d6dfb5c9b6d0962749d9c3c19b839b7872 Mon Sep 17 00:00:00 2001 From: binarycat Date: Tue, 18 Mar 2025 13:25:39 -0500 Subject: [PATCH 7/8] normalize link titles --- src/rustdoc-internals/rustdoc-test-suite.md | 2 +- src/tests/compiletest.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/rustdoc-internals/rustdoc-test-suite.md b/src/rustdoc-internals/rustdoc-test-suite.md index fb4be51c2..c2a2e6495 100644 --- a/src/rustdoc-internals/rustdoc-test-suite.md +++ b/src/rustdoc-internals/rustdoc-test-suite.md @@ -1,6 +1,6 @@ # The `rustdoc` test suite -This page is specifically about the test suite named `rustdoc`, for other test suites used for testing rustdoc, see [Rustdoc Tests](../rustdoc.md#tests). +This page is specifically about the test suite named `rustdoc`, for other test suites used for testing rustdoc, see [Rustdoc tests](../rustdoc.md#tests). The `rustdoc` test suite is specifically used to test the HTML output of rustdoc. diff --git a/src/tests/compiletest.md b/src/tests/compiletest.md index ee66b787b..2c35381ea 100644 --- a/src/tests/compiletest.md +++ b/src/tests/compiletest.md @@ -78,7 +78,7 @@ The following test suites are available, with links for more information: ### Rustdoc test suites -See [Rustdoc § Tests](../rustdoc.md#tests) for more details. +See [Rustdoc tests](../rustdoc.md#tests) for more details. | Test suite | Purpose | |------------------|--------------------------------------------------------------------------| From 59f11cdfe58f0b42525e129320670bb725aaebc8 Mon Sep 17 00:00:00 2001 From: binarycat Date: Tue, 18 Mar 2025 13:29:14 -0500 Subject: [PATCH 8/8] update filename in link --- src/rustdoc.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/rustdoc.md b/src/rustdoc.md index 5ac62aa2c..320dc9d58 100644 --- a/src/rustdoc.md +++ b/src/rustdoc.md @@ -97,7 +97,7 @@ does is call the `main()` that's in this crate's `lib.rs`, though.) * The tests on the structure of rustdoc HTML output are located in `tests/rustdoc`, where they're handled by the test runner of bootstrap and the supplementary script `src/etc/htmldocck.py`. - [These tests have several extra directives available to them](./rustdoc-internals/htmldocck.md). + [These tests have several extra directives available to them](./rustdoc-internals/rustdoc-test-suite.md). [TypeScript-flavored JSDoc]: https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html