Skip to content

source_span_for_markdown_range: fix utf8 violation #141666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 1, 2025
Merged

source_span_for_markdown_range: fix utf8 violation #141666

merged 1 commit into from
Jun 1, 2025

Conversation

lolbinarycat
Copy link
Contributor

@lolbinarycat lolbinarycat commented May 27, 2025 •
edited
Loading

it is non-trivial to reproduce this bug through rustdoc, which uses this function less than clippy, so the regression test was added as a unit test instead of an integration test.

fixes #141665

r? @GuillaumeGomez

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels May 27, 2025
@GuillaumeGomez
Copy link
Member

We need a regression test.

@lolbinarycat
Copy link
Contributor Author

yes, I was under the impression you were working on that (well, a MCVE for the issue). if you want me to do it, i can.

@GuillaumeGomez
Copy link
Member

I triggered the issue with sysinfo when running clippy. And I was planning on doing that tomorrow, I'm supposed to sleep right now. ^^'

@lolbinarycat
Copy link
Contributor Author

Alright, I think I should be able to throw something together.

@rust-log-analyzer

This comment has been minimized.

Sign in to view
@lolbinarycat
Copy link
Contributor Author

my initial ideas for how to trigger this crash were unsuccessful, might be easier to just throw treereduce-rustc at it or something.

also not sure if you had the bug happen on unmodified sysinfo or if there was some change you made.

@fmease
Copy link
Member

fmease commented May 27, 2025

See #141665 (comment) for a minimal repro involving clippy-driver. Fn check_attrs in src/tools/clippy/clippy_lints/src/doc/mod.rs does a bunch of pre- and re-processing before/while using the methods exposed in rustc_resolve::rustdoc, so y'all need to figure out if this can even repro under rustc/rustdoc.

@fmease fmease added T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue. S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 28, 2025
@lolbinarycat lolbinarycat force-pushed the rustdoc-source_span_for_markdown_range-bug-141665 branch from 3194bc1 to 74b1eb3 Compare May 29, 2025 18:56
@lolbinarycat
Copy link
Contributor Author

@fmease do you think we could just add a regression test to the clippy test suite? either as a followup PR or via submodules.

@GuillaumeGomez
Copy link
Member

Isn't it possible to create a span in a unit test and then run our function on it?

@lolbinarycat
Copy link
Contributor Author

A unit test would almost certainly work, but I'm not sure the proper way to get a TyCtx for testing.

GuillaumeGomez
GuillaumeGomez reviewed May 29, 2025
View reviewed changes
compiler/rustc_resolve/src/rustdoc.rs Outdated
@@ -570,7 +570,12 @@ pub fn source_span_for_markdown_range(
{
// If there is either a match in a previous fragment, or
// multiple matches in this fragment, there is ambiguity.
if match_data.is_none() && !snippet[match_start + 1..].contains(pat) {
// the snippet cannot be zero-sized,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also please don't split your comments so much. It makes much harder to read. You have up to 100 columns, make use of it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to use semantic line breaks, but maybe i went a bit overboard. I also am using a laptop with a fairly small screen.

@GuillaumeGomez
Copy link
Member

That's a good point. I suppose there are such unit tests in the compiler making use of TyCtxt internally? Maybe try to check that.

@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@lolbinarycat
Copy link
Contributor Author

The new code should have exactly identical behavior to the old code, but without the crash.

I thought I fixed the inconsistentcy, but the tests are still failing, which means something changed which shouldn't have.

@lolbinarycat
Copy link
Contributor Author

lolbinarycat commented May 31, 2025 •
edited
Loading

Ok, it looks like initializing TyCtx for a unit test is prohibativly complex, but the good news is we don't actually need TyCtx, we just need SourceMap, which is much simpler. We also need the ability to get Symbols, but I think there's a helper function we already have for other rustdoc tests?

EDIT: the symbols are never read, so we can just use placeholder values

@lolbinarycat
Copy link
Contributor Author

Ok, added unit tests and fixed my very silly mistake, will squash once CI passes.

also, sidenote, I think the single raw doc fragment case might be broken. a test like #[doc = " a b"]` should show if this is the case, since the heuristic I added won't be able to find out which backtick is which.

@lolbinarycat
source_span_for_markdown_range: fix utf8 violation
a8b5e70 
it is non-trivial to reproduce this bug through rustdoc,
which uses this function less than clippy,
so the regression test was added as a unit test
instead of an integration test.
@lolbinarycat lolbinarycat force-pushed the rustdoc-source_span_for_markdown_range-bug-141665 branch from 9d48ae2 to a8b5e70 Compare May 31, 2025 19:51
@GuillaumeGomez
Copy link
Member

Nice, thanks!

@bors r+ rollup

@bors
Copy link
Collaborator

bors commented Jun 1, 2025

📌 Commit a8b5e70 has been approved by GuillaumeGomez

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jun 1, 2025
GuillaumeGomez added a commit to GuillaumeGomez/rust that referenced this pull request Jun 1, 2025
@GuillaumeGomez
Rollup merge of rust-lang#141666 - lolbinarycat:rustdoc-source_span_f…
d1fa33d 
…or_markdown_range-bug-141665, r=GuillaumeGomez

source_span_for_markdown_range: fix utf8 violation

it is non-trivial to reproduce this bug through rustdoc, which uses this function less than clippy, so the regression test was added as a unit test instead of an integration test.

fixes rust-lang#141665

r? `@GuillaumeGomez`
This was referenced Jun 1, 2025
Closed
Merged
bors added a commit that referenced this pull request Jun 1, 2025
@bors
Auto merge of #141869 - GuillaumeGomez:rollup-vicg807, r=GuillaumeGomez
99e7c15 
Rollup of 9 pull requests

Successful merges:

 - #140370 (Improve diagnostics for usage of qualified paths within tuple struct exprs/pats)
 - #141224 (terminology: allocated object → allocation)
 - #141622 (implement `va_arg` for `powerpc`)
 - #141666 (source_span_for_markdown_range: fix utf8 violation)
 - #141789 (Exclude `CARGO_HOME` from `generate-copyright` in-tree determination)
 - #141823 (Drive-by refactor: use `OnceCell` for the reverse region SCC graph)
 - #141834 (Add unimplemented `current_dll_path()` for WASI)
 - #141846 (Fix TLS model on bootstrap for cygwin)
 - #141852 (resolve if-let-chain FIXME on bootstrap)

r? `@ghost`
`@rustbot` modify labels: rollup
@bors bors merged commit d843809 into rust-lang:master Jun 1, 2025
9 checks passed
@rustbot rustbot added this to the 1.89.0 milestone Jun 1, 2025
rust-timer added a commit that referenced this pull request Jun 1, 2025
@rust-timer
Unrolled build for #141666
f0d939e 
Rollup merge of #141666 - lolbinarycat:rustdoc-source_span_for_markdown_range-bug-141665, r=GuillaumeGomez

source_span_for_markdown_range: fix utf8 violation

it is non-trivial to reproduce this bug through rustdoc, which uses this function less than clippy, so the regression test was added as a unit test instead of an integration test.

fixes #141665

r? ``@GuillaumeGomez``
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@GuillaumeGomez GuillaumeGomez GuillaumeGomez approved these changes

Assignees

@GuillaumeGomez GuillaumeGomez

Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. T-rustdoc Relevant to the rustdoc team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
1.89.0
Development

Successfully merging this pull request may close these issues.

Panic in source_span_for_markdown_range
6 participants
@lolbinarycat @GuillaumeGomez @rust-log-analyzer @fmease @bors @rustbot