Improved delimiter lexing #194
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@swift-ci please test |
milseman
approved these changes
Mar 2, 2022
| enum Delimiter: Hashable, CaseIterable { | ||
| case traditional | ||
| case experimental | ||
| case reSingleQuote |
There was a problem hiding this comment.
How about rxSingleQuote for experimental?
There was a problem hiding this comment.
Ooh yes, thanks for reminding me. How does this look?
| } | ||
| } | ||
|
|
||
| fileprivate struct DelimiterLexer { |
There was a problem hiding this comment.
In the end, we might want something that can feed Source (or even replace Source)
There was a problem hiding this comment.
Yeah would be nice to unify on an implementation that scans and produces unicode scalars for both
To avoid confusion with more general regex lexical analysis.
Introduce a DelimiterLexer type to perform the lexing.
This matches the behavior of the C++ lexer for string literals.
Allow the C++ lexer to form a tok::regex_literal. This avoids generic fallback behavior, and better allows for things like code completion. The test case for this will be in the C++ repo.
If a single quote is encountered with a prefix of either `(?`, `(?(`, `\k`, `\g` or `(?C`, continue to scan ahead to a closing `'`. Such prefixes would not be valid endings for a regex literal anyway, and this lets us handle the single quote variant of their syntax. For the group name cases, further refine this skipping behavior by only skipping over characters that could possibly appear in that case. This improves diagnostic behavior by ensuring we don't go wandering off into Swift code.
hamishknight
force-pushed
the
quoted-in-context
branch
from
b1172a1 to
2325cef
Compare
|
@swift-ci please test |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refactor the delimiter lexing logic, and change the behavior such that:
rx'...'for experimental syntax.Additionally, implement a heuristic that allows skipping over single quotes in
re'...'literals if the preceding characters are(?,(?(,\k,\gor(?C. These would not be valid literal endings anyway, and this allows us to support their single-quoted syntax.This should be able to integrate without any C++ side changes, though I have some extra test cases I want to commit there whenever this gets integrated in.