Skip to content

Support text segment boundary anchors #178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Mar 3, 2022
Merged

Support text segment boundary anchors #178

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
50ee12e
Support text segment boundary anchors
natecook1000 Feb 22, 2022
13524a9
Switch primitive RegexProtocol conformances to DSLTree
natecook1000 Feb 24, 2022
6b82875
Update Sources/_StringProcessing/ByteCodeGen.swift
natecook1000 Mar 2, 2022
b1ca05f
Update Sources/_StringProcessing/ByteCodeGen.swift
natecook1000 Mar 2, 2022
891e772
Merge branch 'main' into ahoy_grapheme_boundary
natecook1000 Mar 2, 2022
92d1d7f
Update to new test method signature
natecook1000 Mar 2, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions Sources/_StringProcessing/ByteCodeGen.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,12 +99,16 @@ extension Compiler.ByteCodeGen {
}

case .textSegment:
// This we should be able to do!
throw Unsupported(#"\y (text segment)"#)
builder.buildAssert { (input, pos, _) in
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these anchors' semantics dependent on options?

CC @hamishknight

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the y{g} and y{w} matching options flip the behavior between matching extended grapheme cluster boundaries and word boundaries respectively (with the former being the default). Though it sounds like that's what the TODO is referencing.

// FIXME: Grapheme or word based on options
input.isOnGraphemeClusterBoundary(pos)
}

case .notTextSegment:
// This we should be able to do!
throw Unsupported(#"\Y (not text segment)"#)
builder.buildAssert { (input, pos, _) in
// FIXME: Grapheme or word based on options
!input.isOnGraphemeClusterBoundary(pos)
}

case .startOfLine:
builder.buildAssert { (input, pos, bounds) in
Expand Down
16 changes: 11 additions & 5 deletions Sources/_StringProcessing/RegexDSL/DSL.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,25 +17,31 @@ extension String: RegexProtocol {
public typealias Match = Substring

public var regex: Regex<Match> {
let atoms = self.map { atom(.char($0)) }
return .init(ast: concat(atoms))
.init(node: .quotedLiteral(self))
}
}

extension Substring: RegexProtocol {
public typealias Match = Substring

public var regex: Regex<Match> {
let atoms = self.map { atom(.char($0)) }
return .init(ast: concat(atoms))
.init(node: .quotedLiteral(String(self)))
}
}

extension Character: RegexProtocol {
public typealias Match = Substring

public var regex: Regex<Match> {
.init(ast: atom(.char(self)))
.init(node: .atom(.char(self)))
}
}

extension UnicodeScalar: RegexProtocol {
public typealias Match = Substring

public var regex: Regex<Match> {
.init(node: .atom(.scalar(self)))
}
}

Expand Down
16 changes: 14 additions & 2 deletions Tests/RegexTests/MatchTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -876,7 +876,8 @@ extension RegexTests {
#"\d+\b"#,
("123", "123"),
(" 123", "123"),
("123 456", "123"))
("123 456", "123"),
("123A 456", "456"))
firstMatchTests(
#"\d+\b\s\b\d+"#,
("123", nil),
Expand All @@ -892,7 +893,18 @@ extension RegexTests {
// TODO: \G and \K

// TODO: Oniguruma \y and \Y

firstMatchTests(
#"\u{65}"#, // Scalar 'e' is present in both:
("Cafe\u{301}", "e"), // composed and
("Sol Cafe", "e")) // standalone
firstMatchTests(
#"\u{65}\y"#, // Grapheme boundary assertion
("Cafe\u{301}", nil),
("Sol Cafe", "e"))
firstMatchTests(
#"\u{65}\Y"#, // Grapheme non-boundary assertion
("Cafe\u{301}", "e"),
("Sol Cafe", nil))
}

func testMatchGroups() {
Expand Down
10 changes: 10 additions & 0 deletions Tests/RegexTests/RegexDSLTests.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,16 @@ class RegexDSLTests: XCTestCase {
Anchor.endOfLine
}

try _testDSLCaptures(
("Cafe\u{301}", nil),
("Cafe", "Cafe"),
matchType: Substring.self, ==)
{
oneOrMore(.word)
UnicodeScalar("e")
Anchor.textSegmentBoundary
}

try _testDSLCaptures(
("aaaaa1", "aaaaa1"),
("aaaaa2", nil),
Expand Down