Skip to content

Emit custom character classes like an alternation #590

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 27 commits into from
Closed

Emit custom character classes like an alternation #590

wants to merge 27 commits into from

Conversation

rctcwyvrn
Copy link
Contributor

@rctcwyvrn rctcwyvrn commented Jul 26, 2022 •
edited
Loading

I thought this would be relatively easy to implement and I was curious what the performance benefits were so I implemented it quickly

A number of regressions in benchmarks without any code that hits any of the changes (most of these don't even have a custom character class). Most likely due to the instruction I added?

As expected EmailRFCNoMatches is much faster now that it's non-ascii CCC has it's ascii members collected into a bitset

The big benefit is being able to remove all the code in ConsumerInterface that did matching on things we already have instructions for (characters, scalars, case insensitive, etc)

Based on #547

Comparing against benchmark result file before.json
=== Regressions ======================================================================
- EmailLookaheadAll                       88.6ms	85.8ms	2.84ms		3.3%
- DiceRollsInTextAll                      68.7ms	66ms	2.66ms		4.0%
- EmailLookaheadNoMatchesAll              63.2ms	61.1ms	2.09ms		3.4%
- BasicBuiltinCharacterClassAll           16.2ms	14.9ms	1.32ms		8.9%
- EmailLookaheadList                      24.5ms	23.4ms	1.04ms		4.4%
- EmailBuiltinCharacterClassAll           26.2ms	25.1ms	1.02ms		4.1%
- NumbersAll                              11.3ms	10.3ms	986µs		9.6%
- InvertedCCC                             29.6ms	28.9ms	668µs		2.3%
- WordsAll                                26.3ms	25.9ms	411µs		1.6%
- AnchoredNotFoundWhole                   9.73ms	9.41ms	328µs		3.5%
=== Improvements =====================================================================
- EmailRFCNoMatchesAll                    62.8ms	106ms	-43.1ms		-40.7%
- symDiffCCC                              19.4ms	40.7ms	-21.4ms		-52.5%
- IntersectionCCC                         12ms	16.2ms	-4.2ms		-26.0%
- SubtractionCCC                          11.7ms	15.7ms	-3.92ms		-25.0%
- EmailRFCAll                             48.5ms	50.6ms	-2.14ms		-4.2%
- EagarQuantWithTerminalWhole             7.75ms	8.41ms	-663µs		-7.9%

rctcwyvrn added 27 commits July 5, 2022 14:21
@rctcwyvrn
Copy over new ascii bitset
3b6b676
@rctcwyvrn
Add matchBuiltin
33caa79
@rctcwyvrn
Remove debug prints
139daa5
@rctcwyvrn
Remove bitset fast path
9abf4af
@rctcwyvrn
Fully remove remnants of the bitset fast path
286f5d8
@rctcwyvrn
Merge branch 'main' into speedy-builtins
9e915cd
@rctcwyvrn
Completely replace AssertionFunction with regexAssert(by:)
e593ddb
@rctcwyvrn
Merge branch 'main' into speedy-builtins
25dc277
@rctcwyvrn
Cleanup
3e38ac6
@rctcwyvrn
Move match builtin and assert + Add AssertionPayload
e5d8b4a
@rctcwyvrn
Cleanup assertions
0466c25
@rctcwyvrn
Merge branch 'main' into speedy-builtins
87078ad
@rctcwyvrn
Fix tests
f401e84
@rctcwyvrn
Update opcode description for assertBy
b09f45f
@rctcwyvrn
Merge branch 'main' into speedy-builtins
c581ea2
@rctcwyvrn
Merge branch 'main' into speedy-builtins
2a82231
@rctcwyvrn
Update branch to match main
fb1576a
@rctcwyvrn
Use the newly cleaned up _CharacterClassModel
3b9485e
@rctcwyvrn
Add characterClass DSLTree node
64d1ed9
@rctcwyvrn
Bugfixes
2a6fe3c 
- matchBuiltin always fails if at endIndex
- fix switch in isStrictAscii
@rctcwyvrn
Add documentation for matchBuiltin
206bfc6
@rctcwyvrn
Lots of cleanup
b53f524 
- static vars in payloads
- Clean up _CharacterClassModel
- Use the model for bytecodegen and consumer interface
- Merge the grapheme and scalar match builtin cases together
@rctcwyvrn
Move assertion payload
bb5245f
@rctcwyvrn
More minor cleanup
0746847
@rctcwyvrn
Perform boundary check for .anyScalar when in grapheme mode
c718543
@rctcwyvrn
Emit custom character classes via saves and branches
3f0ece5 
- Removes the main consumer interface for ccc
- Removes a lot of the consumer interface code required for ccc
- Adds an optimization for collecting the ascii parts of a ccc
- Use normal matching code in a CCC
@rctcwyvrn
Add some comments
79aabab
hamishknight
hamishknight reviewed Jul 26, 2022
View reviewed changes
Sources/_StringProcessing/ByteCodeGen.swift
Comment on lines +1031 to +1033
/// We allow trivia into CustomCharacterClass, which could result in a CCC that matches nothing
/// ie (?x)[ ]
var guaranteesForwardProgress: Bool {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we should reject (?x)[ ] at parse time (we should reject a character class with no semantic members). That being said, it is possible to have a custom character class that matches nothing, e.g [a--a] (though as I understand it, that would still guarantee forward progress).

@rctcwyvrn rctcwyvrn changed the title Replace custom character classes with an alternation Emit custom character classes like an alternation Jul 26, 2022
natecook1000 added a commit to natecook1000/swift-experimental-string-processing that referenced this pull request Apr 13, 2023
@natecook1000
Remove most consumer functions
1c23c56 
This is based heavily off the work in swiftlang#590, rebased onto main, with
some changes to remove even more consumer uses. Consumer functions
only have two remaining uses: non-ASCII ranges and Unicode lookups
(for things like general category, binary properties, name, etc.).

This change primarily treats custom character classes as alternations
around their contents, with set operations emitted as instructions
instead of implemented via consumer function.
natecook1000 added a commit to natecook1000/swift-experimental-string-processing that referenced this pull request Apr 13, 2023
@natecook1000
Remove most consumer functions
df7fc51 
This is based heavily off the work in swiftlang#590, rebased onto main, with
some changes to remove even more consumer uses. Consumer functions
only have two remaining uses: non-ASCII ranges and Unicode lookups
(for things like general category, binary properties, name, etc.).

This change primarily treats custom character classes as alternations
around their contents, with set operations emitted as instructions
instead of implemented via consumer function.
natecook1000 added a commit to natecook1000/swift-experimental-string-processing that referenced this pull request Apr 13, 2023
@natecook1000
Remove most consumer functions
d157764 
This is based heavily off the work in swiftlang#590, rebased onto main, with
some changes to remove even more consumer uses. Consumer functions
only have two remaining uses: non-ASCII ranges and Unicode lookups
(for things like general category, binary properties, name, etc.).

This change primarily treats custom character classes as alternations
around their contents, with set operations emitted as instructions
instead of implemented via consumer function.
natecook1000 added a commit to natecook1000/swift-experimental-string-processing that referenced this pull request Apr 14, 2023
@natecook1000
Remove most consumer functions
c85d1f3 
This is based heavily off the work in swiftlang#590, rebased onto main, with
some changes to remove even more consumer uses. Consumer functions
only have two remaining uses: non-ASCII ranges and Unicode lookups
(for things like general category, binary properties, name, etc.).

This change primarily treats custom character classes as alternations
around their contents, with set operations emitted as instructions
instead of implemented via consumer function.
natecook1000 added a commit to natecook1000/swift-experimental-string-processing that referenced this pull request Apr 14, 2023
@natecook1000
Remove most consumer functions
60b4d78 
This is based heavily off the work in swiftlang#590, rebased onto main, with
some changes to remove even more consumer uses. Consumer functions
only have two remaining uses: non-ASCII ranges and Unicode lookups
(for things like general category, binary properties, name, etc.).

This change primarily treats custom character classes as alternations
around their contents, with set operations emitted as instructions
instead of implemented via consumer function.
@natecook1000 natecook1000 mentioned this pull request Apr 14, 2023
Merged
@natecook1000
Copy link
Member

Superseded by #660.

natecook1000 added a commit that referenced this pull request Mar 27, 2024
@natecook1000
Remove most consumer functions (#660)
12db649 
This is based heavily off the work in #590, rebased onto main, with
some changes to remove even more consumer uses. Consumer functions
only have two remaining uses: non-ASCII ranges and Unicode lookups
(for things like general category, binary properties, name, etc.).

This change primarily treats custom character classes as alternations
around their contents, with set operations emitted as instructions
instead of implemented via consumer function.
natecook1000 added a commit that referenced this pull request Mar 27, 2024
@natecook1000
Remove most consumer functions (#660)
a79f1cb 
This is based heavily off the work in #590, rebased onto main, with
some changes to remove even more consumer uses. Consumer functions
only have two remaining uses: non-ASCII ranges and Unicode lookups
(for things like general category, binary properties, name, etc.).

This change primarily treats custom character classes as alternations
around their contents, with set operations emitted as instructions
instead of implemented via consumer function.
@natecook1000 natecook1000 mentioned this pull request Mar 27, 2024
Merged
natecook1000 added a commit that referenced this pull request Mar 27, 2024
@natecook1000
Remove most consumer functions (#660) (#733)
d09a6b6 
This is based heavily off the work in #590, rebased onto main, with
some changes to remove even more consumer uses. Consumer functions
only have two remaining uses: non-ASCII ranges and Unicode lookups
(for things like general category, binary properties, name, etc.).

This change primarily treats custom character classes as alternations
around their contents, with set operations emitted as instructions
instead of implemented via consumer function.
natecook1000 added a commit that referenced this pull request Mar 27, 2024
@natecook1000
Remove most consumer functions (#660) (#733)
fac8943 
This is based heavily off the work in #590, rebased onto main, with
some changes to remove even more consumer uses. Consumer functions
only have two remaining uses: non-ASCII ranges and Unicode lookups
(for things like general category, binary properties, name, etc.).

This change primarily treats custom character classes as alternations
around their contents, with set operations emitted as instructions
instead of implemented via consumer function.
@natecook1000 natecook1000 mentioned this pull request Mar 29, 2024
Merged
natecook1000 added a commit that referenced this pull request Apr 1, 2024
@natecook1000
Remove most consumer functions (#734)
3719a06 
This is based heavily off the work in #590, rebased onto main, with
some changes to remove even more consumer uses. Consumer functions
only have two remaining uses: non-ASCII ranges and Unicode lookups
(for things like general category, binary properties, name, etc.).

This change primarily treats custom character classes as alternations
around their contents, with set operations emitted as instructions
instead of implemented via consumer function.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hamishknight hamishknight hamishknight left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@rctcwyvrn @natecook1000 @hamishknight