Update RegexSyntax.md

hamishknight · hamishknight · commit 442c53ffeaf3 · 2022-02-16T13:48:39.000Z
diff --git a/Documentation/Evolution/RegexSyntax.md b/Documentation/Evolution/RegexSyntax.md
@@ -26,14 +26,22 @@ We also intend to achieve at least Level 1 (**TODO: do we want to promise Level
 
 We're proposing the following regular expression syntactic superset for Swift.
 
+### Top-level regular expression
+
+```
+Regex     -> GlobalMatchingOptionSequence? RegexNode
+RegexNode -> '' | Alternation
+```
+
+A top-level regular expression may consist of a sequence of global matching options followed by a `RegexNode`, which is the recursive part of the grammar that may be nested within e.g a group.
+
 ### Alternation
 
 ```
-Regex       -> '' | Alternation
 Alternation -> Concatenation ('|' Concatenation)*
 ```
 
-This is the operator with the lowest precedence in a regular expression, and checks if any of its branches match the input.
+The `|` operator denotes what is formally called an alternation, or a choice between alternatives. Any number of alternatives may appear, including empty alternatives. This operator has the lowest precedence of all operators in a regex literal.
 
 ### Concatenation
 
@@ -42,38 +50,67 @@ Concatenation   -> (!'|' !')' ConcatComponent)*
 ConcatComponent -> Trivia | Quote | Quantification
 ```
 
-Implicitly denoted by adjacent expressions, a concatenation matches against a sequence of regular expression patterns. This has a higher precedence than an alternation, so e.g `abc|def` matches against `abc` or `def`. The `ConcatComponent` token varies across engine, but at least matches some form of trivia, e.g comments, quoted sequences e.g `\Q...\E`, and a quantified expression.
+Implicitly denoted by adjacent expressions, a concatenation matches against a sequence of regular expression patterns. This has a higher precedence than an alternation, so e.g `abc|def` matches against `abc` or `def`. The `ConcatComponent` token varies across engine, but at least matches some form of trivia, e.g comments, quoted sequences e.g `\Q...\E`, and a potentially quantified expression.
 
 ### Quantification
 
 ```
 Quantification -> QuantOperand Quantifier?
-Quantifier     -> ('*' | '+' | '?' | '{' Range '}') QuantKind?
+Quantifier     -> QuantAmount QuantKind?
+QuantAmount    -> '?' | '*' | '+' | '{' Range '}'
 QuantKind      -> '?' | '+'
+Range          -> ',' <Int> | <Int> ',' <Int>? | <Int>
+
+QuantOperand -> AbsentFunction | Atom | Conditional | CustomCharClass | Group
 ```
 
-Specifies that the operand may be matched against a certain number of times.
+A quantification consists of an operand optionally followed by a quantifier that specifier how many times it may be matched. An operand without a quantifier is matched once.
+
+The quantifiers supported are:
+
+- `?`: 0 or 1 matches
+- `*`: 0 or more matches
+- `+`: 1 or more matches
+- `{n,m}`: Between `n` and `m` (inclusive) matches
+- `{n,}`: `n` or more matches
+- `{,m}`: Up to `m` matches
+- `{n}`: Exactly `n` matches
+
+A quantifier may optionally followed by `?` or `+`, which apply certain semantics to the quantification. If neither are specified, by default the quantification happens eagerly, meaning that it will try to maximize the number of matches made. However, if `?` is specified, the number of matches will instead be minimized. If `+` is specified, eager matching occurs, but with the additional semantic that it may not be backtracked into to try a different number of matches.
+
+### Atom
 
-**TODO: Briefly mention each and what it means, noting that options can swap eager/reluctant. Might be a good time to introduce the eager/reluctant/possessive terminology**
+```
+Atom -> Anchor | EscapeSequence | BuiltinCharClass
+```
+
+Atoms are the smallest unit of regular expression syntax that cannot be split into smaller syntactic expressions.
 
 ### Groups
 
 ```
-GroupStart    -> '(?' GroupKind | '('
-GroupKind     -> ':' | '|' | '>' | '=' | '!' | '*' | '<=' | '<!' | '<*'
-               | NamedGroup | MatchingOptionSeq (':' | ')')
-            
-NamedGroup    -> 'P<' GroupNameBody '>'
-               | '<' GroupNameBody '>'
-               | "'" GroupNameBody "'"
+Group      -> GroupStart RegexNode ')'
+GroupStart -> '(?' GroupKind | '('
+GroupKind  -> ':' | '|' | '>' | '=' | '!' | '*' | '<=' | '<!' | '<*'
+            | NamedGroup | MatchingOptionSeq (':' | ')')
+
+NamedGroup -> 'P<' GroupNameBody '>'
+            | '<' GroupNameBody '>'
+            | "'" GroupNameBody "'"
 
 GroupNameBody -> Identifier | BalancingGroupBody
+
+Identifier -> [\w--\d] \w*
 ```
 
 Groups define a new scope within which a recursive regular expression pattern may occur. Groups have different semantics depending on how they are introduced, some may capture the nested match, some may match against the input without advancing, some may change the matching options set in the new scope, etc.
 
 **TODO: Something like "note that there are other things that may syntactically appear similarly to groups, but are their own constructs. See .... in-line options, backreferences, ... **
 
+#### Lookahead and lookbehind
+
+#### Script runs
+
 #### Balancing groups
 
 ```
@@ -86,15 +123,39 @@ Introduced by .NET, balancing groups extend the `GroupNameBody` syntax to suppor
 ### Anchors
 
 ```
-Anchor -> '^' | '$' | '\b' | '\B' | '\A' | '\G' | '\z' | '\Z'
+Anchor -> '^' | '$' | '\A' | '\b' | '\B' | '\G' | '\y' | '\Y' | '\z' | '\Z'
 ```
 
 Anchors match against a certain position in the input rather than on a particular character of the input.
 
+#### Start and end of line
+
+`^` matches against the start of a line of input, `$` matches against the end of a line.
+
+#### Word boundaries
+
+`\b` matches a word boundary, which is [...]
+
 **TODO: List these out with a very brief description of what they mean.**
 
 ### Unicode scalars
 
+```
+UniScalar -> '\u{' HexDigit{1...} '}'
+           | '\u'  HexDigit{4}
+           | '\x{' HexDigit{1...} '}'
+           | '\x'  HexDigit{0...2}
+           | '\U'  HexDigit{8}
+           | '\o{' OctalDigit{1...} '}'
+           | '\' OctalDigit{1...3}
+
+HexDigit    -> [0-9a-zA-Z]
+OctalDigit  -> [0-7]
+```
+
+These sequences define a unicode scalar value to be matched against.
+
+**TODO: Some discussion of the fun `\DDD` syntax**
 
 ### Escape sequences
 
@@ -107,7 +168,7 @@ These escape sequences denote a specific character. Note that `\b` may only be u
 ### Builtin character classes
 
 ```
-BuiltinCharClass -> '\d' | '\D' | '\h' | '\H' | '\R' | '\s' | '\S' | '\v' | '\V' | '\w' | '\W' | '\X'
+BuiltinCharClass -> '.' | '\d' | '\D' | '\h' | '\H' | '\O' | '\R' | '\s' | '\S' | '\v' | '\V' | '\w' | '\W' | '\X'
 ```
 
 ### Custom character classes
@@ -125,13 +186,155 @@ Custom characters classes introduce their own language, in which most regular ex
 
 ### Character properties
 
+```
+CharacterProperty -> ('p{' | 'P{') PropertyName ('=' PropertyName)? '}'
+PropertyName -> [\s\w-]+
+```
+
+### Named characters
+
+```
+NamedCharacter -> '\N{' CharName '}'
+CharName -> 'U+' HexDigit{1...8} | [\s\w-]+
+```
+
+### Trivia
+
+```
+Trivia     -> Comment | Whitespace
+Comment    -> '(?#' (!')')* ')'
+Whitespace -> \s+
+```
+
+Trivia is consumed by the regular expression parser, but has no semantic meaning. Non-semantic whitespace may only occur when the either of the extended syntax matching options `(?x)`, `(?xx)` are enabled.
+
+### Matching options
+
+```
+MatchingOptionSeq -> '^' MatchingOption* 
+                   | MatchingOption+ 
+                   | MatchingOption* '-' MatchingOption*
+
+MatchingOption -> 'i' | 'J' | 'm' | 'n' | 's' | 'U' | 'x' | 'xx' | 'w' | 'D' | 'P' | 'S' | 'W' | 'y{' ('g' | 'w') '}'
+```
+
+### References
+
+```
+NamedRef       -> Identifier
+NumberRef      -> ('+' | '-')? <Decimal Number> RecursionLevel?
+RecursionLevel -> '+' <Int> | '-' <Int>
+```
+
+#### Backreferences
+
+```
+Backreference -> '\g{' NameOrNumberRef '}'
+               | '\g' NumberRef
+               | '\k<' Identifier '>'
+               | "\k'" Identifier "'"
+               | '\k{' Identifier '}'
+               | '\' [1-9] [0-9]+
+               | '(?P=' Identifier ')'
+```
+
+#### Subpatterns
+
+```
+Subpattern -> '\g<' NameOrNumberRef '>' 
+            | "\g'" NameOrNumberRef "'"
+            | '(?' GroupLikeSubpatternBody ')'
+            
+GroupLikeSubpatternBody -> 'P>' <String>
+                         | '&' <String>
+                         | 'R'
+                         | NumberRef
+```
+
+### Conditionals
+
+```
+Conditional      -> ConditionalStart Concatenation ('|' Concatenation)? ')'
+ConditionalStart -> KnownConditionalStart | GroupConditionalStart
+
+KnownConditionalStart -> '(?(' KnownCondition ')'
+GroupConditionalStart -> '(?' GroupStart
+
+KnownCondition -> 'R'
+                | 'R' NumberRef
+                | 'R&' <String> !')'
+                | '<' NameRef '>'
+                | "'" NameRef "'"
+                | 'DEFINE'
+                | 'VERSION' VersionCheck
+                | NumberRef
+                | NameRef
+                
+PCREVersionCheck  -> '>'? '=' PCREVersionNumber
+PCREVersionNumber -> <Int> '.' <Int>
+```
+
+### PCRE backtracking directives
+
+```
+BacktrackingDirective     -> '(*' BacktrackingDirectiveKind (':' <String>)? ')'
+BacktrackingDirectiveKind -> 'ACCEPT' | 'FAIL' | 'F' | 'MARK' | '' | 'COMMIT' | 'PRUNE' | 'SKIP' | 'THEN'
+```
+
+### PCRE global matching options
+
+```
+GlobalMatchingOptionSequence -> GlobalMatchingOption+
+GlobalMatchingOption -> '(*' GlobalMatchingOptionKind ')'
+
+GlobalMatchingOptionKind -> LimitOptionKind '=' <Int>
+                          | NewlineKind | NewlineSequenceKind
+                          | 'NOTEMPTY_ATSTART' | 'NOTEMPTY'
+                          | 'NO_AUTO_POSSESS' | 'NO_DOTSTAR_ANCHOR'
+                          | 'NO_JIT' | 'NO_START_OPT' | 'UTF' | 'UCP'
+  
+LimitOptionKind     -> 'LIMIT_DEPTH' | 'LIMIT_HEAP' | 'LIMIT_MATCH'
+NewlineKind         -> 'CRLF' | 'CR' | 'ANYCRLF' | 'ANY' | 'LF' | 'NUL'
+NewlineSequenceKind -> 'BSR_ANYCRLF' | 'BSR_UNICODE'
+```
 
 ### Callouts
 
+```
+Callout -> PCRECallout | OnigurumaCallout
+
+PCRECallout -> '(?C' CalloutBody ')'
+PCRECalloutBody -> '' | <Number>
+                 | '`' <String> '`'
+                 | "'" <String> "'"
+                 | '"' <String> '"'
+                 | '^' <String> '^'
+                 | '%' <String> '%'
+                 | '#' <String> '#'
+                 | '$' <String> '$'
+                 | '{' <String> '}'
+                 
+OnigurumaCallout -> OnigurumaNamedCallout | OnigurumaCalloutOfContents
+
+OnigurumaNamedCallout   -> '(*' Identifier OnigurumaTag? OnigurumaCalloutArgs? ')'
+OnigurumaCalloutArgs    -> '{' OnigurumaCalloutArgList '}'
+OnigurumaCalloutArgList -> OnigurumaCalloutArg (',' OnigurumaCalloutArgList)*
+OnigurumaCalloutArg     -> [^,}]+
+OnigurumaTag            -> '[' Identifier ']'
+
+OnigurumaCalloutOfContents -> '(?' '{'+ Contents '}'+ OnigurumaTag? Direction? ')'
+OnigurumaCalloutContents   -> <String>
+OnigurumaCalloutDirection  -> 'X' | '<' | '>'
+```
 
 ### Absent functions
 
-
+```
+AbsentFunction -> '(?~' RegexNode ')'
+                | '(?~|' Concatenation '|' Concatenation ')'
+                | '(?~|' Concatenation ')'
+                | '(?~|)'
+```
 
 ## Syntactic differences between engines
 
@@ -141,7 +344,7 @@ In a custom character class, some engines allow for binary set operations that t
 
 | PCRE | ICU | UTS#18 | Oniguruma | .NET | Java |
 |------|-----|--------|-----------|------|------|
-| ❌ | Intersection `&&`, Subtraction `--` | Intersection & Subtraction | Intersection `&&` | Subtraction via `-` | Intersection  `&&` |
+| ❌ | Intersection `&&`, Subtraction `--` | Intersection, Subtraction | Intersection `&&` | Subtraction via `-` | Intersection  `&&` |
 
 [UTS#18][uts18] requires intersection and subtraction, and uses the operation spellings `&&` and `--` in its examples, though it doesn't mandate a particular spelling. In particular, conforming implementations could spell the subtraction `[[x]--[y]]` as `[[x]&&[^y]]`. UTS#18 also suggests a symmetric difference operator `~~`, and uses an explicit `||` operator in examples, though doesn't require either operations.
 
