permit some no-op escape sequences for compatibility purposes

[KiChjang]

Some of the regexes found in https://github.com/ua-parser/uap-core is throwing errors when parsed with the regex crate:

regex parse error:
    (?:\/[A-Za-z0-9\.]+)? *([A-Za-z0-9 \-_\!\[\]:]*(?:[Aa]rchiver|[Ii]ndexer|[Ss]craper|[Bb]ot|[Ss]pider|[Cc]rawl[a-z]*))/(\d+)(?:\.(\d+)(?:\.(\d+))?)?
       ^^
error: unrecognized escape sequence

This sounds like we're deviating from the regex spec here. Can someone confirm?

[BurntSushi]

Which regex spec are you referring to?

Otherwise, yes, this crate disallows unnecessary escapes. This is to permit the addition of new escapes in a backwards compatible way. It is plausible that we could allow certainly no-op escapes (such as for /), but this may or may not solve the large problem.

[KiChjang]

So I was referring to the ES6 spec for regexes. I believe that there are production-grade projects such as the one I linked in my first post which does contain unnecessary backslashes, and I think this crate should definitely provide a way to accept these regexes as-is, without modifying the regexes contained within to conform to the regex syntax introduced in this crate.

For the project I linked, here's a shortlist of what I did to make it compile with this crate:

• \/ -> /
• \! -> !
• \ ->
• |) -> )? (empty alternations)

[BurntSushi]

So I was referring to the ES6 spec for regexes.

This crate definitely does not, and never will, conform to the ES6 specification for regexes. It is a complete non-goal.

I am much more sympathetic to your practical concerns. I'm generally strongly opposed to allowing escapes to always work even when they are no-ops, but I could get on board with selecting a set of commonly escaped characters in the wild.

It might also be smart to try to fix those projects such that they don't use unnecessary escapes.

Empty alternations are something I'd also like to support, but a bug in the compiler prevents it for now.

[RReverser]

We actually made own Rust wrapper for uap-core, and yes, we had to fix unnecessary escapes. Perhaps we could open-source it to avoid duplication of efforts?

[BurntSushi]

@RReverser Were escapes the only reason for that? Or were there other issues that needed to be papered over?

[RReverser]

the only reason for that

Only reason for what? Implementing Rust version of uap-core?

[BurntSushi]

@RReverser In the process of doing that, you said you had to fix unnecessary escapes. Was there anything else you needed to do with the regexes specifically to make them work with Rust's regex engine?

[RReverser]

Ah. Well, another fix I had to do was to replace \d to match only ASCII digits, since it turned out to include Unicode ones as well, which should not be allowed from the point of UA parser, although in all other regards strings should be still matched in Unicode-aware mode (I suppose you remember our discussion about this).

Other than that, no other fixes were necessary, although I did write a bunch of extra analysis and rewrites using regex-syntax to optimise common cases.

[BurntSushi]

In an effort to keep conversation on this topic in one place, I'm going to respond to @zackw's issue #522 here:

To recap, my high level current thinking on this topic:

I'm generally strongly opposed to allowing escapes to always work even when they are no-ops, but I could get on board with selecting a set of commonly escaped characters in the wild.

The problem here is that I don't know how much effort we should expend to make the syntax compatible with other regex engines. Surely, we can all agree that 100% compatibility can never happen or be expected. So we have to choose some set of features that gets us part of the way there. I just don't know which things to choose.

The purpose for the current behavior is to permit backwards compatible additions to the syntax of regexes. Some regex engines were developed with very little foresight. Python's regex engine, for example, will permit just about anything to be escaped. Some escapes are significant, but the escapes that aren't significant behave as if they weren't escaped at all. This makes it impossible to add new escape sequences since existing escapes are valid and have specified match semantics.

This regex library chooses to return an error for non-significant escape sequences precisely because we consider turning invalid syntax into valid syntax to be a backwards compatible change, and we can legitimately get away with that by enforcing it.

This framework does not specifically forbid insignificant escape sequences. We can simply choose the ones we want to explicitly allow. , " and ' are all candidates, but they are far from the only ones. Moreover, even if we were to add new escape sequences in the future, it would probably be poor judgment to use an escape sequence that is commonly used in other regex engines as an insignificant escape sequence. e.g., Prescribing special meaning to \" while another regex engine just treats it as a literal " is probably poor form.

I think the most compelling use cases, from my perspective, are huge libraries of regexes. However, in practice, it seems quite difficult to just expect to be able to compile them without any other changes. As @RReverser notes above, the \d, \w and \s escape sequences are all Unicode aware by default, which is not common in other regex engines, which were likely built in a time before Unicode was as widespread as it is today. Therefore, even if we fix the cases of escape sequences, you still wind up with subtle match differences. If pressed, I am sure I could come up with a list of several more cases. What I'm trying to say here is that it may be unreasonable to expect a large existing library of regexes---written specifically for one particular regex engine---to just work out of the box on a different regex engine, even if it might in practice in some number cases.

[zackw]

I understand and appreciate your position that no-op escapes should not be added just because they are no-op escapes in other regex engines.

I would like to offer an independent argument for no-op \", \' and \/ (didn't think of \/ before, but yes, that one too) based on the fact that ", ', and / are commonly used to delimit regex literals in many different languages and contexts, and \", \', \/ are commonly understood to escape the delimitation (i.e. extend the regex literal past a point where it would otherwise have ended). In many cases, backslashes that escape delimitation will be stripped by the "outer" parser before the regex engine sees them, but not all (e.g. the Python raw strings I mentioned in #522). Therefore, no-op \", \' and \/ is not just a matter of compatibility with other regex engines, but other surrounding contexts besides Rust source code.

[BurntSushi]

@zackw Ah I see. I don't think I appreciated that point about Python's raw strings when I first read it in #522. Thanks for mentioning that again.

OK, so how about we start off by making ", ', (space, 0x20), / and ! escapeable but no-op? This will also cement the syntax such that these characters can never be used as a valid escape sequence that does anything other than match the literal being escaped. I think for these characters, that would be reasonable.

We can add more no-op escapes moving forward if they creep up, but I think these are probably the ones I see escaped most often.