Proposal: Use I-Regexp instead of ECMA-262

[gregsdennis]

The IETF JSON Path working group is putting together I-Regexp (authored by @cabo) intended as an interoperable subset of the various flavors of regular expressions. Think of it as a "capabilities intersection." The idea is that the spec defines only features which are known to be supported by the majority of existing regular expression libraries, which means that most libraries should be able to claim conformance without actually having to make any changes.

I would like to propose that we use that here instead of ECMA-262, which seems to have varying support across languages. It would likely reduce the set of guaranteed-supported expressions, but at least we would be able to legitimately claim some guaranteed support (which I don't think we can do right now).

This spec is still in draft phase and continues to evolve. I'm not too rushed on getting this in.

[jdesrosiers]

I like the idea. I'm assuming our first stable release will come before this is stable, so I'm wondering if we can adopt this later without breaking compatibility? We might need to consider some wording changes to make space for a future change like this, but I think we can make it work.

[handrews]

This sounded like a good idea until I saw this:

5.3. ECMAScript Regexps
Perform the following steps on an I-Regexp to obtain an ECMAScript regexp [ECMA-262]:

• For any dots (.) outside character classes (first alternative of charClass production): replace dot by [^\n\r].
• Envelope the result in ^ and $.

Note that where a regexp literal is required, the actual regexp needs to be enclosed in /.

Unless I'm missing something that means that I-Regexps are anchored at both ends by default, which would be a huge breaking change. That's a completely different mindset for thinking about regexes, and it's not how they work in Perl, Python, Ruby, or JavaScript.

Please correct me if I'm misunderstanding here, but if this is accurate I'm a no.

[gregsdennis]

I'll bring that up, @handrews, and report back. Good catch.

[handrews]

[EDIT: Never mind this comment, I was reading things wrong]

[cabo]

iregexp is for matching regular expressions, which in practice always require anchors (except for rather unusual cases). Since there is no consensus on anchors, the best approach is to leave out this redundant noise -- note that this is not an innovation as XSD regexps have always done this.
Similarly, "." is a meta character with varying interpretation in different dialects, so translating this to a specific dialect may need interpolation.
None of this should be a big surprise to anyone who actually has tried to obtain fully interoperable regexps.

[handrews]

@cabo the issue at hand is not one of requiring the most interoperable regexes. Schema authors can decide their own tradeoffs for that. The issue is whether to break a feature that has had the same behavior (in terms of anchoring and using ECMA as a reference) since the very beginning of JSON Schema.

[cabo]

Sure. I don't have an opinion on that.

Of course, you can standardize on bracketing an iregexp with ^$ for backwards compatibility (I don't think anyone will notice the fact that "." does not include LS/PS in ECMAscript but does in iregexp). You can also use a different JSON member name to introduce iregexps, but that doesn't help you with the old JSON member name.

[handrews]

(I don't think anyone will notice the fact that "." does not include LS/PS in ECMAscript but does in iregexp)

Oh yeah, I agree, that's why I edited out the comment about "." - I had just read that part wrong. I-Regexp doesn't do what I was worried about with that comment anyway.

But right now {"pattern": "foo"} considers "foobar" to be a valid instance, and if it suddenly considers it invalid because "foo" as a pattern now behaves like "^foo$"... that's going to be a very big surprise to people. And considering that we're trying to stabilize behavior in JSON Schema wherever possible these days, I'm skeptical that it is worth doing.

However, an "iregexp" keyword (or "ipattern" or whatever) might be a good keyword proposal.

While there are always a few people who think regexes should be anchored by default (presumably coming from the XSD world), I'd say the vast majority of people who use pattern understand how it works and would be surprised to migrate to the stable version of JSON Schema and have their pattern-using schemas break.

[tviegut]

(I don't think anyone will notice the fact that "." does not include LS/PS in ECMAscript but does in iregexp)

Oh yeah, I agree, that's why I edited out the comment about "." - I had just read that part wrong. I-Regexp doesn't do what I was worried about with that comment anyway.

But right now {"pattern": "foo"} considers "foobar" to be a valid instance, and if it suddenly considers it invalid because "foo" as a pattern now behaves like "^foo$"... that's going to be a very big surprise to people. And considering that we're trying to stabilize behavior in JSON Schema wherever possible these days, I'm skeptical that it is worth doing.

However, an "iregexp" keyword (or "ipattern" or whatever) might be a good keyword proposal.

While there are always a few people who think regexes should be anchored by default (presumably coming from the XSD world), I'd say the vast majority of people who use pattern understand how it works and would be surprised to migrate to the stable version of JSON Schema and have their pattern-using schemas break.

Hi @handrews , in the IEC standards that are pending publication I know that REGEX patterns are utilized extensively for dates (i.e. date, datetime, duration, month, etc.) to ensure that the string representations are expressed as their respective ISO 8601 compliant equivalents. This since dates are not native primitives in JSON. I recall extensive testing with the French N.C. that I will need to double check on to see where that landed once again. Sorry as I just bumped into this thread now. ~Todd

[gregsdennis]

@tviegut this isn't about regular expressions in general. We want to use and currently are using regex. That's not in question.

This issue is about which flavor of regex we want to support in the spec. Currently, we have ECMA-262, but support for that is inconsistent across languages/platforms. We need something that has guaranteed interoperability.

[handrews]

@gregsdennis I was reading @tviegut's comments as indicating what sort of things might break / need to be updated if we made this change.

@tviegut we do often recommend using regexps for date verification since pattern is more reliably implemented than format (as long as your regexps don't do anything too advanced). So that use case makes sense to me.

@gregsdennis I'm curious about what environments can't (as opposed to just currently don't) support ECMA-262. In particular the anchor thing, since we could easily say "if you anchor your regexes and don't use . you will stay within more broadly interoperable functionality." We've never tried to restrict how much regex functionality implementations offer. We just define what subset is reliable. So we could define the I-Regexp-compatible subset of ECMA 262 instead of just saying ECMA 262 in general. But as a "here's what you can use if you're worried about interoperability", not as a "this is the only thing you can use."

[gregsdennis]

My suggestion isn't based on personal experience, but more on recollection of complaints from others regarding ECMA-262 support in their language of choice. It's not always 100%.

.Net, for example, has a 262-compliant mode, but it doesn't support some cases (that I can't specifically recall).

[ucaiug-admin]

@tviegut we do often recommend using regexps for date verification since pattern is more reliably implemented than format (as long as your regexps don't do anything too advanced). So that use case makes sense to me.

@handrews yes that's why pattern was chosen for the IEC standards due to the unreliability of format. So this would impact what we have out in our reference implementations in E.U. messages, etc. Anyway, sorry to "rock the boat" but maybe this helps us navigate things a bit. Thoughts?

Now, I did some further digging since I posted earlier. In reviewing the background further, turns out we had reps from both the U.S. and French teams thoroughly vett out and test the REGEX-s that were going to be published as part of the standard. Now, I see contributed to the draft the following: