`\K` in lookbehind/lookahead should be always invalid

[mvorisek]

\K in lookbehind/lookahead should be always invalid as it affects "the global capture group" only, but lookbehind/lookahead does not participate in global capture group matching.

repro: https://3v4l.org/W9s8q

[NWilson]

What is the "global capture group"? I assume you mean, the span of text matched by the expression (ovector[0] to ovector[1], or $matches[0]). Your example does not use "global matching" (that is, searching for all matches in a string).

Lookbehind and lookahead do participate in all match attempts.

The behaviour of \K inside lookaround assertions (and also scan_substring :) ) is extremely weird and unexpected. Since PCRE2 10.38, the default PCRE2 behaviour has changed to disallow \K inside assertions. However, there is a backwards-compatibility option to restore the historic behaviour, which we recommend against using.

PHP is clearly passing this flag. Perhaps the PHP developers would consider updating their behaviour to the new one. However, this would break any PHP code which was relying on using \K inside lookaround. In practice, I expect this would affect almost no-one.

PCRE2 did not want to force any breaking changes on users, no matter how small, so the PHP developers can choose whether to make this change in their own time, or never.

[mvorisek]

What is the "global capture group"? I assume you mean, the span of text matched by the expression (ovector[0] to ovector[1], or $matches[0]). Your example does not use "global matching" (that is, searching for all matches in a string).

Yes, I mean "global capture group" == $matches[0].

The behaviour of \K inside lookaround assertions (and also scan_substring :) ) is extremely weird and unexpected.

That is the point of this issue.

Since PCRE2 10.38, the default PCRE2 behaviour has changed to disallow \K inside assertions. However, there is a backwards-compatibility option to restore the historic behaviour, which we recommend against using.

PHP is clearly passing this flag. Perhaps the PHP developers would consider updating their behaviour to the new one. However, this would break any PHP code which was relying on using \K inside lookaround. In practice, I expect this would affect almost no-one.

What is the flag and can I list/check the flags on runtime? Is there some docs where can I see all flags with examples?

In practice, I expect this would affect almost no-one.

I see, https://3v4l.org/g4Ldi works. I am closing this issue as all is probably expected due BC reasons.

[NWilson]

The flag is PCRE2_EXTRA_ALLOW_LOOKAROUND_BSK.

Its effect is documented here: https://pcre2project.github.io/pcre2/doc/pcre2api/

I thought you had maybe seen that documentation already, since only last week I updated it to be much more precise in its description of PCRE2_EXTRA_ALLOW_LOOKAROUND_BSK.

You should open an issue on the PHP bugtracker, to see if they would consider removing their use of this (semi-deprecated) flag.

[zherczeg]

My two cents: these flags have not much effect.

\K side effect without placing it into an assertion:

  re> /(?=.{10}(?1))x(\K){0}/
data> x1234567890
Start of matched string is beyond its end - displaying from end to start.
 0: 123456789

[NWilson]

Ouch! That's very bad Zoltan. Hmm. We should fix that... somehow. It would be possible (but nasty) to do it at compile-time, building a graph of parts of the pattern visited by assertions, or else, doing it at run-time somehow...

Yuck yuck yuck.

[zherczeg]

The easiest way to fix this is "improving" \K. The current and start offset can be compared when \K is used, and if \K is earlier:
a) \K fails
b) \K has no effect
c) \K sets the start to the match start (this may have effect when multiple \K is used)

I think b) is not user friendly. This can be hidden behind a flag.

[NWilson]

I hadn't considered that. One issue is that we aim to be "Perl-compatible", and Perl already disallows \K inside lookaround. However, it has the same bug that we do - if the lookaround does a subroutine call it's possible to work around the check.

Perhaps we should discuss with the Perl team how they would consider fixing this.