Space in re's {min,max} should raise an error, rather than fail silently

[reuven]

BPO	42469
Nosy	@rhettinger, @terryjreedy, @ezio-melotti, @serhiy-storchaka, @reuven, @akulakov

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-11-26.07:09:54.395>
labels = ['expert-regex', 'type-bug', '3.10']
title = "Space in re's {min,max} should raise an error, rather than fail silently"
updated_at = <Date 2021-07-17.14:52:02.804>
user = 'https://github.com/reuven'

bugs.python.org fields:

activity = <Date 2021-07-17.14:52:02.804>
actor = 'andrei.avk'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Regular Expressions']
creation = <Date 2020-11-26.07:09:54.395>
creator = 'reuven'
dependencies = []
files = []
hgrepos = []
issue_num = 42469
keywords = []
message_count = 5.0
messages = ['381879', '381919', '381994', '382045', '397725']
nosy_count = 7.0
nosy_names = ['rhettinger', 'terry.reedy', 'ezio.melotti', 'mrabarnett', 'serhiy.storchaka', 'reuven', 'andrei.avk']
pr_nums = []
priority = 'low'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue42469'
versions = ['Python 3.10']

[reuven]

I just discovered that having whitespace inside of {min,max} causes the regexp to report no matches, rather than an error:

>>> import re
>>> s = 'abbcccddddeeeee'
>>> re.findall('d{1, 4}', s)
[]
>>> re.findall('d{1,4}', s)
['dddd']

Ruby and JavaScript have the same behavior, so maybe this is standard in some way. But I find it hard to believe that it's desirable. (My post on Twitter about this confirmed that a whole lot of people were bitten by this bug in the past.)

BSD grep, GNU grep, and GNU Emacs all raise an error upon encountering this whitespace, which strikes me as less surprising and more useful behavior.

[rhettinger]

If this can be fixed without doing brain surgery on the code, it would be nice to have this flagged as an error. People should learn as early as possible that regex patterns tend to be unforgiving about extra spaces.

[terryjreedy]

Pending further information, I believe that expecting '{1, 4}' to be interpreted as an re quantifier involves two mental errors: a) thinking that the domain specific re language allows optional whitespace and b) thinking that '{' is a special character only used for quantifier patterns. I propose changes directed at both errors.

In Python code, a space ' ' is always special (indent, required separator, optional separator) except in comments and strings. In the latter, a space is ordinary except as seen otherwise by reader, including humans. Functions that read a string as Python code make them special again. AFAIK, re functions always see spaces as ordinary unless the re.VERBOSE compile flag is passed, and even then they are only sometimes ignored.

Suggestion 1. Because this is contrary to human habit, put a special disclaimer at the end of the 2-sentence paragraph beginning "Some characters, like '|' or '(', are special." Add something like "Space ' ' is always ordinary, like 'c' is. Do not put one where a 'c' could not go."

"The special characters are:" is misleading because the special bracketed quantifier patterns that follow are composed of ordinary characters. (See below.)

Suggestion 2. Add 'and patterns' after 'characters'. Or put the quantifier patterns after a separate header.

'[' is special in that it always begins a set pattern. ']' is always special when preceded by '['. It is an error if the set is not closed with ']'. In particular, compile('[') raises.

'{' and }' are different.  They are not special by themselves anymore than digits and ',' are, whereas within quantifier patterns, all characters, not just '{' and '}', have special interpretations. In particular, compile('{') matches instead of raising like '[' does.
>>> re.findall('{', 'a{b')
['{']

Only a complete quantifier pattern is special. As near as I can tell, re.compile act at least as if it tries to match '{\d*(,\d*)?}' (with re.ASCII flag to only allow ascii digits?) when it encounters '{'. If this fails, it continues with '{' treated as ordinary. So '{1, 4}', '{1,c4}', '{ 1,4}', '{1,4x}', and '{0x33}' are all compiled as sequences of 6 ordinary characters that match themselves.

Suggestion 3. Somewhere say that '{' and '}' are ordinary unless there is a complete quantifier match, with nothing but digits[,digits] between '{}', with nothing else, including ' ', added.
---

Turning '{' into a special character by making it an error when it does not begin a quantifier would break all intentional uses of '{' as ordinary. I don't see making '{' sort-of special by sometimes raising and sometime not by a new special rule to be much better. In particular, I don't like having only extra ' 's or other whitespace raise. AFAIK, re is unforgiving of any extra chars, not just spaces.

The re.VERBOSE causes whitespace to be sometimes be ignored. The list of situations when not ignored does not, to me, obviously include quantifiers. But it should.

>>> re.findall("""d{1,4}""", 'dddd', re.X)
['dddd']
>>> re.findall("""d{1, 4}""", 'dddd', re.X)
[]

Suggestion 4. Say so, by adding 'or quantifier' after 'character class'

[serhiy-storchaka]

"{" has special meaning only when it is used in the special forms "\{[0-9]+\}", "\{,[0-9]+\}" or "\{[0-9]+,[0-9]+\}". In all other cases it means literal "{". I once tried to emit warnings when non-escaped "{" is used in literal meaning, but it did break a lot of code (for example in string.Template). I decided that it is too destructive change.

Now, we can emit a warning (and later error) if there are extra spaces. It is a breaking change, but I hope it will not break much user code. But which cases should be recognized as error and which silently accepted with literal meaning of "{"? Should we only handle spaces after comma, or before and after digits? Should we check spaces only or tabs and newlines and other whitespaces?

What about the verbose mode? Currently the third-party regex module ignores whitespace and comments in the verbose mode in more places than the re module. The re module consider some sequences atomic and does not allow whitespaces inside them even in the verbose mode, for simplicity and performance.

[akulakov]

I think the biggest issue here is not how to explain it clearly in the docs, but that a lot of users will not confirm this in the docs based on the combination of:

1. c{1,5} is similar to slicing, also similar to python list definition, and python sets, all of which allow spaces for readability.

2. there is no error or warning

3. c{1,5} having one meaning and c{1, 5} having completely different meaning with no warning is deeply weird unless you think thoroughly through all of the implications.

I would prefer a warning to be added. This leads to the question, where do you draw the line after which a warning is no longer added?

I think I would allow this form as the most extreme where warning is still shown:

c{ 1, 5 }

For any extra commas, newlines, etc, I would not show warnings because they look/feel different enough vs. {1,5}.

It seems to me that VERBOSE should have no effect inside of {1,5}.