Codec error handling callback should return newpos > start

[serhiy-storchaka]

According to PEP 293, the codec error handling callback should either raise an exception or return a pair (replacement, newpos). newpos specifies a new position within object, where the encoder/decoder will continue encoding/decoding. Negative values for newpos are treated as being relative to end of object. PEP 293 does not specify the bound of newpos, and this is a problem.

If newpos == start, the codec will not advance. If replacement is empty, it will just hang in an infinite loop, calling the callback again and again. If replacement is not empty, it will additionally add it in an internal buffer, and finally, after long time, you will get a MemoryError. If newpos < start, the behavior may be even more weird.

Before fixing other errors in #81000 it even caused a crash, but now it is only an infinite loop.

I propose to limit the returning value by range [start+1, len(object)].

@malemburg, @doerwalter, @vstinner

[doerwalter]

IMHO this is a reasonable tightening of the spec in PEP 293. An error handling callback that skips back just doesn't make sense. So the restriction that the value of newpos (after being adjusted when it is negative), must be in the range [start+1, len(object)] should be added to PEP 293, and the code should be updated to check that restriction and raise an exception if that restriction is violated. Should this exception be a ValueError?

[serhiy-storchaka]

Yes, ValueError looks the most appropriate exception.

BTW, OverflowError can already be raised for the value outside of the Py_ssize_t range, but I think it is not so good for smaller values outside of the range [start+1, len(object)].