crash appending list and namedtuple

[benrg]

BPO	8847
Nosy	@loewis, @birkenfeld, @rhettinger, @terryjreedy, @jcea, @atsuoishimoto, @amauryfa, @ncoghlan, @pitrou, @vstinner, @jackdied, @tjguk, @merwok, @alex, @bitdancer, @briancurtin, @skrah, @meadori
Files	issue8847.diff issue8847-3.2.diff issue8847-3.3.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-08-01.09:13:02.819>
created_at = <Date 2010-05-28.22:53:45.970>
labels = ['interpreter-core', 'deferred-blocker', 'library', 'OS-windows', 'type-crash']
title = 'crash appending list and namedtuple'
updated_at = <Date 2012-08-01.13:32:32.136>
user = 'https://bugs.python.org/benrg'

bugs.python.org fields:

activity = <Date 2012-08-01.13:32:32.136>
actor = 'loewis'
assignee = 'none'
closed = True
closed_date = <Date 2012-08-01.09:13:02.819>
closer = 'loewis'
components = ['Interpreter Core', 'Library (Lib)', 'Windows']
creation = <Date 2010-05-28.22:53:45.970>
creator = 'benrg'
dependencies = []
files = ['26623', '26633', '26634']
hgrepos = []
issue_num = 8847
keywords = ['patch']
message_count = 42.0
messages = ['106695', '106720', '107103', '107108', '107111', '107112', '107115', '107116', '107117', '107119', '107120', '107124', '107147', '130233', '166907', '166908', '166909', '166929', '166935', '166936', '166942', '166943', '166946', '166954', '166964', '166969', '166971', '167009', '167026', '167033', '167036', '167037', '167057', '167062', '167063', '167098', '167100', '167101', '167106', '167111', '167120', '167127']
nosy_count = 21.0
nosy_names = ['loewis', 'georg.brandl', 'rhettinger', 'terry.reedy', 'jcea', 'ishimoto', 'amaury.forgeotdarc', 'ncoghlan', 'pitrou', 'vstinner', 'jackdied', 'tim.golden', 'eric.araujo', 'mrabarnett', 'alex', 'r.david.murray', 'brian.curtin', 'skrah', 'meador.inge', 'benrg', 'python-dev']
pr_nums = []
priority = 'deferred blocker'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue8847'
versions = ['Python 3.2', 'Python 3.3']

[benrg]

c:\>python
  Python 3.1.2 (r312:79149, Mar 21 2010, 00:41:52) [MSC v.1500 32 bit (Intel)] on win32
  Type "help", "copyright", "credits" or "license" for more information.
  >>> from collections import namedtuple
  >>> foo = namedtuple('foo', '')
  >>> [1] + foo()

At this point the interpreter crashes. Also happens when foo has named arguments, and in batch scripts. foo() + [1] throws a TypeError as expected. [] + foo() returns (). The immediate cause of the crash is the CALL instruction at 1E031D5A in python31.dll jumping into uninitialized memory.

[bitdancer]

I can't reproduce this on either 3.1.2 or py3k trunk.

[terryjreedy]

Running the exact same binary on winxp with an amd athlon processor,
I *did* get a crash after about 5 seconds. "python.exe has encountered a problem and needs to close. We are sorry for the inconvenience."
Trying again with IDLE instead of the command window, I get the same message (for pythonw instead of python) and the shell restarts when I close the message.

Even though foo()+[1] correctly raises a TypeError, the reverse [1] + foo() very bizarrely produces a length 1 tuple whose bizarre member is supposedly an instance of a empty list [] with length 1

from collections import namedtuple
foo = namedtuple('foo', '')
a = [1] + foo()
b=a[0]
print (type(a), len(a), type(b), len(type(b)), type(type(b)))

# <class 'tuple'> 1 [] 1 <class 'list'>

([2]+foo() produces same output)

Other than the above, any attempt to do anything with b or type(b) that I tried crashes. I presume that this is due to attribute lookups on the invalid type(b) (or something like that), which type(b) must bypass. In particular, the OP's crash is due to trying to print the tuple which tries to print its member which looks for type(b).__str__ which crashes.

To anyone: investigating crashers like this is a lot easier with IDLE and an edit window than with the interactive command window.

[terryjreedy]

More experiments

from collections import namedtuple
foo = namedtuple('foo', '')
a = [] + foo()
print (a, type(a), len(a))
# () <class 'tuple'> 0

ie, a standard empty tuple, whereas
a = [1,1] + foo()
crashes immediately. So the behavior of list()+namedtuple depends on the length of the list.

There are also some funny interactions. Adding
try:
a = foo()+[]
except TypeError:
print("correct TypeError")

after the 'foo = ' line in my original 5 line example causes the final print to crash, whereas adding the same 4 lines to the 4 line example at the beginning of this message does not obviously change anything.

David, since you omitted all details, I wonder if you tested in batch mode, as I did, and hence never tried to print the malformed object, or used different OS or hardware.

[jackdied]

I can't reproduce on 3k trunk with Ubuntu 10.04, gcc 4.4.3

namedtuples are just a subclass of tuple with only two dunder methods defined (a plain __new__ with empty __slots__). Can you provoke the same behavior with plain tuples, or a subclass of tuple that looks like one of these?

class Crasher(tuple): pass

class Crasher(tuple):
  __slots__ = ()

class Crasher(tuple):
  def __new__(cls,): return tuple.__new__(cls,)
  __slots__ = ()

[terryjreedy]

Substituting
foo = tuple()
TypeError: can only concatenate list (not "tuple") to list

class Crasher(tuple): pass
foo = Crasher()
a = [1] + foo
b=a[0]
print (type(a), len(a), type(b), len(type(b)), type(type(b)))

<class 'tuple'> 1 [] 1 <class 'list'>

as before, so namedtuple is not, in particular, the culprit.
Other two Crasher versions do the same. I also get a delayed pythonw error message after the print that does not cause the shell to restart. This may partly be an IDLE artifact.

[jackdied]

Two more probes:

1. does it also have the same strange/crashy behavior when you subclass list and concat that to a tuple?

2. does dropping the optimization level down to -O help? This has "compiler quirk" written all over it. The C-code for list and tuple concat are almost identical, and both start with a type check.

[terryjreedy]

"can't reproduce" does not inform as to what *did* happen with which code.

More experiments:
foo = str()
TypeError: can only concatenate list (not "str") to list

class s(str): pass
foo = s()
TypeError: Can't convert 'list' object to str implicitly

Why is it trying to do that? Of course, the interpreter can (implicitly) convert list to tuple, which must be what happens in OP example.

The subclasses of tuple and str do not gain an __radd__ method. If we add one

class s(str):
    def __radd__(s,o): print('in radd, type(o)')
foo = s()
a = [1] + foo
# prints "in radd <class 'list'>"

no implicit conversion is done.

Reversing tuple and list

class Crasher(list): pass
a = () + Crasher() # or Crasher([1])
print(a, type(a), len(a))
#[] <class 'list'> 0 # or [1] <class 'list'> 1

whereas
a = (1,) + Crasher()
crashes, so not completely symmetrical

[terryjreedy]

1. answered before you asked (yes, similar)
2. (same thought) I am using PSF windows installer, however that was prepared. Martin?

[jackdied]

if the id() of the left operand is identical to the id() of the first element in the result it would strongly support compiler skulldugerry.

class Crasher(tuple): pass
foo = Crasher()
x = [1]
a = x + foo
b=a[0]

if id(b) == id(x):
  raise Exception("It's the C compiler what did it!")

The only way I can think of this coming about is the right_op getting new'd and then .extend'ing(left_op). That extend() must be going batsh*t and inserting the left_op instead of it's contained items. The C-code for extend is more fiddly than the code for concatenation so there is more room for the compiler to generate bad code.

[terryjreedy]

Good try, but for one run, the ids of foo, x, a, and b are
>>> 
15719440 15717880 15273104 12266976