BUG: read_xml iterparse doesn't handle multiple toplevel elements with lxml parser

[bailsman]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

from tempfile import NamedTemporaryFile
import pandas as pd

XML = '''
<!-- it will choke on this comment -->
<issue>
  <type>BUG</type>
</issue>
'''.encode('utf-8')

with NamedTemporaryFile() as tempfile:
    tempfile.write(XML)
    tempfile.flush()
    df = pd.read_xml(tempfile.name, iterparse={'issue': ['type']})

Issue Description

Traceback (most recent call last):
  File "k.py", line 14, in <module>
    df = pd.read_xml(tempfile.name, iterparse={'issue': ['type']})
  File "/home/erietveld/.local/lib/python3.8/site-packages/pandas/util/_decorators.py", line 317, in wrapper
    return func(*args, **kwargs)
  File "/home/erietveld/.local/lib/python3.8/site-packages/pandas/io/xml.py", line 1208, in read_xml
    return _parse(
  File "/home/erietveld/.local/lib/python3.8/site-packages/pandas/io/xml.py", line 946, in _parse
    data_dicts = p.parse_data()
  File "/home/erietveld/.local/lib/python3.8/site-packages/pandas/io/xml.py", line 538, in parse_data
    self._parse_nodes() if self.iterparse is None else self._iterparse_nodes()
  File "/home/erietveld/.local/lib/python3.8/site-packages/pandas/io/xml.py", line 677, in _iterparse_nodes
    del elem.getparent()[0]
TypeError: 'NoneType' object does not support item deletion

Note this feature is not yet released and planned for 1.5.0: #45724 (comment)

Expected Behavior

No exception is thrown and the document is parsed successfully. Perhaps don't try to delete if the element has no parent?

diff --git a/pandas/io/xml.py b/pandas/io/xml.py
index 181b0fe..c026a00 100644
--- a/pandas/io/xml.py
+++ b/pandas/io/xml.py
@@ -674,6 +674,8 @@ class _LxmlFrameParser(_XMLFrameParser):

                 elem.clear()
                 while elem.getprevious() is not None:
+                    if elem.getparent() is None:
+                        break
                     del elem.getparent()[0]

         if dicts == []:

Installed Versions

5465f54

[ParfaitG]

I cannot reproduce this issue with your exact code. What version of lxml do you have installed? Check with following:

import lxml
print(lxml.__version__)

The pandas' minimum recommended version for optional dependency for lxml is 4.5.0.

[bailsman]

4.9.0

Edit: this Dockerfile reproduces the issue for me.

FROM python:3.10.5

# pandas-dev main from 2022-06-20T00:00:00Z
RUN echo "lxml==4.9.0\nnumpy==1.22.4\npandas @ https://github.com/pandas-dev/pandas/archive/47494a48edf25d5a49b0fb5b896b454c15c83595.tar.gz\npython-dateutil==2.8.2\npytz==2022.1\nsix==1.16.0" > /requirements.txt && pip install -r /requirements.txt

RUN echo "from tempfile import NamedTemporaryFile\nimport pandas as pd\n\nXML = '''\n<!-- it will choke on this comment -->\n<issue>\n  <type>BUG</type>\n</issue>\n'''.encode('utf-8')\n\nwith NamedTemporaryFile() as tempfile:\n    tempfile.write(XML)\n    tempfile.flush()\n    df = pd.read_xml(tempfile.name, iterparse={'issue': ['type']})" > /47422.py

CMD python3 /47422.py

[ParfaitG]

Thanks. Oddly now, I do reproduce issue with 4.5.2. I may have been using a development code version. Thinking more, this may not be a pandas issue but an an lxml issue especially since we follow the same setup as lxml docs. I will raise an issue on their mailing list with an iterparse reprex.

It may be the unusual case of a comment before root element. They may even provide guidance since the iterparse / modifying the tree docs includes a note on the while loop usage.