BUG: TypeError exception when using read_html with extract_links='all' argument

[tysonite]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import requests
from bs4 import BeautifulSoup
import pandas as pd


with requests.Session() as session:
    response = session.get('https://dzerginsky--nnov.sudrf.ru/modules.php?name=sud_delo', headers={'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:20.0) Gecko/20100101 Firefox/20.0'})
    parsed_html = BeautifulSoup(response.text, features='html.parser')

    div_case_details = parsed_html.body.find('div', attrs={'id': 'tablcont'})
    table = div_case_details.find('table', attrs={'id': 'tablcont'})

    table_MN = pd.read_html(str(table), extract_links='all')

Issue Description

When extract_links='all' set to pd.read_html, TypeError: 'int' object is not subscriptable is raised. Without extract_links argument, table is parsed correctly, but no links are present.

Stacktrace:

    table_MN = pd.read_html(str(table), extract_links='all')
  File "/home/tysonite/court_dzr/.venv/lib/python3.8/site-packages/pandas/util/_decorators.py", line 317, in wrapper
    return func(*args, **kwargs)
  File "/home/tysonite/court_dzr/.venv/lib/python3.8/site-packages/pandas/io/html.py", line 1202, in read_html
    return _parse(
  File "/home/tysonite/court_dzr/.venv/lib/python3.8/site-packages/pandas/io/html.py", line 1015, in _parse
    df.columns = Index(
  File "/home/tysonite/court_dzr/.venv/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 566, in __new__
    subarr = com.asarray_tuplesafe(data, dtype=_dtype_obj)
  File "/home/tysonite/court_dzr/.venv/lib/python3.8/site-packages/pandas/core/common.py", line 238, in asarray_tuplesafe
    values = list(values)
  File "/home/tysonite/court_dzr/.venv/lib/python3.8/site-packages/pandas/io/html.py", line 1016, in <genexpr>
    ((col[0], None if isna(col[1]) else col[1]) for col in df.columns),
TypeError: 'int' object is not subscriptable

Applies to 1.5.0rc0 and main pandas versions on Python 3.8.

Expected Behavior

No exception raised. Links are parsed and added to dataframe.

Installed Versions

INSTALLED VERSIONS

commit : 1843cf2
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.102.1-microsoft-standard-WSL2
Version : #1 SMP Wed Mar 2 00:30:59 UTC 2022
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.6.0.dev0+29.g1843cf23c7
numpy : 1.23.2
pytz : 2022.2.1
dateutil : 2.8.2
setuptools : 44.0.0
pip : 20.0.2
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None

[mroeschke]

This patch fixes your example locally

--- a/pandas/io/html.py
+++ b/pandas/io/html.py
@@ -34,6 +34,7 @@ from pandas.core.dtypes.common import is_list_like
 from pandas import isna
 from pandas.core.construction import create_series_with_explicit_dtype
 from pandas.core.indexes.base import Index
+from pandas.core.indexes.multi import MultiIndex

 from pandas.io.common import (
     file_exists,
@@ -1011,7 +1012,7 @@ def _parse(flavor, io, match, attrs, encoding, displayed_only, extract_links, **
             # Cast MultiIndex header to an Index of tuples when extracting header
             # links and replace nan with None.
             # This maintains consistency of selection (e.g. df.columns.str[1])
-            if extract_links in ("all", "header"):
+            if extract_links in ("all", "header") and isinstance(df.columns, MultiIndex):
                 df.columns = Index(
                     ((col[0], None if isna(col[1]) else col[1]) for col in df.columns),
                     tupleize_cols=False,

We would need a minimal raw html example (without connecting to the internet) to add a unit test for this case

[tysonite]

Seems to me that minimal example like below:

		<table>
		  <tr>
		    <td>
		      <a href='https://google.com'>Google.com</a>
		    </td>
		  </tr>
		</table>

    import pandas as pd

    table = '''
		<table>
		  <tr>
		    <td>
		      <a href='https://google.com'>Google.com</a>
		    </td>
		  </tr>
		</table>
'''
    table_MN = pd.read_html(str(table), extract_links='all')

Gives:

Traceback (most recent call last):
  File "/home/tysonite/court_dzr/script.py", line 92, in <module>
    table_MN = pd.read_html(str(table), extract_links='all')
  File "/home/tysonite/court_dzr/.venv/lib/python3.8/site-packages/pandas/util/_decorators.py", line 317, in wrapper
    return func(*args, **kwargs)
  File "/home/tysonite/court_dzr/.venv/lib/python3.8/site-packages/pandas/io/html.py", line 1202, in read_html
    return _parse(
  File "/home/tysonite/court_dzr/.venv/lib/python3.8/site-packages/pandas/io/html.py", line 1015, in _parse
    df.columns = Index(
  File "/home/tysonite/court_dzr/.venv/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 567, in __new__
    subarr = com.asarray_tuplesafe(data, dtype=_dtype_obj)
  File "/home/tysonite/court_dzr/.venv/lib/python3.8/site-packages/pandas/core/common.py", line 238, in asarray_tuplesafe
    values = list(values)
  File "/home/tysonite/court_dzr/.venv/lib/python3.8/site-packages/pandas/io/html.py", line 1016, in <genexpr>
    ((col[0], None if isna(col[1]) else col[1]) for col in df.columns),
TypeError: 'int' object is not subscriptable

[mroeschke]

Thanks for the minimal example. Opened up a pull request to fix in #48334 so should be fixed when 1.5 is officially released.