RecursionError with pd.DataFrame.from_records

[kdebrab]

Code Sample, a copy-pastable example if possible

import pandas as pd
index = pd.CategoricalIndex([pd.Interval(-20, -10), pd.Interval(-10, 0), pd.Interval(0, 10)])
series_of_dicts = pd.Series([{'a': 1}, {'a': 2}, {'b': 3}], index=index)
frame = pd.DataFrame.from_records(series_of_dicts, index=index)

raises:

RecursionError: maximum recursion depth exceeded in comparison

Problem description

Though using pd.DataFrame.from_records with pd.Series objects is not explicitly supported, it does work in most of the cases. Only in case the index of the Series is a CategoricalIndex of Intervals, it suddenly doesn't work anymore.

E.g. the following works fine:

series_of_dicts2 = pd.Series([{'a': 1}, {'a': 2}, {'b': 3}])
frame = pd.DataFrame.from_records(series_of_dicts2, index=index)

And it even works when series_of_dicts2.index is a CategoricalIndex or a list of Intervals, but not when it is CategoricalIndex of Intervals. It is especially strange because the index of series_of_dicts2 is not even used for the output.

Expected Output

The same as when the following workaround is used:

frame = pd.DataFrame.from_records(series_of_dicts.values, index=index)
Out[2]: 
              a    b
(-20, -10]  1.0  NaN
(-10, 0]    2.0  NaN
(0, 10]     NaN  3.0

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.6.8.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None
pandas : 1.0.2
numpy : 1.18.2
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 40.6.2
Cython : 0.29.4
pytest : 5.3.5
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 1.2.8
lxml.etree : 4.5.0
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.1
IPython : 7.13.0
pandas_datareader: None
bs4 : 4.6.1
bottleneck : 1.3.1
fastparquet : None
gcsfs : None
lxml.etree : 4.5.0
matplotlib : 3.2.0
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.3
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 5.3.5
pyxlsb : None
s3fs : 0.2.1
scipy : 1.4.1
sqlalchemy : 1.3.15
tables : None
tabulate : 0.8.6
xarray : 0.15.0
xlrd : 1.2.0
xlwt : 1.3.0
xlsxwriter : 1.2.8
numba : None

[ShaharNaveh]

I can't reproduce on master.

---

INSTALLED VERSIONS

commit : 19624de
python : 3.8.2.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.25.a-2-hardened
Version : SMP PREEMPT Thu, 12 Mar 2020 23:43:38 +0000
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.1.0.dev0+871.g19624de83
numpy : 1.18.2
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 46.0.0
Cython : 0.29.15
pytest : 5.4.1
hypothesis : 5.6.0
sphinx : 2.4.4
blosc : 1.8.3
feather : None
xlsxwriter : 1.2.8
lxml.etree : 4.5.0
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.1
IPython : 7.13.0
pandas_datareader: None
bs4 : 4.8.2
bottleneck : 1.3.2
fastparquet : 0.3.3
gcsfs : None
matplotlib : 3.2.0
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.1
pandas_gbq : None
pyarrow : 0.16.0
pytables : None
pyxlsb : None
s3fs : 0.4.0
scipy : 1.4.1
sqlalchemy : 1.3.15
tables : 3.6.1
tabulate : 0.8.6
xarray : 0.15.0
xlrd : 1.2.0
xlwt : 1.3.0
numba : 0.48.0

[kdebrab]

FYI, a (truncated) traceback:

Traceback (most recent call last):
  File "C:\Python36\lib\site-packages\IPython\core\interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-90750efbe895>", line 4, in <module>
    frame = pd.DataFrame.from_records(series_of_dicts, index=index)
  File "C:\Python36\lib\site-packages\pandas\core\frame.py", line 1636, in from_records
    arrays, arr_columns = to_arrays(data, columns, coerce_float=coerce_float)
  File "C:\Python36\lib\site-packages\pandas\core\internals\construction.py", line 460, in to_arrays
    if isinstance(data[0], (list, tuple)):
  File "C:\Python36\lib\site-packages\pandas\core\series.py", line 871, in __getitem__
    result = self.index.get_value(self, key)
  File "C:\Python36\lib\site-packages\pandas\core\indexes\category.py", line 530, in get_value
    return series.take([indexer])[0]
  File "C:\Python36\lib\site-packages\pandas\core\series.py", line 871, in __getitem__
    result = self.index.get_value(self, key)
  File "C:\Python36\lib\site-packages\pandas\core\indexes\category.py", line 530, in get_value
    return series.take([indexer])[0]
  File "C:\Python36\lib\site-packages\pandas\core\series.py", line 871, in __getitem__
    result = self.index.get_value(self, key)
[...]
File "C:\Python36\lib\site-packages\pandas\core\indexes\category.py", line 530, in get_value
    return series.take([indexer])[0]
  File "C:\Python36\lib\site-packages\pandas\core\series.py", line 871, in __getitem__
    result = self.index.get_value(self, key)
  File "C:\Python36\lib\site-packages\pandas\core\indexes\category.py", line 530, in get_value
    return series.take([indexer])[0]
  File "C:\Python36\lib\site-packages\pandas\core\series.py", line 871, in __getitem__
    result = self.index.get_value(self, key)
  File "C:\Python36\lib\site-packages\pandas\core\indexes\category.py", line 530, in get_value
    return series.take([indexer])[0]
  File "C:\Python36\lib\site-packages\pandas\core\series.py", line 871, in __getitem__
    result = self.index.get_value(self, key)
  File "C:\Python36\lib\site-packages\pandas\core\indexes\category.py", line 530, in get_value
    return series.take([indexer])[0]
  File "C:\Python36\lib\site-packages\pandas\core\series.py", line 830, in take
    new_values, index=new_index, fastpath=True
  File "C:\Python36\lib\site-packages\pandas\core\series.py", line 311, in __init__
    self._set_axis(0, index, fastpath=True)
  File "C:\Python36\lib\site-packages\pandas\core\series.py", line 384, in _set_axis
    is_all_dates = labels.is_all_dates
  File "pandas\_libs\properties.pyx", line 34, in pandas._libs.properties.CachedProperty.__get__
  File "C:\Python36\lib\site-packages\pandas\core\indexes\base.py", line 1739, in is_all_dates
    return is_datetime_array(ensure_object(self.values))
  File "pandas\_libs\algos_common_helper.pxi", line 33, in pandas._libs.algos.ensure_object
  File "C:\Python36\lib\site-packages\pandas\core\arrays\categorical.py", line 1284, in __array__
    ret = take_1d(self.categories.values, self._codes)
  File "C:\Python36\lib\site-packages\pandas\core\algorithms.py", line 1602, in take_nd
    return arr.take(indexer, fill_value=fill_value, allow_fill=allow_fill)
  File "C:\Python36\lib\site-packages\pandas\core\arrays\interval.py", line 890, in take
    return self._shallow_copy(left_take, right_take)
  File "C:\Python36\lib\site-packages\pandas\core\arrays\interval.py", line 765, in _shallow_copy
    return self._simple_new(left, right, closed=closed, verify_integrity=False)
  File "C:\Python36\lib\site-packages\pandas\core\arrays\interval.py", line 195, in _simple_new
    left = ensure_index(left, copy=copy)
  File "C:\Python36\lib\site-packages\pandas\core\indexes\base.py", line 5357, in ensure_index
    return Index(index_like)
  File "C:\Python36\lib\site-packages\pandas\core\indexes\base.py", line 390, in __new__
    return Int64Index(data, copy=copy, dtype=dtype, name=name)
  File "C:\Python36\lib\site-packages\pandas\core\indexes\numeric.py", line 80, in __new__
    name = maybe_extract_name(name, data, cls)
  File "C:\Python36\lib\site-packages\pandas\core\indexes\base.py", line 5397, in maybe_extract_name
    if name is None and isinstance(obj, (Index, ABCSeries)):
  File "C:\Python36\lib\site-packages\pandas\core\dtypes\generic.py", line 12, in _check
    return getattr(inst, attr, "_typ") in comp
RecursionError: maximum recursion depth exceeded in comparison

[mroeschke]

I am getting the expected result on master. Could use a test

In [39]: import pandas as pd
    ...: index = pd.CategoricalIndex([pd.Interval(-20, -10), pd.Interval(-10, 0), pd.Interval(0, 10)])
    ...: series_of_dicts = pd.Series([{'a': 1}, {'a': 2}, {'b': 3}], index=index)
    ...: frame = pd.DataFrame.from_records(series_of_dicts, index=index)

In [40]: frame
Out[40]:
              a    b
(-20, -10]  1.0  NaN
(-10, 0]    2.0  NaN
(0, 10]     NaN  3.0

In [41]: pd.__version__
Out[41]: '1.1.0.dev0+1974.g0159cba6e'