Description
Pandas version checks
-
I have checked that this issue has not already been reported.
-
[ ] I have confirmed this bug exists on the latest version of pandas.use_nullable_dtypeshas been added by ENH: Add option to use nullable dtypes in read_csv #48776 . -
I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
from io import StringIO
from textwrap import dedent
csv = dedent("""
epoch_1,epoch_2
1665045912937687151,1665045912937689151
,
""")[1:]
pd.read_csv(StringIO(csv), dtype="Int64", use_nullable_dtypes=True).assign(diff=lambda df: df.epoch_2 - df.epoch_1)Issue Description
Fails with KeyError: Int64Dtype() in File [...]/pandas/_libs/parsers.pyx:1417, in pandas._libs.parsers._maybe_upcast(). (Also refer to full stacktrace below).
KeyError Traceback (most recent call last)
Cell In [39], line 1
----> 1 pd.read_csv(StringIO(csv), dtype="Int64", use_nullable_dtypes=True).assign(diff=lambda df: df.epoch_2 - df.epoch_1)
File ~/.local/conda/envs/test_pandas_nightly/lib/python3.10/site-packages/pandas/util/_decorators.py:211, in deprecate_kwarg.<locals>._deprecate_kwarg.<locals>.wrapper(*args, **kwargs)
209 else:
210 kwargs[new_arg_name] = new_arg_value
--> 211 return func(*args, **kwargs)
File ~/.local/conda/envs/test_pandas_nightly/lib/python3.10/site-packages/pandas/util/_decorators.py:331, in deprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper(*args, **kwargs)
325 if len(args) > num_allow_args:
326 warnings.warn(
327 msg.format(arguments=_format_argument_list(allow_args)),
328 FutureWarning,
329 stacklevel=find_stack_level(),
330 )
--> 331 return func(*args, **kwargs)
File ~/.local/conda/envs/test_pandas_nightly/lib/python3.10/site-packages/pandas/io/parsers/readers.py:963, in read_csv(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, skipfooter, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, doublequote, escapechar, comment, encoding, encoding_errors, dialect, error_bad_lines, warn_bad_lines, on_bad_lines, delim_whitespace, low_memory, memory_map, float_precision, storage_options, use_nullable_dtypes)
948 kwds_defaults = _refine_defaults_read(
949 dialect,
950 delimiter,
(...)
959 defaults={"delimiter": ","},
960 )
961 kwds.update(kwds_defaults)
--> 963 return _read(filepath_or_buffer, kwds)
File ~/.local/conda/envs/test_pandas_nightly/lib/python3.10/site-packages/pandas/io/parsers/readers.py:619, in _read(filepath_or_buffer, kwds)
616 return parser
618 with parser:
--> 619 return parser.read(nrows)
File ~/.local/conda/envs/test_pandas_nightly/lib/python3.10/site-packages/pandas/io/parsers/readers.py:1796, in TextFileReader.read(self, nrows)
1789 nrows = validate_integer("nrows", nrows)
1790 try:
1791 # error: "ParserBase" has no attribute "read"
1792 (
1793 index,
1794 columns,
1795 col_dict,
-> 1796 ) = self._engine.read( # type: ignore[attr-defined]
1797 nrows
1798 )
1799 except Exception:
1800 self.close()
File ~/.local/conda/envs/test_pandas_nightly/lib/python3.10/site-packages/pandas/io/parsers/c_parser_wrapper.py:231, in CParserWrapper.read(self, nrows)
229 try:
230 if self.low_memory:
--> 231 chunks = self._reader.read_low_memory(nrows)
232 # destructive to chunks
233 data = _concatenate_chunks(chunks)
File ~/.local/conda/envs/test_pandas_nightly/lib/python3.10/site-packages/pandas/_libs/parsers.pyx:818, in pandas._libs.parsers.TextReader.read_low_memory()
File ~/.local/conda/envs/test_pandas_nightly/lib/python3.10/site-packages/pandas/_libs/parsers.pyx:900, in pandas._libs.parsers.TextReader._read_rows()
File ~/.local/conda/envs/test_pandas_nightly/lib/python3.10/site-packages/pandas/_libs/parsers.pyx:1065, in pandas._libs.parsers.TextReader._convert_column_data()
File ~/.local/conda/envs/test_pandas_nightly/lib/python3.10/site-packages/pandas/_libs/parsers.pyx:1417, in pandas._libs.parsers._maybe_upcast()
KeyError: Int64Dtype()use_nullable_dtypes works well without dtype="Int64".
In [40]: pd.read_csv(StringIO(csv), use_nullable_dtypes=True).assign(diff=lambda df: df.epoch_2 - df.epoch_1)
Out[40]:
epoch_1 epoch_2 diff
0 1665045912937687151 1665045912937689151 2000
1 <NA> <NA> <NA>Without use_nullable_dtypes the coercion works, but there is a loss of precision (ie the diff is reported as 1792 instead of 2000 on my machine).
In [41]: pd.read_csv(StringIO(csv), dtype="Int64").assign(diff=lambda df: df.epoch_2 - df.epoch_1)
Out[41]:
epoch_1 epoch_2 diff
0 1665045912937687296 1665045912937689088 1792
1 <NA> <NA> <NA>Expected Behavior
Produces dataframe with:
epoch_1 epoch_2 diff
0 1665045912937687151 1665045912937689151 2000
1 <NA> <NA> <NA>
Installed Versions
INSTALLED VERSIONS
commit : 2f7dce4
python : 3.10.6.final.0
python-bits : 64
OS : Darwin
OS-release : 21.6.0
Version : Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_GB.UTF-8
LOCALE : en_GB.UTF-8
pandas : 1.6.0.dev0+350.g2f7dce4e6e
numpy : 1.23.3
pytz : 2022.4
dateutil : 2.8.2
setuptools : 65.5.0
pip : 22.3
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 8.5.0
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None