TypeError in to_datetime when passing Int64 column

[sanderland]

Code Sample

df = pd.DataFrame({'a':[1,2,3]})
df.a = df.a.astype("Int64")
pd.to_datetime(df.a, unit='ms')

Problem description

When a nullable int type is used, to_datetime gives the error:

TypeError: Argument 'values' has incorrect type (expected numpy.ndarray, got IntegerArray)

The error is unexpected and hard to track down (reliant on pandas internal storage formats).

Expected Output

Converted date times or NaT when input is NaN

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit : None python : 3.7.3.final.0 python-bits : 64 OS : Linux OS-release : 5.0.0-36-generic machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 0.25.3
numpy : 1.17.0
pytz : 2018.9
dateutil : 2.8.0
pip : 19.0.3
setuptools : 41.0.1
Cython : 0.29.6
pytest : 4.3.1
hypothesis : None
sphinx : 1.8.5
blosc : None
feather : None
xlsxwriter : 1.1.5
lxml.etree : 4.3.2
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.10
IPython : 7.4.0
pandas_datareader: None
bs4 : 4.7.1
bottleneck : 1.2.1
fastparquet : None
gcsfs : None
lxml.etree : 4.3.2
matplotlib : 3.0.3
numexpr : 2.6.9
odfpy : None
openpyxl : 2.6.1
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.2.1
sqlalchemy : 1.3.1
tables : 3.5.1
xarray : None
xlrd : 1.2.0
xlwt : 1.3.0
xlsxwriter : 1.1.5

[AskaryanKarine]

df = pd.DataFrame({'a':[1,2,3]})
df['a'] = df.astype({'a': 'int64'}) 
pd['a'].to_datetime(df['a'], unit='ms')

[sanderland]

df = pd.DataFrame({'a':[1,2,3]})
df['a'] = df.astype({'a': 'int64'}) 
pd['a'].to_datetime(df['a'], unit='ms')

We're aware of the workaround when there are no NaNs in the dataframe. Actually casting to float64 handles NaN->NaT as well. It just seems like an oversight that having a pandas supported Int64 datatype causes an error, particularly one exposing details of implementation.

[jreback]

@sanderland a PR to patch would be welcome

[AlexKirko]

@jreback We could just check if we are passing an ndarray to tslib.array_with_unit_to_datetime and cast arg to ndarray if it's not an ndarray already. Would that be acceptable? I don't think we can get out of introducing a type check and a possible cast unless we wish to change that tslib.array_with_unit_to_datetime, and I don't think that would be a good idea.

arg._values should return an <IntegerArray> in this case and the variable should just fall through until it's time to call tslib, and this is how we end up with a variable of a wrong type being passed.

Going into tslib would be a lot more work, especially because everything there expects ndarray to be passed, so we can't just patch one spot without making a mess.

If this kind of solution is acceptable, I can make a PR.

[sanderland]

@jreback I looked into it and got to similar questions as @AlexKirko : not sure what is proper here, casting to float64 ndarray definitely loses precision if ns are to be supported, casting to ndarray int64 breaks on null values, and going into tslib is hard and potentially inconsistent.

[AlexKirko]

@sanderland @jreback Sorry, wrote something stupid in this post previously: mistook the datetime overflow for the conversion overflow. Edited it.
We only start losing precision above 2^52, everything else is fine. We can expect numbers this large only when dealing with nanoseconds and then 2^52 is 1251 hours. It is unlikely that the combination of this kind of precision and time intervals might come up. Moreover, relative error should be manageable.
I think that casting to float64 is an acceptable solution, as the potential use case where it will create a noticeable loss of precision is rather contrived.
However, the loss will still exist, and the user wouldn't expect one when turning valid Int64 to datetime.

[AlexKirko]

@sanderland @jreback Came up with a way to fix the issue with slightly more overhead but without loss of precision. We'll have to remove the NaNs, cast to int32/int64, and then add NaT back in. I'll make the PR today.
This will happen only when the Series is an IntegerArray, so the overhead for other dtypes will be only one additional type check. For IntegerArrays some additional work will be done and more memory will be used (to store the empty index part of the IntegerArray), but it should be acceptable. I'll time the fixed function just in case.