IntervalArray[datetime64[ns]].shift() raises TypeError

[TomAugspurger]

Code Sample, a copy-pastable example if possible

With pandas 0.25.3 or master,

In [2]: arr = pd.arrays.IntervalArray.from_breaks(pd.date_range('2000', periods=4))

In [3]: arr.shift()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-e840978d5de6> in <module>
----> 1 arr.shift()

~/miniconda3/envs/pandas=0.25.1/lib/python3.7/site-packages/pandas/core/arrays/base.py in shift(self, periods, fill_value)
    567
    568         empty = self._from_sequence(
--> 569             [fill_value] * min(abs(periods), len(self)), dtype=self.dtype
    570         )
    571         if periods > 0:

~/miniconda3/envs/pandas=0.25.1/lib/python3.7/site-packages/pandas/core/arrays/interval.py in _from_sequence(cls, scalars, dtype, copy)
    236     @classmethod
    237     def _from_sequence(cls, scalars, dtype=None, copy=False):
--> 238         return cls(scalars, dtype=dtype, copy=copy)
    239
    240     @classmethod

~/miniconda3/envs/pandas=0.25.1/lib/python3.7/site-packages/pandas/core/arrays/interval.py in __new__(cls, data, closed, dtype, copy, verify_integrity)
    176             copy=copy,
    177             dtype=dtype,
--> 178             verify_integrity=verify_integrity,
    179         )
    180

~/miniconda3/envs/pandas=0.25.1/lib/python3.7/site-packages/pandas/core/arrays/interval.py in _simple_new(cls, left, right, closed, copy, dtype, verify_integrity)
    196                 raise TypeError(msg.format(dtype=dtype))
    197             elif dtype.subtype is not None:
--> 198                 left = left.astype(dtype.subtype)
    199                 right = right.astype(dtype.subtype)
    200

~/miniconda3/envs/pandas=0.25.1/lib/python3.7/site-packages/pandas/core/indexes/numeric.py in astype(self, dtype, copy)
    367                 "values are required for conversion"
    368             ).format(dtype=dtype)
--> 369             raise TypeError(msg)
    370         elif (
    371             is_integer_dtype(dtype) and not is_extension_array_dtype(dtype)

TypeError: Cannot convert Float64Index to dtype datetime64[ns]; integer values are required for conversion

Problem description

We should be able to preserve the interval dtype, by placing NaT.

Expected Output

IntervalArray([NaT, (2000-01-01, 2000-01-02], (2000-01-02, 2000-01-03]],
              closed='right',
              dtype='interval[datetime64[ns]]')

[TomAugspurger]

A few changes that need to happen:

1. IntervalDtype.na_value should be na_value_for_dtype(self.subtype). So for IntervalArray[datetime64[ns]], that would be NaT.
2. intervals_to_interval_bounds, and likely other places, would need to handle NaT in additional to nan. Without this, IntervalArray._from_sequnce([pd.NaT]) fails.
3. ...

[Koustav-Samaddar]

Hi @TomAugspurger , this seems like an interesting problem to solve.

I was able to repro the bug and then went on to create a simpler example using interval[int64] isntead of using DateTime to get a better understanding of the problem. However, this also failed. Just wanted to check if the code snippet I tried is legal.

>>> arr2 = pd.arrays.IntervalArray.from_breaks([1, 2, 3])

>>> arr2
<IntervalArray>
[(1, 2], (2, 3]]
Length: 2, closed: right, dtype: interval[int64]

>>> arr2.dtype
interval[int64]

>>> arr2.shift()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/media/sf_ubuntu-budige/pandas-dev/pandas/core/arrays/base.py", line 628, in shift
    [fill_value] * min(abs(periods), len(self)), dtype=self.dtype
  File "/media/sf_ubuntu-budige/pandas-dev/pandas/core/arrays/interval.py", line 245, in _from_sequence
    return cls(scalars, dtype=dtype, copy=copy)
  File "/media/sf_ubuntu-budige/pandas-dev/pandas/core/arrays/interval.py", line 184, in __new__
    verify_integrity=verify_integrity,
  File "/media/sf_ubuntu-budige/pandas-dev/pandas/core/arrays/interval.py", line 204, in _simple_new
    left = left.astype(dtype.subtype)
  File "/media/sf_ubuntu-budige/pandas-dev/pandas/core/indexes/numeric.py", line 382, in astype
    arr = astype_nansafe(self.values, dtype=dtype)
  File "/media/sf_ubuntu-budige/pandas-dev/pandas/core/dtypes/cast.py", line 868, in astype_nansafe
    raise ValueError("Cannot convert non-finite values (NA or inf) to integer")
ValueError: Cannot convert non-finite values (NA or inf) to integer

I would expect the output to be:

>>> arr2.shift()
<IntervalArray>
[NaN, (1, 2]]
Length: 2, closed: right, dtype: interval[int64]

[yaolan4]

@TomAugspurger Is this good for first-time contributors?

[TomAugspurger]

I'm not sure. We have https://github.com/pandas-dev/pandas/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22 if you're looking for good first issues.

[mroeschke]

Almost solved but not quite right on master

In [34]: In [2]: arr = pd.arrays.IntervalArray.from_breaks(pd.date_range('2000', periods=4))
    ...:
    ...: In [3]: arr.shift()
Out[34]:
<IntervalArray>
[nan, (2000-01-01, 2000-01-02], (2000-01-02, 2000-01-03]]
Length: 3, closed: right, dtype: interval[datetime64[ns]]

In [35]: pd.__version__
Out[35]: '1.1.0.dev0+1078.gb5092d877'

[mroeschke]

Looks like a duplicate of #22428, but will keep it around to demonstrate the behavior with a different dtype

[jbrockmendel]

This works on master and has a test in tests.arrays.interval.test_interval.TestMethods.test_shift_datetime (though no comment as to when that test was added). Closing.