BUG: mask method fails on Timedelta series with numpy 1.20

[jeffzi]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import numpy as np
import pandas as pd

td = pd.Series([pd.Timedelta(days=i) for i in range(3)])

try:
    # same as td.mask([False, False, True])
    td.where([False, False, True], np.nan)
except TypeError as err:
    print(err)
#> The DTypes <class 'numpy.dtype[float16]'> and <class 'numpy.dtype[timedelta64]'> do not have a common DType. For example they cannot be stored in a single array unless the dtype is `object`.

# gives same result as above on numpy < 1.20
print(td.where([False, False, True], pd.NaT))
#> 0      NaT
#> 1      NaT
#> 2   2 days
#> dtype: timedelta64[ns]

# also ok
print(td.where([False, False, True], 10.0))
#> 0               10.0
#> 1               10.0
#> 2    2 days 00:00:00
#> dtype: object

td.where([False, False, True], np.nan, inplace=True) # ok
print(td)
#> 0      NaT
#> 1      NaT
#> 2   2 days
#> dtype: timedelta64[ns]

^{Created on 2021-02-02 by the reprexpy package}

Problem description

With numpy 1.20.0, mask on a Series of dtype timedelta64 raises a TypeError. The code above shows that the issue comes from the internals of where and happens only when a np.nan is passed as replacement value.

I noticed that inplace=True avoids the issue.

Expected Output

td.mask(...) should return the same as where(..., pd.NaT)

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 9d598a5
python : 3.8.6.final.0
python-bits : 64
OS : Linux
OS-release : 5.10.7-3-MANJARO
Version : #1 SMP PREEMPT Fri Jan 15 21:11:34 UTC 2021
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.2.1
numpy : 1.20.0
pytz : 2020.5
dateutil : 2.8.1
pip : 20.2.1
setuptools : 49.2.1
Cython : None
pytest : 6.2.2
hypothesis : 6.1.1
sphinx : 3.4.3
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.20.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.4
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.6.0
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[jbrockmendel]

This works on master, could use a test

[fpavogt]

With pandas v1.2.5, it still does not work on my side as long as numpy>v.1.20.x. The following MWE:

import pandas as pd
vals = pd.DataFrame(range(3), dtype='timedelta64[ns]')
vals.mask(vals.notna())

still fails with the following TypeError:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-16-0e487ce07732> in <module>
      1 import pandas as pd
      2 vals = pd.DataFrame(range(3), dtype='timedelta64[ns]')
----> 3 vals.mask(vals.notna())

~/opt/miniconda3/envs/dvas/lib/python3.8/site-packages/pandas/core/generic.py in mask(self, cond, other, inplace, axis, level, errors, try_cast)
   9315             cond = np.array(cond)
   9316 
-> 9317         return self.where(
   9318             ~cond,
   9319             other=other,

~/opt/miniconda3/envs/dvas/lib/python3.8/site-packages/pandas/core/generic.py in where(self, cond, other, inplace, axis, level, errors, try_cast)
   9284         """
   9285         other = com.apply_if_callable(other, self)
-> 9286         return self._where(
   9287             cond, other, inplace, axis, level, errors=errors, try_cast=try_cast
   9288         )

~/opt/miniconda3/envs/dvas/lib/python3.8/site-packages/pandas/core/generic.py in _where(self, cond, other, inplace, axis, level, errors, try_cast)
   9134 
   9135         else:
-> 9136             new_data = self._mgr.where(
   9137                 other=other,
   9138                 cond=cond,

~/opt/miniconda3/envs/dvas/lib/python3.8/site-packages/pandas/core/internals/managers.py in where(self, other, cond, align, errors, try_cast, axis)
    555             other = extract_array(other, extract_numpy=True)
    556 
--> 557         return self.apply(
    558             "where",
    559             align_keys=align_keys,

~/opt/miniconda3/envs/dvas/lib/python3.8/site-packages/pandas/core/internals/managers.py in apply(self, f, align_keys, ignore_failures, **kwargs)
    425                     applied = b.apply(f, **kwargs)
    426                 else:
--> 427                     applied = getattr(b, f)(**kwargs)
    428             except (TypeError, NotImplementedError):
    429                 if not ignore_failures:

~/opt/miniconda3/envs/dvas/lib/python3.8/site-packages/pandas/core/internals/blocks.py in where(self, other, cond, errors, try_cast, axis)
   1510             # By the time we get here, we should have all Series/Index
   1511             #  args extracted to ndarray
-> 1512             result = expressions.where(cond, values, other)
   1513 
   1514         if self._can_hold_na or self.ndim == 1:

~/opt/miniconda3/envs/dvas/lib/python3.8/site-packages/pandas/core/computation/expressions.py in where(cond, a, b, use_numexpr)
    250     """
    251     assert _where is not None
--> 252     return _where(cond, a, b) if use_numexpr else _where_standard(cond, a, b)
    253 
    254 

~/opt/miniconda3/envs/dvas/lib/python3.8/site-packages/pandas/core/computation/expressions.py in _where_standard(cond, a, b)
    160 def _where_standard(cond, a, b):
    161     # Caller is responsible for extracting ndarray if necessary
--> 162     return np.where(cond, a, b)
    163 
    164 

<__array_function__ internals> in where(*args, **kwargs)

TypeError: The DTypes <class 'numpy.dtype[float16]'> and <class 'numpy.dtype[timedelta64]'> do not have a common DType. For example they cannot be stored in a single array unless the dtype is `object`.

Edit:
For the record, I would add that in the MWE above, the following works without issues:

vals.mask(vals.notna(), pd.NaT)

But ideally, the mask method with default arguments should work fine no matter what dtype of the DataFrame. No ?

[jreback]

@fpavogt see @jbrockmendel comment; try on 1.3.0rc1

[fpavogt]

@jreback and @jbrockmendel: that's my bad ... I meant to have tried with "the content of master", but my brain short-circuited when I saw that v1.2.5 got released only a few hours ago and tried that one instead without even thinking about it. I confirm it all works fine on my end with v1.3.0rc1. Sorry for the confusion !