Series.replace fails to replace value

[tsoernes]

Code Sample, a copy-pastable example if possible

In [93]: ser.eq('nil').sum()
Out[93]: 1

In [94]: ser.replace('nil', pd.NA).eq('nil').sum()
/home/torstein/anaconda3/lib/python3.7/site-packages/pandas/core/missing.py:47: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  mask = arr == x
Out[94]: 1

In [95]: ser.loc[(ser == 'nil').fillna(False)] = pd.NA

In [104]: ser.eq('nil').sum()
Out[104]: 0

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]

INSTALLED VERSIONS

commit : None
python : 3.7.5.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.18-100.fc30.x86_64
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : nb_NO.UTF-8
LOCALE : nb_NO.UTF-8

pandas : 1.0.1
numpy : 1.17.3
pytz : 2019.3
dateutil : 2.8.0
pip : 19.3.1
setuptools : 41.6.0.post20191030
Cython : 0.29.13
pytest : 5.2.2
hypothesis : None
sphinx : 2.2.1
blosc : None
feather : None
xlsxwriter : 1.2.2
lxml.etree : 4.4.1
html5lib : 1.0.1
pymysql : None
psycopg2 : 2.8.4 (dt dec pq3 ext lo64)
jinja2 : 2.10.3
IPython : 7.9.0
pandas_datareader: None
bs4 : 4.8.1
bottleneck : 1.2.1
fastparquet : None
gcsfs : None
lxml.etree : 4.4.1
matplotlib : 2.2.3
numexpr : 2.7.0
odfpy : None
openpyxl : 3.0.0
pandas_gbq : None
pyarrow : 0.15.1
pytables : None
pytest : 5.2.2
pyxlsb : None
s3fs : None
scipy : 1.3.1
sqlalchemy : 1.3.10
tables : 3.5.2
tabulate : 0.8.5
xarray : None
xlrd : 1.2.0
xlwt : 1.3.0
xlsxwriter : 1.2.2
numba : 0.46.0

[MarcoGorelli]

Thanks @tsoernes

Could you include a minimal reproducible example? I couldn't reproduce this

In [1]: import pandas as pd                                                                                                                                                                                      
In [2]: ser = pd.DataFrame({'a': ['nil']})                                                                                                                                                                       

In [3]: ser                                                                                                                                                                                                      
Out[3]: 
     a
0  nil

In [4]: ser.replace('nil', pd.NA)                                                                                                                                                                                
Out[4]: 
      a
0  <NA>

In [5]: ser.replace('nil', pd.NA).eq('nil').sum()                                                                                                                                                                
Out[5]: 
a    0
dtype: int64

[tsoernes]

@MarcoGorelli

In the following example, I extract out two rows from the dataframe and recreate it via a dict.
The first row has a 'nil' in the 'amount' column; the second row has no 'nil'.
If I only recreate the dataframe with the first row, then the problem does not show itself.
When going via JSON, the problem does not persist.

In [282]: di = df.loc[[26123, 26122]].to_dict()

In [283]: di
Out[283]: 
<redacted since problem isolated later in thread>

In [284]: df = pd.DataFrame.from_dict(di)

In [285]: df.eq('nil').sum()
/home/torstein/anaconda3/lib/python3.7/site-packages/pandas/core/ops/array_ops.py:253: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  res_values = method(rvalues)
Out[285]: 
id                  0
properties          0
url                 0
started_at          0
ended_at            0
investee_id         0
lead_investor_id    0
created_at          0
updated_at          0
round               0
amount              1
currency            0
dtype: int64

In [286]: df.replace('nil', pd.NA).eq('nil').sum()
/home/torstein/anaconda3/lib/python3.7/site-packages/pandas/core/missing.py:47: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  mask = arr == x
/home/torstein/anaconda3/lib/python3.7/site-packages/pandas/core/ops/array_ops.py:253: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  res_values = method(rvalues)
Out[289]: 
id                  0
properties          0
url                 0
started_at          0
ended_at            0
investee_id         0
lead_investor_id    0
created_at          0
updated_at          0
round               0
amount              1
currency            0
dtype: int64

[MarcoGorelli]

@tsoernes OK, got it - it fails when there is another row with pd.NA:

>>> ser = pd.DataFrame({'a': ['nil', pd.NA]}) 
>>> ser.replace('nil', 'anything else')                                                                                                                                                                  
/home/SERILOCAL/m.gorelli/pandas/pandas/core/missing.py:48: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  mask = arr == x
      a
0   nil
1  <NA>

[MarcoGorelli]

Problem's related to this:

>>> np.array([['nil', pd.NA]]) == 'nil'
False
>>> np.array([['nil', 'anything else']]) == 'nil'
array([[ True, False]])

[AnnaDaglis]

take

[AnnaDaglis]

@MarcoGorelli
Is there any particular approach here you could suggest to fix the bug? There is no issue when using np.nan, not pd.NA.

>>> np.array([['nil', np.nan]]) == 'nil'
array([[ True, False]])

[MarcoGorelli]

Hi Anna - gonna be honest, I don't know :) I had a quick go at this one but couldn't figure it out. If you have any specific questions I'd imagine you could tag one of the core team members (e.g. Tom Augspurger) and ask.

…

On Tue, 3 Mar 2020 13:59 Anna Daglis, ***@***.***> wrote: @MarcoGorelli <https://github.com/MarcoGorelli> Is there any particular approach here you could suggest to fix the bug? There is no issue when using np.nan, not pd.NA. >>> np.array([['nil', np.nan]]) == 'nil' array([[ True, False]]) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#32075?email_source=notifications&email_token=AH7QVMAQXWDFDIOEHDX3T7TRFUENRA5CNFSM4KXDNN42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENTSZQI#issuecomment-593964225>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH7QVMHQJKUXY46CWPZW2NDRFUENRANCNFSM4KXDNN4Q> .

[AnnaDaglis]

The problem in this case lies in pandas.core.missing, mask_missing(arr, values_to_mask) function, the following clause:

if is_numeric_v_string_like(arr, x):
    # GH#29553 prevent numpy deprecation warnings
    mask = False
else:
    mask = arr == x

The problem arises here arr == x, which fails when arr contains pd.NA, as pointed out above.

@TomAugspurger @jorisvandenbossche
Do you have any suggestions how to tackle this bug? I noticed that handling pd.NA behaviour has been discussed in other issues, e.g. this one: #32265

[TomAugspurger]

I'm not sure offhand, sorry. But it seems like NA values should be considered False there, so perhaps fill it with something other than the value to mask? Some of the methods in core.arrays.boolean may be helpful to look through.

[tsoernes]

Can we get this for 1.0.2 that would be awesome