TST: Difference in drop_duplicates for numeric Series and DataFrames

[livia-b]

Code Sample

import pandas as pd
import numpy as np
df = pd.DataFrame({'a':[1,1,1,'one', 'one'],
                   'b': [2,2,np.nan, np.nan, np.nan],
                   'c': [3,3,np.nan, np.nan, 'three']
                  })
print df[['b']].drop_duplicates() 

print df['b'].drop_duplicates()

Expected Output

Why the result is not consistent?
For a dataframe (first case), the NA values are considered equal (we get 2 unique values), while for a series (second case), the NA values are not (similarly to numpy.unique, we get 3 unique values)

Curiously, in the case of a mixed type column, len(df[['c']].drop_duplicates() ) == len(df['c'].drop_duplicates())

output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.12.final.0
python-bits: 64
OS: Darwin
OS-release: 15.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8

pandas: 0.18.1
nose: None
pip: 8.0.3
setuptools: 20.1.1
Cython: 0.23.4
numpy: 1.11.0
scipy: 0.17.1
statsmodels: None
xarray: None
IPython: 4.1.2
sphinx: None
patsy: None
dateutil: 2.5.3
pytz: 2016.4
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.5.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: 0.9.2
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None

[jorisvandenbossche]

Not directly sure where it has been changed/fixed, but I see a consistent return value in master (and 0.19.0rc1):

In [7]: df[['b']].drop_duplicates() 
Out[7]: 
     b
0  2.0
2  NaN

In [8]: df['b'].drop_duplicates()
Out[8]: 
0    2.0
2    NaN
Name: b, dtype: float64

[jorisvandenbossche]

Would it actually be useful to be able to specify how to treat NaNs?

[livia-b]

For my specific application not really (I found this behaviour only because I forgot to use dropna to clean my data) , I am not sure how the other users need to handle it...

[jreback]

I think #13514 solved this, it appears enough tests. but if someone can review, could always add some specific ones for this consistency.