BUG pd.get_dummies does not propagate `NaN` correctly

[beckermr]

Code Sample, a copy-pastable example if possible

In [17]: s
Out[17]: 
0      a
1      b
2      c
3    NaN
dtype: object
In [18]: pd.get_dummies(s)
Out[18]: 
   a  b  c
0  1  0  0
1  0  1  0
2  0  0  1
3  0  0  0

Problem description

The current implementation does not propagate NaN correctly. NaN means the data is missing. So if you turn missing data to all zeros when one-hot encoding, you are asserting that the proper label is none of the labels you have. In reality, the proper label could be one of the labels you have, you just do not know the proper label.

I realize people use the dummy_na option here, but if that option is not passed, then the output should have the NaNs put in the right spot.

Expected Output

It should propagate the NaNs.

In [18]: pd.get_dummies(s)
Out[18]: 
   a  b  c
0  1  0  0
1  0  1  0
2  0  0  1
3  NaN  NaN  NaN

Output of pd.show_versions()

In [19]: pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.1.final.0
python-bits: 64
OS: Darwin
OS-release: 15.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.19.2
nose: 1.3.7
pip: 9.0.1
setuptools: 33.1.1.post20170320
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
statsmodels: 0.8.0
xarray: None
IPython: 5.3.0
sphinx: None
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: None
tables: 3.3.0
numexpr: 2.6.2
matplotlib: 2.0.0
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: 1.1.5
pymysql: None
psycopg2: 2.6.2 (dt dec pq3 ext lo64)
jinja2: 2.9.5
boto: 2.46.1
pandas_datareader: None

[jreback]

dummy_na : bool, default False
    Add a column to indicate NaNs, if False NaNs are ignored.

In [1]: s = Series(['a', 'b', 'c', np.nan])

In [2]: pd.get_dummies(s)
Out[2]: 
   a  b  c
0  1  0  0
1  0  1  0
2  0  0  1
3  0  0  0

In [3]: pd.get_dummies(s, dummy_na=True)
Out[3]: 
   a  b  c  NaN
0  1  0  0    0
1  0  1  0    0
2  0  0  1    0
3  0  0  0    1

I think this is pretty clear.

[beckermr]

Nope. I am well aware of the dummy_na option, as I stated above in my description. This bug is when this option is not used.

[TomAugspurger]

This bug is when this option is not used.

I wouldn't call it a bug though, it's behaving as intended and documented (whether or not that intention is "correct").

Do you have an example use-case where you want the NaNs to propagate? I've only ever used get_dummies as a pre-processing step for fitting a model, so I've already handled NaNs, or want the dummy_na=True behavior.

[beckermr]

I am happy with calling it an API change rather than a bug! :)

I would say two things:

1. The way the API is defined for get_dummies is not logically consistent with how NaN is used in basically all other parts of the pandas API. Internal and logical consistency of the pandas API is certainly enough of a reason in and of itself to motivate the change. There does not have to be a use case to desire API consistency. In fact, it is usually the opposite. There generally there has to be a compelling use case to demand API inconsistency.

2. The example I am thinking of is working on modeling problems where you want to explicitly handle missing labels in some way that does not involve using an extra dummy column or setting them to all zeros. This could include multilabel classification with some missing labels where you would not discard examples with incomplete label sets.

Right now, to do this, you have to force pandas to create the dummy column and then do the replacement afterwards. This is, of course, not a big deal. However, as stated in 1) above, the purity of the API is a much better motivation for this change.

[jreback]

how does scikit-learn handle missing values in LabelEncoder? (e.g. what is the policy)

[beckermr]

@jreback IMO, this is a separate issue that is not terribly pertinent to pandas API purity.

In any case, the standard label encoder throws a TypeError. Scikit-learn does have facilities for handling missing labels. See the label propagation routines here. They use -1 to mark missing labels under the assumption that all labels have to be non-negative integers.

In principle, the LabelEncoder could do the same thing, but given that support for multi-label classification is somewhat case-by-case, I think that throwing an error is probably better.

[beckermr]

I don't know if scikit-learn has an official policy for missing labels per se. However, pandas does have a policy for missing things IIUIC. Put in np.nan.

[TomAugspurger]

Would expanding dummy_na to take True / False / 'keep' (or propagate) work? With the third option being your desired output? I don't think this is worth breaking API for, but your use-case seems worth supporting.

[beckermr]

Sure that would work! I do feel the need to point out that this change would fix, not break the API, but this is all semantics.