in consistent result with replace(np.nan, None, inplace=True)

[fsck-mount]

I'm trying to replace np.nan with None, so that I can query the parquet files from presto like is null or is not null.
I've done
df.column_name.replace(np.nan, None, inplace=True)

Expected it to fill 'nan' with None. But, it will some of the columns with the value from columns where it is not nan.

But I couldn't understand why it filled another additional fields, and why only some of the fields filled up why not all though it is not expected behaviour?

>>> data = [
... {'hello': 1, 'mad': 2, 'world': 3},
... {'mad': 2, 'world': 3},
... {'world': 1}
... ]
>>> df = pd.DataFrame(data)
>>> df
   hello  mad  world
0    1.0  2.0      3
1    NaN  2.0      3
2    NaN  NaN      1
>>> df.hello.dropna()
0    1.0
Name: hello, dtype: float64
>>> import numpy as np
>>> df.hello.replace(np.nan, None, inplace=True)
>>> df.hello
0    1.0
1    1.0
2    1.0
Name: hello, dtype: float64
>>>

INSTALLED VERSIONS

commit: None
python: 2.7.12.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-1032-aws
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.20.3
pytest: 3.2.1
pip: 9.0.1
setuptools: 36.4.0
Cython: 0.26.1
numpy: 1.13.1
scipy: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: 0.1.2
pandas_gbq: None
pandas_datareader: None

[jreback]

you would have to show a minimal reproducible example that can be copy pasted.

[CRiddler]

Use df.where instead.

>>> df
    A    B
0  1.0  NaN
1  2.0  3.0
2  3.0  1.0
3  NaN  NaN
>>> df.where(pd.notnull(df), None)
    A     B
0     1  None
1     2     3
2     3     1
3  None  None

See:
#1972 and
https://stackoverflow.com/questions/14162723/replacing-pandas-or-numpy-nan-with-a-none-to-use-with-mysqldb

[fsck-mount]

@jreback sorry for the delayed response. Updated the question, please check.
@CRiddler yep, it's related to the above issue. Figured out, and done using where.

However, as in case of #1972, filling the value with 0 fine, somewhat acceptable. But, filling the value with other rows values is not.

[CRiddler]

The issue you're hitting is that None is the default argument for value, therefore when you use None as the argument, the replace method is falling back to the method argument. Which in this case is 'pad'.

You can circumvent this behavior by passing the to_replace argument a dictionary that maps np.nan to None. Building on your example…

>>> data = [
... {'hello': 1, 'mad': 2, 'world': 3},
... {'mad': 2, 'world': 3},
... {'world': 1}
... ]
>>> 
>>> df = pd.DataFrame(data)
>>> df
   hello  mad  world
0    1.0  2.0      3
1    NaN  2.0      3
2    NaN  NaN      1


>>> df['hello'].replace(np.nan, None)
0    1.0
1    1.0
2    1.0

Note that with the above command we're ACTUALLY telling pandas:

df['hello'].replace(np.nan, method='pad') which is why the value of 1 is being propagated.

Achieve desired result by passing a dictionary into replace.

>>> df['hello'].replace({np.nan:None})
0       1
1    None
2    None

[fsck-mount]

@CRiddler Thanks for the explanation. Closing the issue as it is not a bug.

[fsck-mount]

@CRiddler One small question, what's the behaviours for method="pad", not sure whether you might have observed it or not. In the original question I posted before editing., not all values are changed. Most of the values are changed. And others are left with np.nan

Why this occurred ?

[CRiddler]

method='pad' is the same as method='fill', so in this case you were forward filling the last observable value over nan values.

>>> df
   Hello  Mad  World
0    1.0  NaN    1.0
1    2.0  NaN    2.0
2    3.0  3.0    NaN
3    NaN  4.0    NaN
4    NaN  5.0    5.0


>>> df['Hello'].replace(np.nan, None)
0    1.0
1    2.0
2    3.0
3    3.0
4    3.0
Name: Hello, dtype: float64

>>> df['Hello'].ffill()              
0    1.0
1    2.0
2    3.0
3    3.0
4    3.0
Name: Hello, dtype: float64

I can't remember the specifics of your original question, but Series.replace(np.nan, None) should behave the same as Series.ffill()

[fsck-mount]

@CRiddler

you were forward filling the last observable value over nan values

This clarifies my question. Thanks :) 👍