BUG: Different results from DataFrame.apply and str accessor

[pLeBlanc93]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

# Your code here
import pandas as pd

print(pd.__version__)

df = pd.DataFrame(zip('abc', 'def'))
print(df.apply(lambda f: "/".join(f), axis=1))
print(df.apply(lambda f: "/".join(f.str.upper()), axis=1))

Problem description

Using the str accessor gives different results on 1.1.5 and 1.2.x:

1.1.5
0    a/d
1    b/e
2    c/f
dtype: object
0    A/D
1    B/E
2    C/F
dtype: object

1.2.0+22.g6cdb4e7fb
0    a/d
1    b/e
2    c/f
dtype: object
0    A/D
1    A/D
2    A/D
dtype: object

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 6cdb4e7
python : 3.7.9.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : Intel64 Family 6 Model 63 Stepping 2, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None

pandas : 1.2.0+22.g6cdb4e7fb
numpy : 1.19.5
pytz : 2020.5
dateutil : 2.8.1
pip : 20.3.3
setuptools : 51.0.0.post20201207
Cython : 0.29.21
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[xiechen0620]

I noticed the same issue which @pLeBlanc93 post.

When using str accessor mothed with df.apply on dtype of series is object in version 1.2 ,
I could get excepted result when using axis=0,
and all result will be same as the first rows when using axis=1.

On dtype is 'string', everything is OK

here is my test code

temp_df = pd.DataFrame(
    data=np.random.rand(5, 2)
)

as type of 'str'

(temp_df
 .astype(str)
 .apply(
     func=lambda ser: ser.str.replace(' ', ''),
     **axis=1**
 )
)

result:

	0	1
0	0.5970435390448061	0.7482773295278011
1	0.5970435390448061	0.7482773295278011
2	0.5970435390448061	0.7482773295278011
3	0.5970435390448061	0.7482773295278011
4	0.5970435390448061	0.7482773295278011

(temp_df
 .astype(str)
 .apply(
     func=lambda ser: ser.str.replace(' ', ''),
     **axis=0**
 )
)

result:

	0	1
0	0.5970435390448061	0.7482773295278011
1	0.6590413779662467	0.3598485907864998
2	0.4987766492919723	0.28087749748165025
3	0.9220290923949616	0.6242588924861366
4	0.6872302443544485	0.2683357660138924

as type of 'string'

(temp_df
 .astype('string')
 .apply(
     func=lambda ser: ser.str.replace(' ', ''),
     **axis=1**
 )
)

result:

	0	1
0	0.5970435390448061	0.7482773295278011
1	0.6590413779662467	0.3598485907864998
2	0.4987766492919723	0.28087749748165025
3	0.9220290923949616	0.6242588924861366
4	0.6872302443544485	0.2683357660138924

(temp_df
 .astype('string')
 .apply(
     func=lambda ser: ser.str.replace(' ', ''),
     **axis=0**
 )
)

result:

	0	1
0	0.5970435390448061	0.7482773295278011
1	0.6590413779662467	0.3598485907864998
2	0.4987766492919723	0.28087749748165025
3	0.9220290923949616	0.6242588924861366
4	0.6872302443544485	0.2683357660138924

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 3e89b4c
python : 3.9.1.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.18362
machine : AMD64
processor : Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : Japanese_Japan.932

pandas : 1.2.0
numpy : 1.19.5
pytz : 2020.5
dateutil : 2.8.1
pip : 20.3.3
setuptools : 51.1.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 1.3.7
lxml.etree : None
html5lib : None
pymysql : 0.10.1
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.19.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.3
numexpr : None
odfpy : None
openpyxl : 3.0.5
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.6.0
sqlalchemy : 1.3.22
tables : None
tabulate : None
xarray : None
xlrd : 2.0.1
xlwt : None
numba : None

[xiechen0620]

Beside that, I noticed when using old dtype, id of all slices would be same if axis=1, but unique when using new dtype.
I have no idea if there is any connection.

old dtype

(temp_df
 .apply(
     func=id,
     axis=1
 )
)

result

0    3088070195376
1    3088070195376
2    3088070195376
3    3088070195376
4    3088070195376
dtype: int64

new dtype

(temp_df
.convert_dtypes() 
.apply(
     func=id,
     axis=1
 )
)

result

0    3088070246512
1    3088070249440
2    3088070248672
3    3088070248096
4    3088070249056
dtype: int64

[asishm]

behavior changed in fbd13a9

REF: Dispatch string methods to ExtensionArray #36357

cc @TomAugspurger

[SKrPl]

Going through the code I found the following:
Due to the optimizations done in 34909, for FrameColumnApply the Series object is created only once in the series_genarator method and the value of the series is changed. As the series object is created only once, pandas.core.strings.accessor.StringMethods is also instantiated only once. Due to this the self._array attribute is only set once i.e. when the instantiation happens for the first time, and it contains the value of first rows. Due to this the output has transformed data of first row.
This doesn't happen when axis=0 because there FrameRowApply is used and for each column a new Series object is instantiated. So, the StringMethod class is also instantiated for each column, as a result of which self._array attribute has the correct data.

[simonjayhawkins]

@jbrockmendel any thoughts here.

Although the regression is in #36357, imo the problem is due to #34909

#34909 has caused several regressions. do we want to keep that performance gain and try to patch this further.

I assume that, we have no control over other 3rd party registered accessors, so would want a fix here that works with any accessor and not just a fix for the str accessor.

@jorisvandenbossche can you reproduce this with geopandas or am I wrong in my assumption and the problem is with the str accessor.

[jbrockmendel]

As the series object is created only once, pandas.core.strings.accessor.StringMethods is also instantiated only once

Yah, best guess is we need to clear the cache between steps of the iteration