BUG: incompatible dtype when creating string column with loc

[mortewle]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
df = pd.DataFrame({"x": [1]})
df.loc[df["x"] == 1, "y"] = "1"

Issue Description

I get a FutureWarning when conditionally creating a new string column in loc.

It does not happen if the column is numeric, assumably because float64 is the default dtype.

The warning dissappears when setting the column to NA beforehand, but I assume this is not meant as a nessecary additional step for non-numeric columns only.

FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '1' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
df.loc[df["x"] == 1, "y"] = "1"

Expected Behavior

No warning.

Installed Versions

2.1.0

[mroeschke]

cc @MarcoGorelli

@c0mm0d0re happy to have a PR to fix this

[MarcoGorelli]

Thanks for the ping - just got back from holiday will take a look

[MarcoGorelli]

OK looks like this is due to

pandas/pandas/core/indexing.py

Lines 1844 to 1845 in c74a071

	# FIXME: GH#42099#issuecomment-864326014
	self.obj[key] = infer_fill_value(value)

For now I'd suggest doing something like

In [18]: ser = pd.Series(index=df.index, dtype=object)

In [19]: ser
Out[19]:
0    NaN
1    NaN
dtype: object

In [20]: ser[df['x']==1] = '1'

In [21]: ser
Out[21]:
0      1
1    NaN
dtype: object

In [22]: df['y'] = ser

In [23]: df
Out[23]:
   x    y
0  1    1
1  2  NaN

[MichaelTiemannOSC]

Yes. I note that StringDtype has the na_value <NA>, but the raw string type has conflicting treatment based on scalar (np.nan) or array (string not float) context.