BUG: setting 1.02 into float32 Series upcasts to float64

[batterseapower]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

x = pd.Series(index=['a', 'b'], dtype='f4')
x.iloc[0] = 1.02
print(x.dtype)

Issue Description

In 2.1.1, this code prints the following warning:

FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '1.02' has dtype incompatible with float32, please explicitly cast to a compatible dtype first.

It then shows that the post-assignment dtype is float64, which is presumably why the warning is being raised: Pandas is converting the backing array from f4 to f8 behind the scenes for some reason.

Do you really intend this to raise an exception in the future? It seems like it would be better to just assign the literal to to the existing float32 array. I checked what earlier versions of Pandas did, and it looks like Pandas 1.4.4 kept the dtype as float32 after the assignment, while Pandas 1.5.2 changes the dtype to float64. I'm not sure whether that was intentional or not - it doesn't seem to be explicitly called out in the release notes for e.g. 1.5.0 (though it's possible I missed something).

The warning can of course be suppressed by writing np.float32(1.02) instead of the simple literal, but it doesn't seem great to require users to write this kind of stuff for every literal assignment.

Expected Behavior

Just assign the literal to the float32 array with no warning and no exception about dtype conversion.

Installed Versions

INSTALLED VERSIONS ------------------ commit : e86ed37 python : 3.10.10.final.0 python-bits : 64 OS : Linux OS-release : 4.18.0-348.20.1.el8_5.x86_64 Version : #1 SMP Thu Mar 10 20:59:28 UTC 2022 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_GB.UTF-8 LOCALE : en_GB.UTF-8

pandas : 2.1.1
numpy : 1.23.5
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.7.2
pip : 23.1.2
Cython : 3.0.0
pytest : 7.3.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.9.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.13.2
pandas_datareader : None
bs4 : 4.12.2
bottleneck : 1.3.7
dataframe-api-compat: None
fastparquet : None
fsspec : 2023.6.0
gcsfs : None
matplotlib : 3.7.1
numba : 0.56.4
numexpr : 2.8.7
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 10.0.1
pyreadstat : None
pyxlsb : None
s3fs : 2023.6.0
scipy : 1.10.1
sqlalchemy : None
tables : None
tabulate : None
xarray : 2023.4.2
xlrd : 2.0.1
zstandard : None
tzdata : 2023.3
qtpy : 2.3.1
pyqt5 : None

[hvsesha]

Hi @batterseapower

Yes you are right .Below code will remove the warning
x = pd.Series(index=['a', 'b'], dtype='float32')

print(x.dtype)

[batterseapower]

@hvsesha I think maybe your comment got mangled. I'm not sure exactly what you are suggesting, but I don't see that changing the dtype from "f4" to "float32" makes any difference.

I am suppressing the warning in my own code by adding a np.float32() cast around my literals, but it doesn't seem like good behaviour to warn about this or break this kind of code in a future Pandas version.

[csotto]

I am having the same problem as @batterseapower.
On my case I am updating a DataFrame row (index dtype int64 and all columns are dtype float32) from a list.
Given:
result=[1698262980000, 0.861, 0.861, 0.857, 0.858, 19336.7, 0.0, 0.0, 8295.444, 0.0, 8295.444, 0.0]
df.loc[result[0], "open" : "closed"] = result[1:] # 'open' <=> first column, 'closed' <=> last column to be updated

Will match the index column and print deprecation warning message for each numeric value, except 0.0 (!?) as follows:
FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '0.861' has dtype incompatible with float32, please explicitly cast to a compatible dtype first.
.
I am using pandas v2.1.1. My expected behaviour is exactly the same stated by @batterseapower

[rhshadrach]

cc @MarcoGorelli

[MarcoGorelli]

thanks for the report

I think this is a case when the warning is correct (the value being set is indeed changing the dtype), but it's uncovered a related bug (that the dtype shouldn't be changing in this case)

In [4]: x = pd.Series(index=['a', 'b'], dtype='float32')

In [5]: x
Out[5]:
a   NaN
b   NaN
dtype: float32

In [6]: x.iloc[0] = .02
<ipython-input-6-5b8507309c35>:1: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '0.02' has dtype incompatible with float32, please explicitly cast to a compatible dtype first.
  x.iloc[0] = .02

In [7]: x.dtype
Out[7]: dtype('float64')

Looks like, for nullable dtypes, the upcasting doesn't happen:

In [8]: x = pd.Series(index=['a', 'b'], dtype='Float32')

In [9]: x.iloc[0] = .02

In [10]: x
Out[10]:
a    0.02
b    <NA>
dtype: Float32

[rhshadrach]

We end up with float64 because of np_can_hold_element:

pandas/pandas/core/dtypes/cast.py

Lines 1829 to 1834 in 0cbe41a

	if lib.is_integer(element) or lib.is_float(element):
	casted = dtype.type(element)
	if np.isnan(casted) or casted == element:
	return casted
	# otherwise e.g. overflow see TestCoercionFloat32
	raise LossySetitemError

The check casted == element fails there for anything that doesn't have a clean binary representation. E.g.

x = pd.Series(index=['a', 'b'], dtype='float32')
x.iloc[0] = 0.5
print(x.dtype)
float32

[mroeschke]

I think in np_can_hold_element we should be able to check if the value is in the np.finfo bounds instead of using casted == element

[MarcoGorelli]

I think in np_can_hold_element we should be able to check if the value is in the np.finfo bounds instead of using casted == element

The issue is that if we only check the bounds then we get these kinds of results:

In [9]: ser = Series([1.1, 2.2, 3.3, 4.4], dtype=np.float32)

In [10]: ser[0] = np.uint32(np.iinfo(np.uint32).max)

In [11]: ser[0]
Out[11]: 4294967300.0

In [12]: np.uint32(np.iinfo(np.uint32).max)
Out[12]: 4294967295

To be honest I think the warning (and future error) is correct here, as float32 can't hold 0.2:

In [28]: np.float32(.2).item()
Out[28]: 0.20000000298023224

In [29]: np.float64(.2).item()
Out[29]: 0.2