setting column value with .loc based on a condition

[jGaboardi]

Code Sample, a copy-pastable example if possible

>>> import pandas as pd
>>> from shapely.geometry import LineString

>>> line_1 = LineString(((0,0), (1,1)))
>>> line_2 = LineString(((3,3), (5,5)))

>>> df = pd.DataFrame(data={"line_id":"999", "geometry":[line_1]})
>>> print("inital dataframe:\n%s\n" % df)

# Isolate column value
>>> condition = df["line_id"] == "999"
>>> print("location of condition:\n%s\n" % df.loc[condition, "geometry"])

# [Fail] Set geometry value by the location of condition
>>> try:
>>>     df.loc[condition,  "geometry"] = line_2
>>> except ValueError as e:
>>>     print(e, "\n")

# [Pass] Set geometry value by index location
>>> df.loc[0, "geometry"] = line_2
>>> print(df)

Problem description

Migrated from geopandas/geopandas#831.

The flexibility to set values based on a certain condition when index location is unknown is critical for some spatial workflows. This seems to be unexpected behavior in pandas (though I may be wrong). Also, as @martinfleis suggested, it may be related to np.array(LineString(((0,0), (1,1)))) returning an array of coordinates.

Expected Output

see problem description

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]

INSTALLED VERSIONS

commit : None
python : 3.7.3.final.0
python-bits : 64
OS : Darwin
OS-release : 19.0.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 0.25.1
numpy : 1.17.2
pytz : 2019.3
dateutil : 2.8.0
pip : 19.2.3
setuptools : 41.4.0
Cython : 0.29.13
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 7.8.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None

[jbrockmendel]

cc @jorisvandenbossche might be geopandas related

[martinfleis]

@jbrockmendel it was reported to geopandas but we found out that the issue is in pandas as seen above. So even though we might be able to do some kind of patch in geopandas, it does not resolve the issue.

[jbrockmendel]

Looks like in pandas.core.internals.blocks.setitem on L841 we effectively do arr_value = np.array(line_2), which gives a 2D array because of (I guess) __array_interface__. To fix this in pandas, we'd need to make the check for "should we call np.array here" a few lines up smarter. PR would be welcome.

Don't these objects live inside of a geopandas ExtensionArray? I would expect that skirt this problem nicely.

[martinfleis]

Don't these objects live inside of a geopandas ExtensionArray? I would expect that skirt this problem nicely.

@jbrockmendel If it is in GeometryArray (ExtensionArray), then the issue is not present.

As you say, the check needs to be smarter, because if you try to pass shapely.MultiLineString instead of shapely.LineString as above, it fails as well, but at different point. That fails even for GeometryArray and standard loc (without condition as above), although at different points. I assume it is related and should be ideally fixed by the same PR. MRE below.

import pandas as pd
from shapely.geometry import LineString

line_1 = LineString(((0,0), (1,1)))
line_2 = LineString(((3,3), (5,5)))

df = pd.DataFrame(data={"line_id":"999", "geometry":[line_1]})
multi = line_1.union(line_2)

df.loc[0,  "geometry"] = multi

Traceback:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-36-9a8cad09ef06> in <module>
----> 1 df.loc[0,  "geometry"] = multi

~/anaconda3/envs/geo_dev/lib/python3.8/site-packages/pandas/core/indexing.py in __setitem__(self, key, value)
    203             key = com.apply_if_callable(key, self.obj)
    204         indexer = self._get_setitem_indexer(key)
--> 205         self._setitem_with_indexer(indexer, value)
    206 
    207     def _validate_key(self, key, axis: int):

~/anaconda3/envs/geo_dev/lib/python3.8/site-packages/pandas/core/indexing.py in _setitem_with_indexer(self, indexer, value)
    591             # actually do the set
    592             self.obj._consolidate_inplace()
--> 593             self.obj._data = self.obj._data.setitem(indexer=indexer, value=value)
    594             self.obj._maybe_update_cacher(clear=True)
    595 

~/anaconda3/envs/geo_dev/lib/python3.8/site-packages/pandas/core/internals/managers.py in setitem(self, **kwargs)
    558 
    559     def setitem(self, **kwargs):
--> 560         return self.apply("setitem", **kwargs)
    561 
    562     def putmask(self, **kwargs):

~/anaconda3/envs/geo_dev/lib/python3.8/site-packages/pandas/core/internals/managers.py in apply(self, f, axes, filter, do_integrity_check, consolidate, **kwargs)
    436                     kwargs[k] = obj.reindex(b_items, axis=axis, copy=align_copy)
    437 
--> 438             applied = getattr(b, f)(**kwargs)
    439             result_blocks = _extend_blocks(applied, result_blocks)
    440 

~/anaconda3/envs/geo_dev/lib/python3.8/site-packages/pandas/core/internals/blocks.py in setitem(self, indexer, value)
    900 
    901         # value must be storeable at this moment
--> 902         arr_value = np.array(value)
    903 
    904         # cast the values to a type that can hold nan (if necessary)

TypeError: float() argument must be a string or a number, not 'LineString'

[AliOmran19991]

Hi @martinfleis
So no way to run your above code?

multi = line_1.union(line_2)
df.loc[0, "geometry"] = multi
Thanks.

[martinfleis]

@AliOmran19991 Probably not in pure pandas. There is a way in geopandas - you have to assign it to GeoSeries and then move it back to gdf.

import geopandas as gpd
from shapely.geometry import LineString

line_1 = LineString(((0,0), (1,1)))
line_2 = LineString(((3,3), (5,5)))

df = gpd.GeoDataFrame(data={"line_id":"999", "geometry":[line_1]})
multi = line_1.union(line_2)

geom = df.geometry
geom.loc[0] = multi
df['geometry'] = geom

[jorisvandenbossche]

The problem we are running into here is the general problem that comes up regularly: the dividing line between "scalar" objects and "lists/arrays of scalar objects" is not a clear line. And Pandas has especially problems with scalars that are iterable themselves (I think @jbrockmendel we also ran into this few months ago with one of your groupby refactoring PRs).

That comes up for geopandas because certain geometries are iterable. It also comes up when eg storing lists or dicts in a column (which is currently not "really" supported, you can do it in an object dtype, but if we want to add a more proper list-type, we will also run into this).

[jbrockmendel]

Two more xrefs that come to mind as potential fixes:

#29557 was specifically intended to put all nested-listlike logic in one place
#27462 to have is_scalar recognize EA-scalars