BUG: Bitwise operators for AND ( & ) and OR( | ) doesn't work with pyarrow integers

[Rob-DL]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

# Using Numpy:
import numpy as np
import pandas as pd
df_np = pd.DataFrame({"x": [1,2,3]}, dtype=np.int64)
df_np["x"] & 2

# Work as expected, see issue description for output


# Using Arrow
df_arrow = pd.DataFrame({"x": [1,2,3]}, dtype="int64[pyarrow]")
df_arrow["x"] & 2

# Raises exception, see below for traceback

Issue Description

Output using Numpy

Out[129]:
0    0
1    2
2    2
Name: x, dtype: int64

Output using Arrow

ArrowNotImplementedError                  Traceback (most recent call last)
<ipython-input-6-3cfcafe716ae> in <cell line: 2>()
      1 df_arrow = pd.DataFrame({"x": [1,2,3]}, dtype="int64[pyarrow]")
----> 2 df_arrow["x"] & 2

~\.venv\analytics\lib\site-packages\pandas\core\ops\common.py in new_method(self, other)
     79         other = item_from_zerodim(other)
     80
---> 81         return method(self, other)
     82
     83     return new_method

~\.venv\analytics\lib\site-packages\pandas\core\arraylike.py in __and__(self, other)
     68     @unpack_zerodim_and_defer("__and__")
     69     def __and__(self, other):
---> 70         return self._logical_method(other, operator.and_)
     71
     72     @unpack_zerodim_and_defer("__rand__")

~\.venv\analytics\lib\site-packages\pandas\core\series.py in _logical_method(self, other, op)
   6105         rvalues = extract_array(other, extract_numpy=True, extract_range=True)
   6106
-> 6107         res_values = ops.logical_op(lvalues, rvalues, op)
   6108         return self._construct_result(res_values, name=res_name)
   6109

~\.venv\analytics\lib\site-packages\pandas\core\ops\array_ops.py in logical_op(left, right, op)
    384     if should_extension_dispatch(lvalues, rvalues):
    385         # Call the method on lvalues
--> 386         res_values = op(lvalues, rvalues)
    387
    388     else:

~\.venv\analytics\lib\site-packages\pandas\core\ops\common.py in new_method(self, other)
     79         other = item_from_zerodim(other)
     80
---> 81         return method(self, other)
     82
     83     return new_method

~\.venv\analytics\lib\site-packages\pandas\core\arraylike.py in __and__(self, other)
     68     @unpack_zerodim_and_defer("__and__")
     69     def __and__(self, other):
---> 70         return self._logical_method(other, operator.and_)
     71
     72     @unpack_zerodim_and_defer("__rand__")

~\.venv\analytics\lib\site-packages\pandas\core\arrays\arrow\array.py in _logical_method(self, other, op)
    537
    538     def _logical_method(self, other, op):
--> 539         return self._evaluate_op_method(other, op, ARROW_LOGICAL_FUNCS)
    540
    541     def _arith_method(self, other, op):

~\.venv\analytics\lib\site-packages\pandas\core\arrays\arrow\array.py in _evaluate_op_method(self, other, op, arrow_funcs)
    529             else:
    530                 pa_scalar = pa.scalar(other)
--> 531             result = pc_func(self._data, pa_scalar)
    532         else:
    533             raise NotImplementedError(

~\.venv\analytics\lib\site-packages\pyarrow\compute.py in wrapper(memory_pool, *args)
    237             if args and isinstance(args[0], Expression):
    238                 return Expression._call(func_name, list(args))
--> 239             return func.call(args, None, memory_pool)
    240     else:
    241         def wrapper(*args, memory_pool=None, options=None, **kwargs):

~\.venv\analytics\lib\site-packages\pyarrow\_compute.pyx in pyarrow._compute.Function.call()

~\.venv\analytics\lib\site-packages\pyarrow\error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~\.venv\analytics\lib\site-packages\pyarrow\error.pxi in pyarrow.lib.check_status()

ArrowNotImplementedError: Function 'and_kleene' has no kernel matching input types (int64, int64)

Expected Behavior

I expect that the behaviour of the bitwise operators & and | is the same for numpy and pyarrow integer types.

Installed Versions

INSTALLED VERSIONS

commit : 0f43794
python : 3.10.11.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19044
machine : AMD64
processor : Intel64 Family 6 Model 140 Stepping 1, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 2.0.3
numpy : 1.23.5
pytz : 2023.3
dateutil : 2.8.2
setuptools : 68.0.0
pip : 23.0.1
Cython : None
pytest : 7.4.0
hypothesis : None
sphinx : 6.2.1
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 7.34.0
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : 2023.6.0
gcsfs : None
matplotlib : 3.7.2
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 11.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.11.1
snappy : None
sqlalchemy : 2.0.19
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[mroeschke]

Thanks. From the traceback looks like pyarrow doesn't support bitwise operators between integers. Could you open an issue in the arrow repo?

[yashmonk123]

hello!! I want to take this ,can you assign me this . Also please specify in which module i have to work?

[yashmonk123]

I have checked that pyarrow is compatible with Python 3.5, 3.6, 3.7 and 3.8. and your installed python : 3.10.11.final.0 . you can try out with this import numpy as np
import pandas as pd
import pyarrow as pa

Create a PyArrow DataFrame

df_arrow = pd.DataFrame({"x": [1, 2, 3]}, dtype="int64[pyarrow]")

Convert the PyArrow column to a NumPy array

x_values = df_arrow["x"].values.to_numpy()

Perform the bitwise operation using NumPy

result = x_values & 2

Convert the result back to a PyArrow array

result_array = pa.array(result, type=pa.int64())

Create a new PyArrow DataFrame with the result

df_result = pd.DataFrame({"result": [result_array]})

print(df_result)

[Rob-DL]

Thanks for the suggestion, I'm aware of this workaround and can confirm that it works. The reason I think this should be fixed is to make the transition from pandas 1.x to 2.x as seamless as possible. Also, bitwise operations are pretty fundamental and should just work. According to https://arrow.apache.org/docs/python/generated/pyarrow.compute.and_kleene.html#pyarrow.compute.and_kleene these operations are supported by Arrow, but I don't know why they're failing.

@yashmonk123 If you'd like to pick this up, I'd very much appreciate it. To get the issue assigned I think you can use the take command (see https://pandas.pydata.org/docs/dev/development/contributing.html). Other than that, I don't think I can be of much help.

[mroeschke]

For this issue to be fixed, ideally it should be upstream in arrow https://github.com/apache/arrow

[jorisvandenbossche]

I think it are the existing "bit_wise_and" and "bit_wise_or" compute functions that we want to use (and those are implemented specifically for integers)