BUG: pyarrow_array_to_numpy_and_mask is not taking offset into account when creating mask

[westonpace]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd
from pyarrow import Table

df = pd.DataFrame({'int_na': [0, None, 2, 3, None, 5, 6, None, 8]}, dtype=pd.Int64Dtype())
print(df)

    int_na

0 0 
1 <NA>
 2 2 
3 3 
4 <NA>
 5 5 
6 6 
7 <NA> 
8 8

Table.from_pandas(df).slice(2, None).to_pandas()

  int_na
0 2
1 <NA>
2 1
3 5
4 <NA>
5 1
6 8

Problem description

The pandas array does not accurately reflect the pyarrow array.

Expected Output

Table.from_pandas(df).slice(2, None).to_pandas()

  int_na
0 2
1 3
2 <NA>
3 5
4 6
5 <NA>
6 8

Output of pd.show_versions()

INSTALLED VERSIONS

commit : b5958ee
python : 3.9.1.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-58-generic
Version : #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.1.5
numpy : 1.19.4
pytz : 2020.4
dateutil : 2.8.1
pip : 20.3.3
setuptools : 49.6.0.post20201009
Cython : 0.29.21
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 3.0.0.dev391+g26aef88b5.d20201216
pytables : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[jorisvandenbossche]

The issue is here:

pandas/pandas/core/arrays/_arrow_utils.py

Lines 29 to 35 in 122d502

	data = np.frombuffer(buflist[1], dtype=dtype)[arr.offset : arr.offset + len(arr)]
	bitmask = buflist[0]
	if bitmask is not None:
	mask = pyarrow.BooleanArray.from_buffers(
	pyarrow.bool_(), len(arr), [None, bitmask]
	)
	mask = np.asarray(mask)

We take the offset into account when converting the actual data, but not when converting the mask.
The BooleanArray.from_buffers also has an offset keyword, so that might be a solution (didn't check yet if it actually does what we need here)