Skip to content

fix: large_list and large_string unit test for read_parquet_metadata #3089

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 4, 2025
Merged

fix: large_list and large_string unit test for read_parquet_metadata #3089

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e15aa08
Adds unit test for pyarrow large lists and strings
ashrielbrian Feb 4, 2025
87097d5
Fixes typo in test func name
ashrielbrian Feb 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions tests/unit/test_s3_parquet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,27 @@ def test_read_parquet_metadata_nonexistent_file(path):
wr.s3.read_parquet_metadata(path + "non-existent-file.parquet")


def test_read_parquet_metadata_large_dtype(path):
schema = pa.schema(
[
pa.field("c0", pa.large_list(pa.large_string())),
pa.field("c1", pa.large_string()),
]
)
c0 = pa.array([["a", "b", "c"], ["d", "e", "f"], ["g", "h", "i"]])
c1 = pa.array(["a", "b", "c"])
df = pa.table([c0, c1], schema=schema)

# use pyarrow-backed dataframe to simulate the large_list and large_string dtypes
pandas_df = df.to_pandas(types_mapper=lambda pa_dtype: pd.ArrowDtype(pa_dtype))

wr.s3.to_parquet(pandas_df, path)
columns_types, _ = wr.s3.read_parquet_metadata(path)
assert len(columns_types) == len(df.columns)
assert columns_types.get("c0") == "array<string>"
assert columns_types.get("c1") == "string"


@pytest.mark.parametrize(
"partition_cols",
[
Expand Down