AWS SDK for pandas 3.12.0

Notable Changes ⚠️

• AWS Lambda Layers: pyarrow was upgraded to 20.0.0

Features / Enhancements 🚀

• feat: add pyarrow_additional_kwargs to athena.to_iceberg by @jaidisido in #3094
• feat: add dtype argument to delete_from_iceberg by @jaidisido in #3099
• feat: add redshift and rds data api query params by @kukushking in #3111
• chore: ray 2.45 by @kukushking in #3143
• chore: upgrade setuptools & pin cmake by @kukushking in #3152
• chore: update pyarrow version by @EthanBunce in #3156

Bugfixes 🐛

• fix: Supports pyarrow's large_list by @ashrielbrian in #3086
• fix: large_list and large_string unit test for read_parquet_metadata by @ashrielbrian in #3089
• fix: handle partitions with empty table in read_parquet with dataset=True by @cournape in #2983
• fix: oracle decimal error by @kukushking in #3103
• chore: add redshift COPY with SERIALIZETOJSON test case by @kukushking in #3104
• fix: Changing dtype lowercasing to start of athena2pyarrow function by @DanielMy777 in #3122
• fix: concat with union categories by @kukushking in #3127
• fix: upgrade CDK RDS serverless databases to V2 by @jaidisido in #3128
• fix(athena): ensure thread safety when reading local Athena cache by @david-mateo in #3137

Security / Dependency Updates 🛡️

• chore(deps): bump the production-dependencies group with 2 updates by @dependabot in #3071
• chore(deps-dev): bump the development-dependencies group with 3 updates by @dependabot in #3072
• chore(deps): bump the production-dependencies group with 3 updates by @dependabot in #3075
• chore(deps-dev): bump the development-dependencies group with 3 updates by @dependabot in #3076
• chore: update dependecies (geopandas 1+) by @kukushking in #3079
• chore(deps): bump botocore from 1.36.3 to 1.36.6 in the production-dependencies group by @dependabot in #3081
• chore(deps-dev): bump the development-dependencies group across 1 directory with 3 updates by @dependabot in #3083
• chore(deps): bump the production-dependencies group with 2 updates by @dependabot in #3087
• chore(deps-dev): bump the development-dependencies group with 5 updates by @dependabot in #3088
• chore(deps): bump the production-dependencies group with 3 updates by @dependabot in #3097
• chore(deps-dev): bump the development-dependencies group with 5 updates by @dependabot in #3098
• chore(deps): bump the production-dependencies group with 3 updates by @dependabot in #3101
• chore(deps-dev): bump the development-dependencies group with 4 updates by @dependabot in #3102
• chore(deps): bump the production-dependencies group with 4 updates by @dependabot in #3105
• chore(deps-dev): bump the development-dependencies group with 5 updates by @dependabot in #3106
• chore(deps-dev): bump the development-dependencies group with 5 updates by @dependabot in #3109
• chore(deps): bump jinja2 from 3.1.5 to 3.1.6 by @dependabot in #3110
• chore(deps): bump the production-dependencies group with 2 updates by @dependabot in #3108
• chore(deps): bump aiohttp from 3.11.13 to 3.11.14 in the production-dependencies group by @dependabot in #3112
• chore(deps-dev): bump the development-dependencies group with 3 updates by @dependabot in #3113
• chore(deps): bump setuptools from 76.0.0 to 77.0.3 in the production-dependencies group by @dependabot in #3115
• chore(deps-dev): bump the development-dependencies group with 4 updates by @dependabot in #3116
• chore(deps-dev): bump the development-dependencies group with 4 updates by @dependabot in #3120
• chore(deps): bump setuptools from 77.0.3 to 78.1.0 in the production-dependencies group by @dependabot in #3119
• chore(deps-dev): bump the development-dependencies group with 4 updates by @dependabot in #3126
• chore(deps): bump the production-dependencies group with 4 updates by @dependabot in #3125
• chore(deps-dev): bump the development-dependencies group across 1 directory with 4 updates by @dependabot in #3133
• chore(deps-dev): bump h11 from 0.14.0 to 0.16.0 by @dependabot in #3139
• chore(deps-dev): bump the development-dependencies group with 5 updates by @dependabot in #3140
• chore(deps-dev): bump the development-dependencies group with 4 updates by @dependabot in #3141
• chore(deps): bump the production-dependencies group across 1 directory with 4 updates by @dependabot in #3142
• chore(deps): bump the production-dependencies group with 2 updates by @dependabot in #3146
• chore(deps-dev): bump the development-dependencies group with 5 updates by @dependabot in #3147
• chore(deps): bump the production-dependencies group with 2 updates by @dependabot in #3150
• chore(deps-dev): bump the development-dependencies group with 4 updates by @dependabot in #3151
• chore(deps-dev): bump tornado from 6.4.2 to 6.5.1 by @dependabot in #3153

Documentation / Workflows 📚

• chore: Prepare 3.12.0 by @kukushking in #3154
• chore: update layers.rst by @EthanBunce in #3158

New Contributors

• @ashrielbrian made their first contribution in #3086
• @cournape made their first contribution in #2983
• @DanielMy777 made their first contribution in #3122
• @david-mateo made their first contribution in #3137
• @EthanBunce made their first contribution in #3156

Full Changelog: 3.11.0...3.12.0

AWS SDK for pandas 3.11.0

Notable Changes ⚠️

• AWS SDK for pandas now supports Python 3.13! 🎉
• Python 3.8 is no longer supported (reached end-of-life Oct 7 2024) 🚫
• AWS Lambda Layers: pyarrow was upgraded to 18.1.0
• AWS Lambda Layers: numpy was upgraded to 2.2.1

Features / Enhancements 🚀

• add support for Python 3.13 & deprecate Python 3.8 by @kukushking in #3045
• return opensearch aggregation top hits by @kukushking in #3059

Bugfixes 🐛

• fix: layers - add gcc10 to AL2 build image; build numpy and pandas from source by @kukushking in #3069

Security / Dependency Updates 🛡️

• update pyarrow to 18.1.0 in lambda layers by @kukushking in #3065
• unpin numpy in lambda layers (allow numpy 2.x) by @kukushking in #3068
• chore(deps): bump the production-dependencies group with 2 updates by @dependabot in #3039
• chore(deps-dev): bump the development-dependencies group with 4 updates by @dependabot in #3040
• chore(deps): bump the production-dependencies group with 3 updates by @dependabot in #3043
• chore(deps-dev): bump the development-dependencies group across 1 directory with 5 updates by @dependabot in #3047
• chore(deps): bump jinja2 from 3.1.4 to 3.1.5 by @dependabot in #3055
• chore(deps-dev): bump the development-dependencies group across 1 directory with 11 updates by @dependabot in #3057
• chore(deps): bump the production-dependencies group across 1 directory with 2 updates by @dependabot in #3064

Documentation / Workflows 📚

• exclude py3.13 windows minimal test by @jaidisido in #3058
• prepare 3.11.0 release by @kukushking in #3062
• chore: update layers.rst by @kukushking in #3070

Full Changelog: 3.10.1...3.11.0

AWS SDK for pandas 3.10.1

Bug fixes 🐛

• fix: update references in introduction notebook by @emmanuel-ferdman in #3009
• fix: read parquet file in chunked mode per row group by @FredericKayser in #3016
• fix: add missing raise statement in RS Data API by @jaidisido in #3025

Documentation 📚

• chore: Prepare 3.10.1 release by @kukushking in #3031

Security / Dependency Updates 🛡️

• chore(deps-dev): bump the development-dependencies group across 1 directory with 8 updates by @dependabot in #3013
• chore(deps): bump the production-dependencies group with 4 updates by @dependabot in #3012
• chore(deps): bump the production-dependencies group with 4 updates by @dependabot in #3014
• chore(deps-dev): bump the development-dependencies group with 4 updates by @dependabot in #3015
• chore(deps-dev): bump the development-dependencies group with 4 updates by @dependabot in #3023
• chore(deps): bump the production-dependencies group with 4 updates by @dependabot in #3022
• chore(deps-dev): bump tornado from 6.4.1 to 6.4.2 by @dependabot in #3027
• chore(deps): bump the production-dependencies group with 3 updates by @dependabot in #3028
• chore(deps-dev): bump the development-dependencies group with 3 updates by @dependabot in #3029

New Contributors

• @FredericKayser made their first contribution in #3016

Full Changelog: 3.10.0...3.10.1

AWS SDK for pandas 3.10.0

Features

• feat: Support numpy 2.0 by @LeonLuttenberger in #2944
• feat(redshift): Automatically add new DataFrame columns to Redshift tables during write operation by @jack-dell in #2948
• feat: modify_refresh_interval flag in opensearch index_documents by @AvihaiSam in #2980
• feat: support postgresql array types by @kukushking in #2985

Bug fixes 🐛

• fix: OpenSearch import errors by @LeonLuttenberger in #2939
• fix(athena): Enable use of dataframe type, in athena2pyarrow type by @eliabrio in #2953
• fix: correct skip.header.line.count value to 1 by @jaidisido in #2977
• fix: data_api uuid typehint by @jethroguce in #2961
• fix: enforce ssl by @kukushking in #2979
• fix: handle case of str to bool env variable in config by @jaidisido in #2966
• fix: return only "current" iceberg columns by @kukushking in #2982
• fix: Athena read_sql_query cache errors for qmark style parametrized queries by @LeonLuttenberger in #2957
• fix: Prevent truncated Parquet files in S3 after failed CreateMultipartUpload (2987) by @rdwebster in #2993

Documentation 📚

• docs: use autodoc typehints by @LeonLuttenberger in #2940
• chore: Prepare 3.10.0 release by @kukushking in #2992

Security / Dependency Updates 🛡️

• chore(deps-dev): bump the development-dependencies group with 5 updates by @dependabot in #2942
• chore(deps): bump the production-dependencies group with 6 updates by @dependabot in #2941
• chore(deps-dev): bump jupyterlab from 4.2.4 to 4.2.5 by @dependabot in #2945
• chore(deps-dev): bump the development-dependencies group with 2 updates by @dependabot in #2947
• chore(deps): bump the production-dependencies group with 2 updates by @dependabot in #2946
• chore(deps): bump cryptography from 42.0.8 to 43.0.1 by @dependabot in #2949
• chore(deps): bump fiona from 1.9.6 to 1.10.0 by @dependabot in #2952
• chore(deps): bump the production-dependencies group with 2 updates by @dependabot in #2954
• chore(deps-dev): bump the development-dependencies group with 5 updates by @dependabot in #2955
• chore(deps): bump setuptools from 74.1.2 to 75.0.0 in the production-dependencies group by @dependabot in #2962
• chore(deps-dev): bump the development-dependencies group with 5 updates by @dependabot in #2963
• chore(deps-dev): bump the development-dependencies group with 5 updates by @dependabot in #2970
• chore(deps): bump the production-dependencies group with 3 updates by @dependabot in #2969
• chore(deps): bump the production-dependencies group with 3 updates by @dependabot in #2975
• chore(deps-dev): bump the development-dependencies group with 4 updates by @dependabot in #2976
• chore(deps): bump aiohttp from 3.10.8 to 3.10.9 in the production-dependencies group by @dependabot in #2988
• chore(deps-dev): bump the development-dependencies group with 4 updates by @dependabot in #2989

New Contributors

• @eliabrio made their first contribution in #2953
• @jack-dell made their first contribution in #2948
• @jethroguce made their first contribution in #2961
• @AvihaiSam made their first contribution in #2980
• @rdwebster made their first contribution in #2993

Full Changelog: 3.9.1...3.10.0

AWS SDK for pandas 3.9.1

Bug fixes 🐛

• bucketing error with newer version of Modin (0.31.0) by @LeonLuttenberger in #2896
• athena.read_sql_query failing for time columns by @LeonLuttenberger in #2895
• add an argument to control handling nulls in merge criteria by @brendan-cook-87 in #2892
• address Ray deprecation warnings by @LeonLuttenberger in #2929
• Add deprecation notice to s3.select_query by @LeonLuttenberger in #2928

Documentation 📚

• Add steps to install "using pip" to the "AWS Glue Python Shell Jobs" part of the install docs by @mc51 in #2911

Security Dependency Updates 🛡️

• bump aiohttp from 3.10.1 to 3.10.2 by @dependabot in #2930

New Contributors

• @brendan-cook-87 made their first contribution in #2892

Full Changelog: 3.9.0...3.9.1

AWS SDK for pandas 3.9.0

Enhancements 🎉

• Support ORC and CSV in redshift.copy_from_files function by @LeonLuttenberger in #2849
• Support different merge conditions in athena.to_iceberg function by @aldder in #2861
• Manage NULL values in athena.to_iceberg merge statement by @aldder in #2872
• Upgrade Ray to 2.30 by @jaidisido in #2870

Bug fixes 🐛

• Fix tz attribute check, it was checking dtype instead of dt by @sanrodari in #2855
• PyArrow backend trimming timestamp to date by @LeonLuttenberger in #2875
• Replace deprecated ray parallelism arg with override_num_blocks by @kukushking in #2876

Documentation 📚

• Adding NotebookVersion Parameter as specified in official AWS Docs by @DaxterXS in #2867
• Fix default value in doc string for athena_query_wait_polling_delay parameters by @niklasr22 in #2873

Security Dependency Updates 🛡️

• Bump urllib3 from 1.26.18 to 1.26.19 by @dependabot in #2859
• Bump certifi from 2024.6.2 to 2024.7.4 by @dependabot in #2886

New Contributors

• @aldder made their first contribution in #2861
• @DaxterXS made their first contribution in #2867
• @sanrodari made their first contribution in #2855
• @niklasr22 made their first contribution in #2873

Full Changelog: 3.8.0...3.9.0

AWS SDK for pandas 3.8.0

Enhancements 🎉

• support client-side parameter resolution in athena.create_ctas_table by @LeonLuttenberger in #2797
• add commit_transaction to postgres.to_sql by @jaidisido in #2795
• add columns parameters support by @jaidisido in #2814
• add overwrite_method to postgresql.to_sql by @LeonLuttenberger in #2820
• add upsert mode to sqlserver.py and corresponding tests by @AntonMantulo in #2835

Bug fixes 🐛

• correct athena timezone TypeError message by @jaidisido in #2796
• update license reference by @emmanuel-ferdman in #2798
• add missing index in athena.to_iceberg to_parquet by @jaidisido in #2799
• support json type in athena2pandas by @DavidKatz-il in #2806
• adapt parquet datasource to ray 2.20 by @jaidisido in #2809
• add primary keys when upserting in Postgres by @AntonMantulo in #2819
• add s3_output parameter to athena.delete_from_iceberg_table by @LeonLuttenberger in #2829
• fix s3.read_parquet_metadata not throwing proper error for empty path by @LeonLuttenberger in #2843

Documentation 📚

• fix enumerations in doc pages by @LeonLuttenberger in #2821
• fix readthedocs errors by pinning sphinx version by @LeonLuttenberger in #2822

Security Dependency Updates 🛡️

• bump werkzeug from 3.0.1 to 3.0.3 by @dependabot in #2807
• bump jinja2 from 3.1.3 to 3.1.4 by @dependabot in #2808
• bump requests from 2.31.0 to 2.32.0 by @dependabot in #2827
• bump pymysql from 1.1.0 to 1.1.1 by @dependabot in #2830

New Contributors 👏🏼

• @emmanuel-ferdman made their first contribution in #2798
• @DavidKatz-il made their first contribution in #2806
• @AntonMantulo made their first contribution in #2819

Full Changelog: 3.7.3...3.8.0

AWS SDK for pandas 3.7.3

Bug fixes 🐛

• Iceberg schema evolution fails for map, array and struct types by @LeonLuttenberger in #2755
• trickle down s3_output in athena.to_iceberg by @jaidisido in #2767
• respect order of columns in to_iceberg by @jaidisido in #2768
• add PyArrow fixed_size_binary dtype support by @jaidisido in #2775
• Opensearch serverless vector search collections - remove default _id by @kukushking in #2784
• missing keys in list_to_arrow_table by @kukushking in #2778
• prevent athena.to_iceberg overwrite to delete table in order to preserve Iceberg transactions history by @erwan-simon in #2776

Documentation 📚

• simplify README, remove AWS Glue for Ray references by @jaidisido in #2750
• fix YAML formatting in Ray Remote tutorial by @LeonLuttenberger in #2770

Security Dependency Updates 🛡️

• bump idna from 3.6 to 3.7 by @dependabot in #2772
• bump aiohttp from 3.9.3 to 3.9.4 by @dependabot in #2777

New Contributors 👋

• @erwan-simon made their first contribution in #2776

Full Changelog: 3.7.2...3.7.3

AWS SDK for pandas 3.7.2

Features/Enhancements 🚀

• Add support for DeltaLake's DynamoDB lock mechanism by @LeonLuttenberger in #2705

Bug fixes 🐛

• wr.athena.to_iceberg - Insert query has mismatched column types #2678 by @GalvFionic in #2715
• allow s3_output in athena.to_iceberg by @jaidisido in #2727
• replace deprecated np.split_array by @jaidisido in #2735
• Athena to_iceberg fails with non-lowercase column names by @LeonLuttenberger in #2736
• Support Ray 2.10 by @kukushking in #2741

New Contributors

• @GalvFionic made their first contribution in #2715

Full Changelog: 3.7.1...3.7.2

AWS SDK for pandas 3.7.1

Bug fixes 🐛

• fix breaking change in _create_table by @jaidisido in #2711
• pin pyarrow to version 8 and above by @jaidisido in #2709

Documentation 📚

• fix redshift.to_sql doc indentation error by @LeonLuttenberger in #2706

Full Changelog: 3.7.0...3.7.1