PDEP-10: Add pyarrow as a required dependency #52711
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| # PDEP-10: PyArrow as a required dependency | ||
|
|
||
| - Created: 17 April 2023 | ||
| - Status: Under discussion | ||
| - Discussion: [#52711](https://github.com/pandas-dev/pandas/pull/52711) | ||
| [#52509](https://github.com/pandas-dev/pandas/issues/52509) | ||
| - Author: [Matthew Roeschke](https://github.com/mroeschke) | ||
| - Revision: 1 | ||
|
|
||
| ## Abstract | ||
|
|
||
| This PDEP proposes that: | ||
|
|
||
| - PyArrow becomes a runtime dependency starting pandas 2.1 | ||
| - The minimum version of PyArrow supported starting pandas 2.1 is version 7. | ||
|
There was a problem hiding this comment. Would this version be consistent across the entire pandas API? e.g. If I wanted to bump the pyarrow version for just the CSV parser to something higher, would I be able to do it? There was a problem hiding this comment. The minimum version would be consistent across the library, but IMO that shouldn't stop development of features that exist in newer versions of pyarrow (we already do this with version checking or try/except)
phofl marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - The minimum version of PyArrow will be bumped every major pandas release to the highest | ||
|
There was a problem hiding this comment. Hmm.. This might be too aggressive and might also make it hard to predict what the minimum version will be. I'd recommend following what we do for numpy, which is according to NEP 29, support There was a problem hiding this comment. I think the challenge with offering a similar support window for the two libraries is that NumPy has a very stable ABI whereas PyArrow does not There was a problem hiding this comment. Not sure if I'm missing something, but sounds like what @lithomas1 is proposing is pretty much the same as what's written in the proposal but phrased in a different way. Has the proposal been updated, or am I misunderstanding that supporting the releases of the last 24 months, and supporting the highest/oldest version two years old? There was a problem hiding this comment. I read this as every major release we will bump the min required version of pyarrow to the latest version, but might be misreading here. Note that my proposed change would be different in that we would drop Arrow versions in both major/minor versions (as opposed to every major version), just like we do with numpy (once we reach the end of the NEP support window).
I might have missed some more discussion on this, but I thought we were going to restrict current usage of pyarrow to just what's exposed through Python. There was a problem hiding this comment.
To the latest version About bumping in major or minor releases, I don't have a preference, either is fine for me. There was a problem hiding this comment. I can rephrase this to make it more clear but @datapythonista has it correct. The only distinction here, compared to what we do with numpy today, is that pyarrow would be bumped only during a pandas major release.
Under this proposal, PyArrow will only be used as a runtime dependency There was a problem hiding this comment. Can you consider this to upgrading pyarrow in both major and minor versions to be consistent with numpy? I ask because it is probably tricky for downstream to predict the length of our major release cycle (for 2.0 I think we delayed it twice. IIRC 1.4 was supposed to be 2.0). There was a problem hiding this comment.
Sure that would be okay with me too |
||
| PyArrow version that has been released for at least 2 years, and the minimum PyArrow version will be | ||
| maintained for every minor version in the major version series. | ||
|
|
||
| ## Background | ||
|
|
||
| PyArrow is an optional dependency of pandas that provides a wide range of supplemental features to pandas: | ||
|
|
||
| - Since pandas 0.21.0, PyArrow provided I/O reading functionality for Parquet | ||
| - Since pandas 1.2.0, pandas integrated PyArrow into the `ExtensionArray` interface to provide an optional string data type backed by PyArrow | ||
| - Since pandas 1.4.0, PyArrow provided I/0 reading functionality for CSV | ||
| - Since pandas 1.5.0, pandas provided an `ArrowExtensionArray` and `ArrowDtype` to support all PyArrow data types within the `ExtensionArray` interface | ||
| - Since pandas 2.0.0, All I/O readers have the option to return PyArrow-backed data types, and many methods now utilize PyArrow compute functions to | ||
phofl marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| accelerate PyArrow-backed data in pandas, notibly string and datetime types. | ||
|
|
||
| As of pandas 2.0, one can feasibly utilize PyArrow as an alternative data representation to NumPy with advantages such as: | ||
|
|
||
| 1. Consistent ``NA`` support for all data types | ||
| 2. Broader support of data types such as ``decimal``, ``date`` and nested types | ||
Dr-Irv marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ## Motivation | ||
|
|
||
| While all the functionality described in the previous paragraph is currently optional, PyArrow has significant integration into many areas | ||
| of pandas. With our roadmap noting that pandas strives for better Apache Arrow interoperability [^1] and many projects [^2], within or beyond the Python ecosystem, adopting or interacting with the Arrow format, making PyArrow a required dependency provides an additional signal of confidence in the Arrow | ||
phofl marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| ecosystem to pandas users. | ||
|
|
||
| Additionally, requiring PyArrow would simplify the related development within pandas and potentially improve NumPy functionality that would be better suited | ||
| by PyArrow including: | ||
|
|
||
| - Avoiding runtime checking if PyArrow is available to perform PyArrow object inference during constructor or indexing operations | ||
|
There was a problem hiding this comment. Are there any small code samples we can add to drive this point home? I think still we would make a runtime determination whether to return a pyarrow or numpy-backed object even if both are installed, no? There was a problem hiding this comment. not sure this comment by Will has been addressed (unless I missed it?) to make it easier to find: the link is here, and says:
|
||
| - Avoiding NumPy object data types more by default for analogous types that have native PyArrow support such as decimal, binary, and nested types | ||
|
There was a problem hiding this comment. I might be too optimistic, but having pyarrow as a required dependency has the potential to make the c/cython-code for read_csv and read_json obsolete (if they are on par and similarly fast). There was a problem hiding this comment. that would be a compile time dependency which we are not contemplating at the current time; possibly could propose in the future
phofl marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ## Drawbacks | ||
|
|
||
| Including PyArrow would naturally increase the installation size of pandas. For example, installing pandas and PyArrow using pip from wheels, numpy and pandas | ||
| are about `70MB`, and PyArrow is around `120MB`. An increase of installation size would have negative impliciation using pandas in space-constrained development | ||
| or deployment environments such as AWS Lambda. | ||
|
|
||
| Additionally, if a user is installing pandas in an environment where wheels are not available and needs to build from source, the user will need to build Arrow C++ and related dependencies. These environments include | ||
|
There was a problem hiding this comment. I think it would be good to more explicitly say that you need to do this (installing Arrow C++) manually and this is not possible through There was a problem hiding this comment. thanks for providing details here - is this ( There was a problem hiding this comment. Whoops, sorry there was a "not" missing in "not possible through But on the actual question how much of a hurdle this is: I would say, try it out yourself :) That's the best way to get an idea of how difficult it is, otherwise you can only take (or not) my words in saying that: yes, this is a huge hurdle. Installing pyarrow and Arrow C++ from source is far from trivial. There was a problem hiding this comment. thanks - from the installation instructions, it certainly seems tricky https://arrow.apache.org/docs/developers/cpp/building.html# |
||
|
|
||
| - Alpine linux (commonly used as a base for Docker containers) | ||
| - WASM (pyodide and pyscript) | ||
| - Python development versions | ||
|
|
||
| Lastly, pandas development and releases will need to be mindful of PyArrow's development and release cadance. For example when supporting a newly released Python version, pandas will also need to be mindful of PyArrow's wheel support for that Python version before releasing a new pandas version. | ||
|
|
||
| ### PDEP-1 History | ||
phofl marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| - 17 April 2023: Initial version | ||
phofl marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| [^1] <https://pandas.pydata.org/docs/development/roadmap.html#apache-arrow-interoperability> | ||
| [^2] <https://arrow.apache.org/powered_by/> | ||
attack68 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Uh oh!
There was an error while loading. Please reload this page.