Skip to content

[ArrowStringArray] PERF: use pyarrow.compute.utf8_length if available #41248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 4, 2021
Merged

[ArrowStringArray] PERF: use pyarrow.compute.utf8_length if available #41248

merged 1 commit into from
May 4, 2021

Conversation

simonjayhawkins
Copy link
Member 

[ 25.00%] ··· strings.Methods.time_len                                                                                                                    ok
[ 25.00%] ··· ============== ==========
                  dtype                
              -------------- ----------
                   str        19.1±0ms 
                  string      6.82±0ms 
               arrow_string   1.93±0ms 
              ============== ==========

@simonjayhawkins simonjayhawkins added Performance Memory or execution speed performance Strings String extension data type and string data labels May 1, 2021
@simonjayhawkins simonjayhawkins added this to the 1.3 milestone May 1, 2021
@simonjayhawkins simonjayhawkins changed the title [ArrowStringArray] use pyarrow.compute.utf8_length if available [ArrowStringArray] PERF: use pyarrow.compute.utf8_length if available May 4, 2021
jreback
jreback reviewed May 4, 2021
View reviewed changes
pandas/core/arrays/string_arrow.py
@@ -858,6 +859,14 @@ def _str_isupper(self):
else:
return super()._str_isupper()

def _str_len(self):
# utf8_length added in pyarrow 4.0.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should do this as a more general check i think (e.g. maybe defined a pyarrow_gt_400 or similar to what we do elsewhere rather than rely on checks like this (e.g. create a pyarrow.compat or similar). but let's create an issue and do a followup.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm

@jreback jreback merged commit 7c89062 into pandas-dev:master May 4, 2021
@jreback
Copy link
Contributor

jreback commented May 4, 2021

thanks comment above

@simonjayhawkins simonjayhawkins deleted the utf8_length branch May 4, 2021 18:15
JulianWgs pushed a commit to JulianWgs/pandas that referenced this pull request Jul 3, 2021
@simonjayhawkins @JulianWgs
[ArrowStringArray] use pyarrow.compute.utf8_length if available (pand…
fbaf64f 
…as-dev#41248)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jreback jreback jreback left review comments

Assignees
No one assigned
Labels
Performance Memory or execution speed performance Strings String extension data type and string data
Projects
None yet
Milestone
1.3
Development

Successfully merging this pull request may close these issues.
2 participants
@simonjayhawkins @jreback